Data processing method based on big data governance
Through big data governance methods, matching financial data analysis methods and data sources and screening data, the problems of data confusion and uncertainty in big data analysis are solved, the efficiency and accuracy of financial data processing are improved, and data quality is ensured. .
Patent Information
- Application Number
- CN202410976996.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-22
AI Technical Summary
In big data analysis, the confusion and uncertainty of the source of data lead to cumbersome data processing, low efficiency and difficult to guarantee results, affecting the accuracy and reliability of financial decisions.
Through a method based on big data governance, input financial data problems, match applicable financial data analysis methods, collect data in multiple directions, mark classification, calculate the matching rate of data sources and analysis methods, filter normal and abnormal data, and ensure data quality and accurate processing. sex.
It improves the efficiency and accuracy of data processing, ensures the quality of financial data, reduces processing errors, and enhances the systematicity and reliability of data processing.
Smart Images

Figure CN119006186A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and specifically to a data processing method based on big data governance. Background Art
[0002] With the development of the information age, big data technology has become an indispensable part of modern enterprise operations and management, and the application of big data governance analysis in financial decision-making has become increasingly important. Through in-depth analysis and mining of big data, valuable information can be obtained from massive amounts of data, thereby improving the accuracy and efficiency of financial decisions, helping companies make more accurate investment decisions, more comprehensive risk assessments, and more efficient preparation of financial reports.
[0003] However, when evaluating the effect of big data analysis, we need to pay attention to its comprehensiveness, authenticity and reliability to ensure the credibility and repeatability of the conclusions. Since the data information obtained through big data technology is relatively mixed, there are certain differences between the data, and it is difficult to ensure the reliability and accuracy of the data source, the data processing process is relatively cumbersome, which delays efficiency and makes it difficult to guarantee the processing results. To this end, we propose a data processing method based on big data governance. Summary of the invention
[0004] The purpose of the present invention is to provide a data processing method based on big data governance to solve the problems raised by the above-mentioned background technology.
[0005] The purpose of the present invention can be achieved by the following technical solution: A data processing method based on big data governance, comprising the following steps:
[0006] Step 1: Input the financial data problem to be processed, obtain the commonly used financial data analysis methods, match the financial data problem with the financial data analysis methods, obtain their matching rate, and select the financial data analysis method with the highest matching rate as the financial data processing analysis method;
[0007] Step 2: According to the problem to be processed and analyzed, collect relevant data from multiple directions and monitor the source of the data. The data source is recorded as the source database, and the data in each source database is marked and classified to obtain the data output coefficient of each source database;
[0008] Step 3: Screen the collected data, divide the data into normal and abnormal according to the data characteristics, and select the data that meets the requirements for financial data analysis and problem solving. At the same time, mark and classify the normal data and abnormal data during the screening process, thereby obtaining the index ratio of normal data and abnormal data to the total collected data of this period;
[0009] Step 4: Calculate the problems that need to be processed and analyzed based on the extracted data and the applicable financial data analysis method, and determine the accuracy of financial data processing by the data index, source database level and analysis method matching rate.
[0010] As a further improvement of the present invention, a commonly used financial data analysis method is obtained, and the specific acquisition method is as follows:
[0011] Financial data analysis methods are obtained through Internet big data or manual entry by financial personnel, including comparative analysis, ratio analysis and trend analysis. Based on the applicable scope and limitations of various financial data analysis methods relative to financial data processing, appropriate analysis methods are selected for the current problems to be processed. At the same time, when facing financial data processing problems that are beyond the analysis scope of a single financial data analysis method, a combination of multiple analysis methods is used to handle them.
[0012] As a further improvement of the present invention, the matching rate between the financial data analysis method and the financial data processing is obtained, specifically:
[0013] For financial data problems, the financial data analysis method currently used to process the financial data problem will be used to set the matching rate between the financial data analysis method and the financial data processing according to the probability of matching with it. If the matching probability is higher, the matching rate will be higher. The analysis method with a high matching rate will be given priority for processing the financial data problem. The financial data problem processed by the analysis method with a high matching rate will have a higher accuracy rate.
[0014] As a further improvement of the present invention, relevant data is collected from multiple directions and the data source is monitored. The specific collection method is as follows:
[0015] Data collection is completed through external websites of Internet big data, internal proprietary databases and manual entry by financial personnel, and the source of the data is set as the source database. Different first tags are given to data collected from different source databases, including the source of the data and the category of the data in the source data.
[0016] As a further improvement of the present invention, the data output coefficients of each source database are obtained, and the specific analysis process is as follows:
[0017] The first marks of the collected data in different source databases are statistically sorted, and the source databases with the largest, medium and smallest data collection amounts are marked as first-, second- and third-level source databases, respectively, and the corresponding data output coefficients are obtained according to the different levels of the source databases.
[0018] As a further improvement of the present invention, the collected data is screened for positive and abnormal conditions, specifically:
[0019] Conditional screening, sorting screening, classification and aggregation are used to complete the screening of collected data, including removing duplicate data, processing missing values and outliers, further reviewing the screened data, and classifying the screened data into positive and abnormal ones based on the characteristics of the abnormal data.
[0020] As a further improvement of the present invention, the normal and abnormal indexes of the data are obtained, and the specific analysis method is as follows:
[0021] The normal data and abnormal data after the division are respectively marked with a second label, and the number of the second labels is counted to obtain the proportion index of the normal data and the abnormal data in the total collected data of this period.
[0022] As a further improvement of the present invention, the second marker is matched with the first marker to obtain the source database to which the normal data and the abnormal data belong, and the data output coefficients of the normal data and the abnormal data relative to the source database are calculated, and the source database with more data output coefficients and more normal data is set as a superior database.
[0023] As a further improvement of the present invention, the financial data problem to be processed and analyzed is calculated, and the specific analysis method is:
[0024] The extracted and collected normal data is input into the financial data analysis method for data processing. When the proportion of normal data collected in the current period's data processing is high, the data comes from a superior database, and the matching rate of the financial data analysis method is high, it is an excellent data processing method.
[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention can set a source database for data extraction from big data and classify it into levels, so as to facilitate the extraction of data from the superior source database, facilitate the determination of data anomalies, ensure the data quality for financial data processing, and improve the efficiency of data acquisition. At the same time, a matching financial data analysis method can be selected for data processing according to the data problem to be processed, and the accuracy of data processing can be further improved in combination with high-quality data.
[0026] The present invention systematically combines data extraction, screening, analysis and judgment of financial data processing, thereby subdividing data processing into steps, which is beneficial to ensuring the quality of data processing, reducing data processing errors and improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.
[0028] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION
[0029] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] See also Figure 1 As shown, a data processing method based on big data governance includes the following steps:
[0031] Step 1: Input the financial data problem to be processed, obtain the commonly used financial data analysis methods, match the financial data problem with the financial data analysis methods, obtain their matching rate, and select the financial data analysis method with the highest matching rate as the financial data processing analysis method;
[0032] Obtain commonly used financial data analysis methods. The specific acquisition methods are as follows:
[0033] Obtain financial data analysis methods through Internet big data or manual input by financial personnel, including comparative analysis, ratio analysis and trend analysis... Based on the applicable scope and limitations of various financial data analysis methods relative to financial data processing, select appropriate analysis methods for the current problems to be processed. At the same time, when facing financial data processing problems that are beyond the scope of a single financial data analysis method, use a combination of multiple analysis methods to handle them;
[0034] Comparative analysis is a method of comparing actual financial data with specific standards, determining the difference in quantity, analyzing and judging the current financial status and investment performance of an individual or family. It is mainly used to reveal the quantitative relationship and gap in financial activities through comparative analysis, find problems, and is suitable for explaining the quantitative relationship and quantitative difference between financial information, and pointing out the direction for further analysis.
[0035] Ratio analysis is an analytical method that compares two related items that affect the financial situation and calculates the ratio between them to determine the degree of change in economic activities. These ratios can reflect the composition and efficiency of assets and liabilities, and are suitable for analyzing financial ratios to understand the financial situation and operating results of the company.
[0036] Trend analysis is a financial analysis method that uses the same indicators in the accounting statements of individuals or families in successive periods, and uses the calculation of indexes or completion rates to determine and analyze the changes and trends of related items in each period. It is suitable for revealing changes in financial status and operating results, their causes and nature, and helping to predict the future.
[0037] Obtain the matching ratio between the financial data analysis method and the financial data processing, specifically:
[0038] When facing a financial data problem to be processed, the financial data analysis method currently available for problem processing is used to set the matching rate between the financial data analysis method and the financial data processing according to the probability of matching with it. The higher the matching probability, the higher the matching rate. The analysis method with a high matching rate is given priority. The financial data problem processed by the analysis method with a high matching rate has a higher accuracy rate.
[0039] Several financial data analysis methods are set, each of which corresponds to several problem factors; the current financial data problem to be processed is decomposed to obtain corresponding problem factors; the decomposed problem factors are matched with the problem factors corresponding to the comparative analysis method, the ratio analysis method and the trend analysis method to obtain the number of matches, the number of matches is divided by the total number of decomposed problem factors to obtain the matching rate, and the financial data analysis method with the largest matching rate is marked as the financial data analysis method corresponding to the financial data problem to be processed.
[0040] If the problem factors decomposed from the financial data problem to be processed are not completely matched, the financial data analysis method corresponding to the financial data problem to be processed that has been matched is marked as the first method; the remaining unmatched problem factors are re-matched to obtain the second method, and so on, until the problem factors decomposed from the financial data problem to be processed are completely matched; the first method, the second method, etc. are combined with each other and processed together; so that financial data processing problems that are beyond the analysis scope of a single financial data analysis method can be processed by combining multiple financial data analysis methods.
[0041] For example, if the current financial data problem to be processed is: it is necessary to understand the company's current financial status and operating results, and predict the change status and reasons for the next period, the problem factors to be processed are financial status, operating results, subsequent change status and reason prediction, and the problem factors are set as factors A, B, and C respectively;
[0042] Among them, the comparative analysis method has financial status analysis, so it has factor A. The ratio analysis method uses financial ratio analysis to understand the financial status and operating results of the company, so it has factors A and B. The trend analysis method is suitable for revealing the changes in financial status and operating results and their causes and nature, and helping to predict the future, so it has factors A, B, and C. Match the factors of each analysis method with the factors of the financial data problem to be processed and substitute them into the formula The matching rate ∑ is obtained, among which the matching rate of ∑3 is the closest, and the trend analysis method represented by ∑3 is preferred;
[0043] For example, if the current financial data problem to be processed is: it is necessary to understand the company's current financial status and operating results, and predict the change status and reasons for the next period, and the impact of factors related to the change reasons on financial indicators, the problem factors to be processed are financial status, operating results, subsequent change status and reason prediction, and the impact of factors related to the change reasons on financial indicators. The problem factors are set as factors A, B, C, and D respectively;
[0044] Among them, the comparative analysis method has financial status analysis, so it has factor A. The ratio analysis method uses financial ratio analysis to understand the financial status and operating results of the company, so it has factors A and B. The trend analysis method is suitable for revealing the changes in financial status and operating results and their causes and nature, and helping to predict the future, so it has factors A, B, and C. The factor analysis method is used to analyze the degree of influence of several related factors on a certain financial indicator, so it has factor D. Match the factors of each analysis method with the factors of the financial data problem to be processed and substitute them into the formula Then a matching rate ∑ is obtained, among which ∑3 has the closest matching rate, but fails to completely match the problem factors corresponding to the financial data problem to be processed, then ∑3 is marked as the first method, and the remaining unmatched problem factors of the financial data problem to be processed can be matched with ∑4, then ∑4 is marked as the second method, and the first method and the second method are used together to analyze and process the financial data problem to be processed.
[0045] Step 2: According to the problem to be processed and analyzed, collect relevant data from multiple directions and monitor the source of the data. The data source is recorded as the source database, and the data in each source database is marked and classified to obtain the data output coefficient of each source database;
[0046] Collect relevant data from multiple directions and monitor the data sources. The specific collection methods are as follows:
[0047] Complete data collection through external websites of Internet big data, internal proprietary databases, and manual entry by financial personnel, and set the source of the data as the source database. Perform different first tags on the data collected from different source databases, including the source of the data and the category of the data in the source data.
[0048] If a total of P data are collected for the processing of the financial data problem in this period, of which i data are collected from external websites of Internet big data, internal proprietary databases, and manual entry by financial personnel, then the external website, internal proprietary database, and manual entry can all be set as source databases. At the same time, different first tags are given to the data collected in different source databases. The data collected from the external website is marked as U, the data collected from the internal proprietary database is marked as V, and the data collected manually entered by financial personnel is marked as W.
[0049] The data output coefficients of each source database are obtained. The specific analysis process is as follows:
[0050] The first marks of the collected data in different source databases are statistically sorted, and the source databases with the largest, medium and smallest data collection amounts are marked as first-, second- and third-level source databases, respectively, and corresponding data output coefficients are obtained according to the different levels of the source databases;
[0051] When a total of P data are collected for the current financial data problem processing, of which the external website of the Internet big data, the internal own database, and the manual entry of financial personnel collect i data respectively, i=i1, i2, i3, i1+i2+i3=P, substitute into the formula The data output coefficients of different source databases are obtained, where P is the total data collection volume, i is the data collection volume of different source databases, and U', V', and W' respectively represent the proportion of the data output coefficients of the corresponding source databases. U', V', and W' are compared with each other and sorted according to the proportion of the data output coefficients. The corresponding source databases are set as the first, second, and third level source databases.
[0052] Step 3: Screen the collected data, divide the data into normal and abnormal according to the data characteristics, and select the data that meets the requirements for financial data analysis and problem solving. At the same time, mark and classify the normal data and abnormal data during the screening process, thereby obtaining the index ratio of normal data and abnormal data to the total collected data of this period;
[0053] The collected data is screened for positive and abnormal conditions, specifically:
[0054] Use conditional screening, sorting screening, classification and aggregation methods to complete the screening of collected data, including removing duplicate data, processing missing values and outliers, further reviewing the screened data, and classifying the screened data into positive and abnormal according to the characteristics of abnormal data;
[0055] Conditional screening refers to filtering out data that meets the conditions according to the set conditions, such as filtering out data within a certain time period, data of a certain category, etc.; sorting screening refers to sorting data according to the set sorting rules for easy reference and analysis; classification and aggregation refers to classifying and summarizing the filtered data according to certain classification standards for further analysis and report output;
[0056] The financial data that have passed the preliminary screening will be further judged and analyzed in combination with the data source, and several preliminary screening conditions will be set. The preliminary screening conditions include that the growth rate of operating income is lower than the growth rate of accounts receivable, and is contrary to the operating cash flow; the operating profit has increased significantly, while the growth rate of operating costs and sales expenses is small; the value-added tax and corporate income tax expenses are abnormally low, which does not match the growth rate of income and profits...etc.
[0057] If the collected data meets the preliminary screening conditions, it is judged as abnormal data; if it does not meet the preliminary screening conditions, it is judged as normal data.
[0058] The normal and abnormal indexes of the data are obtained. The specific analysis method is as follows:
[0059] The normal data and abnormal data after the division are respectively marked for the second time, and the number of the second marks is counted to obtain the proportion index of the normal data and the abnormal data in the total collected data of this period;
[0060] When a total of P data are collected for the current financial data processing, and after the division, the second mark of normal data is X, and the second mark of abnormal data is Y, then substitute the formula The proportion indexes PX and PY of normal data and abnormal data in the total collected data are obtained.
[0061] Step 4: Calculate the problems that need to be processed and analyzed based on the extracted data and the applicable financial data analysis method, and determine the accuracy of financial data processing by the data index, source database level and analysis method matching rate.
[0062] The second mark is matched with the first mark to obtain the source database to which the normal data and the abnormal data belong, and the data output coefficients of the normal data and the abnormal data relative to the source database are calculated, and the source database with more data output coefficients and more normal data is set as the superior database;
[0063] When a total of P data are collected for the current financial data processing, the number of normal data second marks is X, and the number of abnormal data second marks is Y, the second marks X and Y can be divided into their respective source databases through the source databases U, V, and W for the first marks of the output data, and set as X1, Y1, X2, Y2, X3, and Y3, and substituted into the formula The proportion index of normal data and abnormal data in the data output coefficient of each source database is obtained, i represents the output quantity of data from each source database, U', V', W' represent the proportion of the output coefficient of the data from each source database relative to the total collected data;
[0064] Compare U', V', W' and U1', V1', W1' with each other. The source database with a large output coefficient ratio and a large normal data output ratio index is set as the superior database, the source database with a large output coefficient ratio and a small normal data output ratio index is set as the inferior database, and the source database with a small output coefficient ratio and a large normal data output ratio index is set as the medium database.
[0065] Calculate the financial data problems that need to be processed and analyzed. The specific analysis method is as follows:
[0066] The normal data collected is extracted and input into the financial data analysis method for data processing. When the proportion of normal data collected in the current period data processing is high, the data comes from a superior database, and the matching rate of the financial data analysis method is high, the accuracy of the current period financial data processing is high;
[0067] When the index proportion of normal data collected in the current data processing is greater than the index proportion of abnormal data, it indicates that the quality of data collection in this period is high and suitable for data processing. Otherwise, it indicates that the quality of data collection in this period is low and not suitable for data processing. In this case, data collection needs to be performed again to ensure the accuracy of data processing. When the data in this period comes from a superior database, it indicates that the data is collected on a large data platform and has a high quality, which is conducive to ensuring the accuracy of data processing. When the financial data analysis method used in the current data processing has a high matching rate, it indicates that the financial data analysis method selected this time is more applicable, which is conducive to further improving the accuracy of data processing.
[0068] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. A data processing method based on big data governance, characterized in that: The following steps are involved: Step 1: Input the financial data problem to be processed, obtain the preset financial data analysis method, match the financial data problem with the financial data analysis method, obtain the matching rate, and select the financial data analysis method with the highest matching rate to analyze and process the required financial data problem; Step 2: According to the problem to be processed and analyzed, collect relevant data from multiple directions and monitor the source of the data. The data source is recorded as the source database, and the data in each source database is marked and classified to obtain the data output coefficient of each source database; Step 3: Screen the collected data, divide the data into normal and abnormal according to the data characteristics, and select the data that meets the requirements for financial data analysis and processing. At the same time, mark and classify the normal data and abnormal data during the screening process, so as to obtain the index ratio of normal data and abnormal data to the total collected data of this period; Step 4: Calculate the problems that need to be processed and analyzed based on the extracted data and the applicable financial data analysis method, and determine the accuracy of financial data processing by the data index, source database level and analysis method matching rate.
2. According to the data processing method based on big data management according to claim 1, it is characterized in that: The specific method of obtaining the preset financial data analysis method is: Financial data analysis methods are obtained through Internet big data or manual entry by financial personnel. Financial data analysis methods include comparative analysis, ratio analysis and trend analysis. Based on the scope of application and limitations of financial data analysis methods relative to financial data processing, the corresponding financial data analysis method is selected for the current problem to be processed. At the same time, when facing financial data processing problems that exceed the analysis scope of a single financial data analysis method, a combination of multiple financial data analysis methods is used to handle them.
3. According to the data processing method based on big data management according to claim 1, it is characterized in that: Obtain the matching ratio between the financial data analysis method and the financial data processing, specifically: For financial data problems, the financial data analysis method currently used to process the financial data problem will be used, and the matching rate between the financial data analysis method and the financial data processing will be set according to the probability of matching with it. If the matching probability is higher, the matching rate will be higher, and the analysis method with a high matching rate will be given priority as the processing of financial data problems.
4. According to the data processing method based on big data governance according to claim 1, it is characterized in that: The specific collection methods for collecting relevant data from multiple directions and monitoring data sources are as follows: Data collection is completed through external websites of Internet big data, internal proprietary databases and manual entry by financial personnel, and the source of the data is set as the source database. Different first tags are given to data collected from different source databases, including the source of the data and the category of the data in the source data.
5. The data processing method based on big data governance according to claim 1 is characterized in that: The specific analysis process of obtaining the data output coefficients of each source database is as follows: The first marks of the collected data in different source databases are statistically sorted, and the source databases with the largest, medium and smallest data collection amounts are marked as first-, second- and third-level source databases, respectively, and the corresponding data output coefficients are obtained according to the different levels of the source databases.
6. The data processing method based on big data governance according to claim 1 is characterized in that: The positive and abnormal screening of the collected data is specifically as follows: Conditional screening, sorting screening, and classification summary methods are used to complete the screening of collected data, including removing duplicate data, processing missing values and outliers, further reviewing the screened data, and classifying the screened data into positive and abnormal ones based on the characteristics of the abnormal data.
7. The data processing method based on big data governance according to claim 1 is characterized in that: The normal and abnormal indexes of the obtained data are specifically analyzed in the following manner: The normal data and abnormal data after the division are respectively marked with a second label, and the number of the second labels is counted to obtain the proportion index of the normal data and the abnormal data in the total collected data of this period.
8. The data processing method based on big data governance according to claim 1 is characterized in that: The second marker is matched with the first marker to obtain the source database to which the normal data and the abnormal data belong, and the data output coefficients of the normal data and the abnormal data relative to the source database are calculated. The source database with more data output coefficients and more normal data is set as the superior database.
9. The data processing method based on big data governance according to claim 1 is characterized in that: The financial data problem to be processed and analyzed is calculated, and the specific analysis method is as follows: The extracted and collected normal data is input into the financial data analysis method for data processing. When the proportion of normal data collected in the current period's data processing is high, the data comes from a superior database, and the matching rate of the financial data analysis method is high, it is an excellent data processing method.
