A method and system for audit data analysis based on big data

Abstract

The present invention relates to the field of data processing technology, and in particular to an audit data analysis method and system based on big data. The method includes: obtaining each numerical feature dimension and each text feature dimension of the audit data; determining the amount of information of each numerical feature dimension; determining the importance of each numerical feature dimension; taking the ratio of the number of data types of each text feature dimension and the number of audit data as the importance of each text feature dimension; determining the characteristic distance between each audit data and the remaining audit data; taking the characteristic distance as the distance between the calculated data in the LOF algorithm, and obtaining the local outlier factor of each data point through the LOF algorithm, so as to complete the audit data analysis based on big data. The present invention improves the calculation method of the characteristic distance by determining the amount of information and importance of each numerical feature dimension and the importance of each text feature dimension, so that the LOF algorithm can more accurately distinguish normal data from abnormal data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an audit data analysis method and system based on big data. Background Art

[0002] Audit data analysis plays a vital role in modern auditing. Its core lies in helping enterprises discover potential risks through the analysis of operational data. The results of audit data analysis are highly dependent on the accuracy of audit data. In the process of collecting audit data, statistical errors are inevitable, which results in abnormal data in the audit data, affecting the results of audit data analysis. Therefore, it is necessary to process the audit data abnormally when conducting audit data analysis. Usually, the Local Outlier Factor (LOF) algorithm is used to identify outliers in audit data. When the audit data has fewer dimensions or a simpler structure, it has a better abnormal data detection effect.

[0003] The patent application document with publication number CN115952067A discloses a method for detecting abnormal behavior of database operation and a readable storage medium, wherein the detection method comprises the following steps: S01) parsing, converting and collecting database audit log data; S02) using LOF algorithm to detect abnormal points; S03) using quartile algorithm to detect abnormal behavior of database operation; S04) combining the abnormal detection results of LOF algorithm and quartile algorithm, jointly detecting output abnormality, and obtaining the final abnormal behavior detection result of database operation.

[0004] However, the above document does not solve the problem of measuring the gaps between data only through Euclidean distance when using the LOF algorithm. When the audit data is high-dimensional data or there are large scale differences between the data of each dimension in the audit data, the LOF algorithm cannot accurately distinguish between normal data and abnormal data, which affects the reliability of audit data analysis. Summary of the invention

[0005] In order to solve the problem that when the audit data is high-dimensional data or there are large scale differences between the various dimensional data in the audit data, the LOF algorithm cannot accurately distinguish normal data from abnormal data using Euclidean distance, which affects the reliability of audit data analysis, the present invention provides an audit data analysis method and system based on big data.

[0006] In a first aspect, the present invention provides an audit data analysis method based on big data, which adopts the following technical solution:

[0007] A method for analyzing audit data based on big data, comprising: obtaining each numerical feature dimension and each text feature dimension of audit data; determining the amount of information of each numerical feature dimension according to the standard deviation of data values ​​in each numerical feature dimension and the difference between the mean of data values ​​greater than the mean and less than the mean in each numerical feature dimension, wherein the amount of information is positively correlated with the difference between the standard deviation and the mean; obtaining the correlation coefficient between each numerical feature dimension and other numerical feature dimensions, determining the importance of each numerical feature dimension according to the difference between the maximum value and the minimum value in the absolute value of the information and the correlation coefficient, wherein the importance is positively correlated with the difference between the maximum value and the minimum value in the absolute value of the information and the correlation coefficient. Positive correlation; the ratio of the number of data types in each text feature dimension to the number of audit data is used as the importance of each text feature dimension; the feature distance between each audit data and the remaining audit data is determined, and the feature distance is positively correlated with the importance of each numerical feature dimension and the product of the difference in data values ​​between each audit data and the remaining audit data in the same numerical feature dimension, the importance of each text feature dimension and the gap between each audit data and the remaining audit data in the same text feature dimension; the feature distance is used as the distance between the calculated data in the LOF algorithm, and the local outlier factor of each data point is obtained through the LOF algorithm to complete the audit data analysis based on big data.

[0008] The beneficial effects are as follows: the information content of the numerical feature dimension is determined by the difference between the standard deviation and the mean, and its importance is evaluated according to the correlation coefficient of each feature dimension, ensuring that important features are given priority in different data sets and situations, thereby improving the adaptability and flexibility of the analysis; the ratio of the number of data types in the text feature dimension to the amount of audit data is used as an importance consideration, which increases the analysis capability of unstructured data, allowing audit data analysis to capture the information in the data more comprehensively;

[0009] The calculation of feature distance comprehensively considers the importance of numerical features and text features, can better reflect the real gap between data, improves the calculation effect of local outlier factor of LOF algorithm, and more accurately identifies abnormal data; it can be applied to various types of audit data analysis, has good scalability, and is suitable for audit needs in different fields; in summary, through comprehensive analysis and dynamic evaluation of data features, the accuracy, flexibility and efficiency of audit data analysis are improved, which can effectively cope with complex audit scenarios.

[0010] Furthermore, the audit data is audit data after duplicate data is removed through retrieval.

[0011] Furthermore, the data values ​​in each numerical feature dimension are normalized results using Z-score standardization.

[0012] Furthermore, the amount of information satisfies the following relationship:

[0013] ; In the formula, For the The amount of information in the numerical feature dimension, For the The standard deviation of the data values ​​in the numerical feature dimension, For the In the numerical feature dimensions, the The mean of the data values ​​of the mean of the numerical feature dimensions, For the The numerical feature dimensions are smaller than The mean of the data values ​​of the mean of the numerical feature dimensions.

[0014] The beneficial effects are: by combining the standard deviation and the mean difference to quantify the amount of information, the information carrying capacity of each feature dimension can be quantitatively analyzed, which helps to compare different feature dimensions and select features with larger amounts of information for in-depth analysis; through the distribution of data values ​​on both sides of the mean, a larger difference means that the feature dimension has richer information in the data set, and differential analysis can help discover potential abnormal data or trends and improve the accuracy of audit results.

[0015] Further, the correlation coefficient is a result of the ratio of the covariance between the two numerical feature dimensions and the standard deviation of the two numerical feature dimensions.

[0016] Furthermore, the importance satisfies the following relationship:

[0017] ; In the formula, For the The importance of the numerical feature dimension, For the The amount of information in the numerical feature dimension, For the The maximum value of the absolute value of the correlation coefficient between the numerical feature dimension and other numerical feature dimensions, For the The minimum value of the absolute value of the correlation coefficient between the numerical feature dimension and other numerical feature dimensions, For the The mean of the absolute values ​​of the correlation coefficients between the numerical feature dimensions and other numerical feature dimensions, For preset adjustments Corrected ability parameters.

[0018] The beneficial effects are: by calculating the importance of each feature dimension, features that have a significant impact on the target variable can be selected more effectively; the calculation of importance helps to obtain the correlation between different features and improve the accuracy and stability of the analysis.

[0019] Furthermore, the characteristic distance satisfies the following relationship:

[0020] ; In the formula, For the Audit data and The characteristic distance between the audit data, is the number of numerical feature dimensions of the audit data, For the The importance of the numerical feature dimension, and Respectively Audit data and The audit data is in The data values ​​in the numerical feature dimensions, is the number of text feature dimensions of the audit data, For the The importance of the text feature dimension, For the The number of data types in the text feature dimension, is the number of audit data, For the Audit data and The audit data is in The gap between the text feature dimensions.

[0021] The beneficial effects are: by calculating the feature distance by considering both numerical features and text features at the same time, the multidimensional relationship between the data is reflected, thereby improving the reliability of the audit results; the introduction of feature importance weights can effectively highlight the features that have a greater impact on the analysis results, reduce the interference of secondary features, and improve the accuracy of audit data analysis; by calculating the gaps in the text feature dimensions, the differences between text data can be quantified to help identify potential problems or anomalies.

[0022] Furthermore, the gap is obtained in the following manner: if two audit data are the same in the same text feature dimension, the gap is 0, and if they are different, the gap is 1.

[0023] The beneficial effects are: using 0 and 1 to quantify gaps makes data relatively simple and easy to understand, reduces manual comparison time, and improves audit work efficiency; provides quantitative data support to help identify potential problems and risks.

[0024] Furthermore, the method of completing the audit data analysis based on big data includes: in response to the local outlier factor being greater than a preset threshold, determining that the audit data is abnormal data, and issuing a prompt to complete the audit data analysis based on big data.

[0025] In a second aspect, the present invention provides an audit data analysis system based on big data, which adopts the following technical solution:

[0026] An audit data analysis system based on big data includes: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned audit data analysis method based on big data is implemented.

[0027] By adopting the above technical solution, the above-mentioned audit data analysis method based on big data is generated into a computer program and stored in a memory to be loaded and executed by a processor, so that a terminal device is made according to the memory and the processor for easy use.

[0028] The present invention has the following technical effects:

[0029] By determining the amount of information and importance of each numerical feature dimension and the importance of each text feature dimension, the calculation method of feature distance is improved, so that the LOF algorithm can more accurately distinguish normal data from abnormal data; by calculating the feature distance by comprehensively considering the importance of each numerical feature dimension and the importance of each text feature dimension, the impact of irrelevant or redundant features on anomaly detection is reduced, and the overall performance and efficiency of audit data analysis based on big data are improved. By relying on the high-dimensional characteristics of audit data and the scale differences between data of each dimension, the limitations of traditional Euclidean distance in high-dimensional space are overcome, and the robustness and reliability of audit data analysis are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-restrictive manner, and the same or corresponding reference numerals represent the same or corresponding parts.

[0031] Figure 1 It is a method flow chart of an audit data analysis method based on big data in an embodiment of the present invention.

[0032] Figure 2 It is a schematic diagram of various numerical feature dimensions and various text feature dimensions of audit data in an audit data analysis method based on big data in an embodiment of the present invention.

[0033] Figure 3This is the first method of analyzing audit data based on big data in the embodiment of the present invention. A schematic diagram of the relative sizes of the main data distribution ranges in the numerical feature dimensions. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0035] It should be understood that when the terms "first", "second", etc. are used in the claims, descriptions, and drawings of the present invention, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the description and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0036] The embodiment of the present invention discloses an audit data analysis method based on big data, referring to Figure 1 , comprising steps S1 to S6:

[0037] S1: Obtain each numerical feature dimension and each text feature dimension of the audit data.

[0038] Specifically, the audit data is the audit data after the duplicate data is removed by retrieval, and each audit data is divided into a data feature part and a text feature part, see Figure 2 .

[0039] S2: Determine the amount of information in each numerical feature dimension.

[0040] It should be noted that since data of different dimensions in the audit data have different amounts of information, in the process of audit data analysis, data of dimensions with higher amounts of information should have higher importance, while data of dimensions with lower amounts of information should have lower importance. Therefore, the present invention calculates the amount of information of each numerical feature dimension based on the standard deviation and mean of the data of each numerical feature dimension in the audit data.

[0041] The amount of information of each numerical feature dimension is determined according to the standard deviation of the data values ​​in each numerical feature dimension and the difference between the mean of the data values ​​greater than the mean and less than the mean in each numerical feature dimension.

[0042] Specifically, the data values ​​in each numerical feature dimension are normalized results using standard scores (Z-score Normalization, Z-score).

[0043] Specifically, the amount of information satisfies the following relationship:

[0044] ;

[0045] In the formula, For the The amount of information in the numerical feature dimension, For the The standard deviation of the data values ​​in the numerical feature dimension, For the In the numerical feature dimensions, the The mean of the data values ​​of the mean of the numerical feature dimensions, For the The numerical feature dimensions are smaller than The mean of the data values ​​of the mean of the numerical feature dimensions.

[0046] in, For the The standard deviation of the numerical feature dimension data is larger, The higher the volatility and diversity of the data in the numerical feature dimension, the The greater the amount of information in the numerical feature dimension, the smaller the value is, indicating that The lower the volatility and diversity of the data in the numerical feature dimension, the The smaller the amount of information in each numerical feature dimension; For the The relative size of the main data distribution range in the numerical feature dimension (see Figure 3 ), the larger the value, the The mean difference between the data part with a numerical feature dimension greater than the mean and the data part with a numerical feature dimension less than the mean is large, indicating that The more likely the data of the numerical feature dimension is to have obvious separation near the mean, the higher the volatility and diversity of the data, and the The greater the amount of information in the numerical feature dimension, the smaller the value, indicating that The difference between the mean values ​​of the data parts with numerical feature dimension values ​​greater than the mean and those with values ​​less than the mean is small, indicating that The more concentrated the data is around the mean, the lower the volatility and diversity of the data. The smaller the amount of information in the numerical feature dimension.

[0047] S3: Determine the importance of each numerical feature dimension.

[0048] It should be noted that, since there may be certain correlations between data of some dimensions in the audit data, under this premise, there may be excessive reliance on data of some dimensions in the audit data and neglect of other valuable information. In order to avoid ignoring the impact of the correlation between data of various dimensions in the audit data on the importance of each numerical feature dimension, the present invention calculates the importance of each numerical feature dimension based on the amount of information of each numerical feature dimension and the correlation coefficient between the data of each dimension.

[0049] The correlation coefficient between each numerical feature dimension and other numerical feature dimensions is obtained, and the importance of each numerical feature dimension is determined according to the difference between the maximum value and the minimum value of the absolute value of the information amount and the correlation coefficient.

[0050] Specifically, the correlation coefficient is the result of the ratio of the covariance between the two numerical feature dimensions and the standard deviation of the two numerical feature dimensions.

[0051] Specifically, the importance satisfies the following relationship:

[0052] ;

[0053] In the formula, For the The importance of the numerical feature dimension, For the The amount of information in the numerical feature dimension, For the The maximum value of the absolute value of the correlation coefficient between the numerical feature dimension and other numerical feature dimensions, For the The minimum value of the absolute value of the correlation coefficient between the numerical feature dimension and other numerical feature dimensions, For the The mean of the absolute values ​​of the correlation coefficients between the numerical feature dimensions and other numerical feature dimensions, For preset adjustments Corrected ability parameters.

[0054] Implementers can set adjustments based on actual implementation conditions. Modify the capability parameter, for example, 2.

[0055] in, For the The maximum difference in the absolute value of the correlation coefficient between the numerical feature dimension and other numerical feature dimensions. The larger the value, the greater the The more likely it is that a numerical feature dimension has a certain correlation with only some numerical feature positions, the The data of the numerical feature dimension may provide more independent information and is more likely to help identify the gap between the data independently. Therefore, the larger the value, the better. The greater the importance of the numerical feature dimension, the smaller the value is, indicating that A numerical feature dimension may have a large correlation with all other numerical feature dimensions or a small correlation with all other numerical feature dimensions. right make corrections; The smaller the time, the more There may be a small correlation between a numerical feature dimension and all other numerical feature dimensions. right The greater the upward correction, the greater the The greater the importance of a numerical feature dimension, The larger the value, the The more likely it is that a numerical feature dimension has a greater correlation with all other numerical feature dimensions, the more likely it is that there is a greater correlation between them. right The greater the downward correction, the greater the The importance of the numerical feature dimension is smaller; at the same time, the amount of information in the numerical feature dimension The judgment of the importance of numerical feature dimensions is also of certain value. A dimension with more information means that it contains more changes and patterns, and this dimension can better capture anomalies or deviations in the data. The bigger, the The greater the importance of a numerical feature dimension, the smaller the information content of the dimension, which often means that the data of this dimension has less difference between samples, and the data of this dimension may not capture anomalies or deviations in the data well. The smaller, the The smaller the importance of the numerical feature dimension.

[0056] S4: The ratio of the number of data types of each text feature dimension to the amount of audit data is taken as the importance of each text feature dimension.

[0057] S5: Determine the characteristic distance between each audit data and the remaining audit data.

[0058] It should be noted that in the process of identifying outliers in audit data, text feature data also has a certain degree of importance. Therefore, the present invention calculates the feature distance between each audit data according to the importance of each numerical feature dimension and the importance of the text feature dimension.

[0059] Specifically, the characteristic distance satisfies the following relationship:

[0060] ;

[0061] In the formula, For the Audit data and The characteristic distance between the audit data, is the number of numerical feature dimensions of the audit data, For the The importance of the numerical feature dimension, and Respectively Audit data and The audit data is in The data values ​​in the numerical feature dimensions, is the number of text feature dimensions of the audit data, For the The importance of the text feature dimension, For the The number of data types in the text feature dimension, is the number of audit data, For the Audit data and The audit data is in The gap between the text feature dimensions.

[0062] Specifically, the gap is obtained in that if two audit data are the same in the same text feature dimension, the gap is 0, and if they are different, the gap is 1.

[0063] in, For the The importance of the text feature dimension, The larger the The text feature dimension has more data types and plays a more important role in distinguishing abnormal data points when calculating the feature distance between data. The larger the The greater the importance of a text feature dimension, The smaller the The smaller the importance of the first text feature dimension, the smaller the The importance of the numerical feature dimension The larger the Audit data and The characteristic distance between the audit data The larger the proportion, the more important the feature is for identifying abnormal data. The importance of the numerical feature dimension The more hours, the Audit data and The characteristic distance between the audit data The smaller the proportion, the smaller the influence of unimportant features on identifying abnormal data. Similarly, I will not go into details here.

[0064] S6: Using the characteristic distance as the distance between the data calculated in the LOF algorithm, the local outlier factor of each data point is obtained through the LOF algorithm to complete the audit data analysis based on big data.

[0065] Specifically, the audit data analysis based on big data includes:

[0066] In response to the local outlier factor being greater than a preset threshold, the audit data is identified as abnormal data and a prompt is issued. At this time, it is important to check whether the audit data is caused by a statistical error. After processing all data that may be outliers, the audit data is analyzed again. This can reduce the impact of data caused by statistical errors on the audit data analysis results to a certain extent, and complete the audit data analysis based on big data.

[0067] Implementers can set the threshold according to actual implementation conditions, for example, 1.

[0068] An embodiment of the present invention also discloses an audit data analysis system based on big data, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, an audit data analysis method based on big data according to the present invention is implemented.

[0069] The above system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, and their configuration and functions are known in the art, so they will not be described in detail here.

[0070] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, or any other medium that can be used to store the required information and can be accessed by an application, module, or both. Any such computer storage medium may be part of the device or accessible or connectable to the device.

[0071] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

[0072] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An audit data analysis method based on big data, characterized in that: include: Obtain each numerical feature dimension and each text feature dimension of the audit data; According to the standard deviation of the data values ​​in each numerical feature dimension and the difference between the mean of the data values ​​greater than the mean and less than the mean in each numerical feature dimension, the amount of information of each numerical feature dimension is determined, and the amount of information is positively correlated with the difference between the standard deviation and the mean; the correlation coefficient between each numerical feature dimension and the remaining numerical feature dimensions is obtained, and the importance of each numerical feature dimension is determined according to the difference between the maximum value and the minimum value in the absolute value of the information amount and the correlation coefficient, and the importance is positively correlated with the difference between the maximum value and the minimum value in the absolute value; the ratio of the number of data types in each text feature dimension to the number of audit data is used as the importance of each text feature dimension; Determine the characteristic distance between each audit data and the rest of the audit data. The characteristic distance is positively correlated with the importance of each numerical characteristic dimension and the product of the difference between the data value of each audit data and the rest of the audit data in the same numerical characteristic dimension, the importance of each textual characteristic dimension and the gap between each audit data and the rest of the audit data in the same textual characteristic dimension; The characteristic distance is used as the distance between the calculated data in the LOF algorithm, and the local outlier factor of each data point is obtained through the LOF algorithm to complete the audit data analysis based on big data.

2. According to the big data-based audit data analysis method of claim 1, it is characterized in that: The audit data is audit data after duplicate data is removed through retrieval.

3. The audit data analysis method based on big data according to claim 1 is characterized in that: The data values ​​in each numerical feature dimension are normalized results using Z-score standardization.

4. The audit data analysis method based on big data according to claim 1 is characterized in that: The amount of information satisfies the following relationship: ； In the formula, For the The amount of information in the numerical feature dimension, For the The standard deviation of the data values ​​in the numerical feature dimension, For the In the numerical feature dimensions, the The mean of the data values ​​of the mean of the numerical feature dimensions, For the The numerical feature dimensions are smaller than The mean of the data values ​​of the mean of the numerical feature dimensions.

5. The audit data analysis method based on big data according to claim 1 is characterized in that: The correlation coefficient is the result of the ratio of the covariance between two numerical feature dimensions and the standard deviation of the two numerical feature dimensions.

6. The audit data analysis method based on big data according to claim 1 is characterized in that: The importance satisfies the following relationship: ； In the formula, For the The importance of the numerical feature dimension, For the The amount of information in the numerical feature dimension, For the The maximum value of the absolute value of the correlation coefficient between the numerical feature dimension and other numerical feature dimensions, For the The minimum value of the absolute value of the correlation coefficient between the numerical feature dimension and other numerical feature dimensions, For the The mean of the absolute values ​​of the correlation coefficients between the numerical feature dimensions and other numerical feature dimensions, For preset adjustments Corrected ability parameters.

7. The audit data analysis method based on big data according to claim 1 is characterized in that: The characteristic distance satisfies the following relationship: ； In the formula, For the Audit data and The characteristic distance between the audit data, is the number of numerical feature dimensions of the audit data, For the The importance of the numerical feature dimension, and Respectively Audit data and The audit data is in The data values ​​in the numerical feature dimensions, is the number of text feature dimensions of the audit data, For the The importance of the text feature dimension, For the The number of data types in the text feature dimension, is the number of audit data, For the Audit data and The audit data is in The gap between the text feature dimensions.

8. The audit data analysis method based on big data according to claim 1 or 7, characterized in that: The gap is obtained in the following way: if two audit data are the same in the same text feature dimension, the gap is 0, and if they are different, the gap is 1.

9. The audit data analysis method based on big data according to claim 1 is characterized in that: The above mentioned method is to complete the audit data analysis based on big data, including: In response to the local outlier factor being greater than a preset threshold, the audit data is identified as abnormal data, and a prompt is issued to complete the audit data analysis based on big data.

10. An audit data analysis system based on big data, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, an audit data analysis method based on big data according to any one of claims 1 to 9 is implemented.

Images