Data distribution inspection method and device, electronic equipment and storage medium

Abstract

The invention provides a data distribution inspection method and device, electronic equipment and a storage medium, and the method comprises the steps: determining at least one group of to-be-inspected data and at least one corresponding distribution type based on pre-distribution inspection configuration information; judging whether each group of to-be-inspected data needs to be subjected to data processing operation including outlier elimination and / or non-negative processing or not, and performing data processing operation corresponding to the first judgment result on the to-be-inspected data to obtain target data corresponding to the to-be-inspected data; based on the target data and the corresponding distribution type, using an inspection tool to perform at least one distribution inspection corresponding to the distribution type on the target data to obtain a distribution inspection result corresponding to the target data; and performing threshold detection and nonparametric detection on the distribution detection result to obtain at least one detection probability value so as to determine a target distribution type corresponding to the to-be-detected data. Through the method, the efficiency and the accuracy of carrying out distribution inspection on the data are improved.

Description

Technical Field

[0001] The present application relates to the technical field of data statistical analysis, and in particular to a data distribution verification method, device, electronic device and storage medium. Background Art

[0002] Distribution test is a type of method used in statistics to determine whether a set of data follows a specific probability distribution. It generally includes conventional distribution test methods and methods based on machine learning or fitting methods based on specific distributions. In practical applications, it is necessary to select appropriate distribution test methods based on specific problems and data characteristics. At the same time, in order to draw more reliable conclusions, it is necessary to combine multiple distribution test methods for comprehensive data analysis.

[0003] At present, distribution tests on data are generally set up in statistical analysis systems. Distribution tests on data in such systems reduce test efficiency and reduce user experience. In addition, directly using data for distribution tests will produce test bias, so that the distribution test results have a significant impact on the statistical analysis results, reducing the accuracy of distribution tests on data. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a data distribution verification method, device, electronic device and storage medium, which configures at least one group of data to be tested and sets at least one distribution type for each group of data to be tested, performs data preprocessing operations such as outlier removal and non-negative processing on the data, thereby avoiding the problem of the inspection service refusing to accept data and the inspection result deviation caused by the serious deviation of the data from the mean. The processed data to be tested is subjected to distribution verification in a batch processing manner, and the distribution verification results are subjected to threshold detection and non-parametric testing to determine the target distribution type that the data to be tested is relatively close to, thereby improving the distribution verification efficiency of the data and thereby improving the accuracy of the distribution verification of the data.

[0005] The present invention provides a data distribution verification method, which includes:

[0006] Based on the preset distribution test configuration information, at least one group of data to be tested is determined in the preset data set, and at least one distribution type of distribution test is set for each group of the data to be tested;

[0007] For each group of the data to be tested, determine whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing, obtain a first determination result, and perform the data processing operation corresponding to the first determination result on the data to be tested, to obtain target data corresponding to the data to be tested;

[0008] Based on the target data and the corresponding distribution type, using a preset inspection tool to perform at least one distribution inspection corresponding to the distribution type on the target data to obtain at least one distribution inspection result corresponding to the target data;

[0009] Threshold detection and non-parametric test are performed on the distribution test results to obtain at least one test probability value, and the test probability values ​​are sorted according to the number to determine the target distribution type corresponding to the data to be tested.

[0010] Furthermore, the distribution inspection method also includes:

[0011] Based on the distribution test result and the target distribution type, a distribution test chart corresponding to the data to be tested is generated using a preset chart generation method to visualize the distribution characteristics and distribution relationships corresponding to the data to be tested.

[0012] Further, for each group of the data to be tested, determining whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing to obtain a first determination result, and performing the data processing operation corresponding to the first determination result on the data to be tested to obtain target data corresponding to the data to be tested, includes:

[0013] For each group of the data to be tested, determining whether the data to be tested contains outlier data;

[0014] If there are outlier data, a data processing operation of removing outliers is performed on the data to be tested to obtain the data to be processed corresponding to the data to be tested;

[0015] If there is no outlier data, the data to be tested is determined as data to be processed;

[0016] Determine whether at least one value of the data to be checked is a negative number;

[0017] If there is no negative value, the data to be processed is determined as the target data corresponding to the data to be checked;

[0018] If there is at least one negative value, a data processing operation of non-negative processing is performed on the data to be processed to obtain target data corresponding to the data to be tested.

[0019] Furthermore, the data processing operation of removing outliers from the data to be inspected to obtain the data to be processed corresponding to the data to be inspected includes:

[0020] Based on the mean and standard deviation corresponding to the data to be tested, a first boundary value and a second boundary value are determined respectively using a preset standard score expression to determine a first numerical interval consisting of the first boundary value and the second boundary value;

[0021] Determine whether each value in the to-be-tested data belongs to the first value interval, and obtain a second determination result;

[0022] Based on the second judgment result, the values ​​that do not belong to the first value interval are eliminated from the data to be checked to obtain the data to be processed corresponding to the data to be checked.

[0023] Furthermore, the data processing operation of removing outliers from the data to be inspected to obtain the data to be processed corresponding to the data to be inspected includes:

[0024] Based on the values ​​included in the data to be tested, determine the first quartile, the third quartile and the interquartile range corresponding to the data to be tested;

[0025] Based on the first quartile, the third quartile and the interquartile range, a third boundary value and a fourth boundary value are determined respectively using a preset interquartile range expression to determine a second numerical interval consisting of the third boundary value and the fourth boundary value;

[0026] Determine respectively whether each value in the to-be-tested data belongs to the second value interval, and obtain a third determination result;

[0027] Based on the third judgment result, the values ​​that do not belong to the second value interval are eliminated from the data to be checked to obtain the data to be processed corresponding to the data to be checked.

[0028] Furthermore, the data processing operation of performing non-negative processing on the data to be processed to obtain target data corresponding to the data to be tested includes:

[0029] Arrange the values ​​included in the data to be processed in ascending order according to the values, and determine a first value of the values ​​arranged in a first order based on the ascending order;

[0030] Adding each value included in the data to be processed to the absolute value of the first value to obtain accumulated data corresponding to the data to be processed;

[0031] Each value included in the accumulated data is added to a preset value to obtain target data corresponding to the data to be tested.

[0032] Further, the distribution test result is subjected to threshold detection and non-parametric test to obtain at least one test probability value, and the test probability values ​​are sorted according to the number to determine the target distribution type corresponding to the data to be tested, including:

[0033] Determine whether each of the distribution test results is greater than a preset threshold value to obtain a fourth determination result;

[0034] Based on the fourth judgment result, at least one target distribution test result greater than a preset threshold is screened out from the distribution test results, and a non-parametric test is performed on the target distribution test result to obtain at least one test probability value;

[0035] Arrange the test probability values ​​in descending order according to their numerical values, and determine a first test probability value arranged in a first order among the test probability values ​​based on the descending order;

[0036] A target distribution test result corresponding to the first test probability value is determined, and a distribution type corresponding to the target distribution test result is determined as the target distribution type corresponding to the data to be tested.

[0037] The present application also provides a data distribution verification device, the distribution verification device comprising:

[0038] A test configuration module, used to determine at least one group of data to be tested in a preset data set based on preset distribution test configuration information, and set at least one distribution type for each group of data to be tested that needs to be tested;

[0039] A data processing module is used to determine, for each group of the data to be tested, whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing, to obtain a first judgment result, and to perform the data processing operation corresponding to the first judgment result on the data to be tested, to obtain target data corresponding to the data to be tested;

[0040] A distribution inspection module, configured to perform at least one distribution inspection corresponding to the distribution type on the target data using a preset inspection tool based on the target data and the corresponding distribution type, and obtain at least one distribution inspection result corresponding to the target data;

[0041] The result analysis module is used to perform threshold detection and non-parametric test on the distribution test results to obtain at least one test probability value, and sort the test probability values ​​according to the quantity to determine the target distribution type corresponding to the data to be tested.

[0042] Furthermore, the distribution inspection device further includes a chart generation module, and the chart generation module is used to:

[0043] Based on the distribution test result and the target distribution type, a distribution test chart corresponding to the data to be tested is generated using a preset chart generation method to visualize the distribution characteristics and distribution relationships corresponding to the data to be tested.

[0044] Further, when the data processing module is used to determine, for each group of the data to be tested, whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing, to obtain a first judgment result, and to perform the data processing operation corresponding to the first judgment result on the data to be tested to obtain target data corresponding to the data to be tested, the data processing module is used to:

[0045] For each group of the data to be tested, determining whether the data to be tested contains outlier data;

[0046] If there are outlier data, a data processing operation of removing outliers is performed on the data to be tested to obtain the data to be processed corresponding to the data to be tested;

[0047] If there is no outlier data, the data to be tested is determined as data to be processed;

[0048] Determine whether at least one value of the data to be checked is a negative number;

[0049] If there is no negative value, the data to be processed is determined as the target data corresponding to the data to be checked;

[0050] If there is at least one negative value, a data processing operation of non-negative processing is performed on the data to be processed to obtain target data corresponding to the data to be tested.

[0051] Furthermore, when the data processing module is used to perform a data processing operation of removing outliers on the data to be inspected to obtain the data to be processed corresponding to the data to be inspected, the data processing module is used to:

[0052] Based on the mean and standard deviation corresponding to the data to be tested, a first boundary value and a second boundary value are determined respectively using a preset standard score expression to determine a first numerical interval consisting of the first boundary value and the second boundary value;

[0053] Determine whether each value in the to-be-tested data belongs to the first value interval, and obtain a second determination result;

[0054] Based on the second judgment result, the values ​​that do not belong to the first value interval are eliminated from the data to be checked to obtain the data to be processed corresponding to the data to be checked.

[0055] Furthermore, when the data processing module is used to perform a data processing operation of removing outliers on the data to be inspected to obtain the data to be processed corresponding to the data to be inspected, the data processing module is used to:

[0056] Based on the values ​​included in the data to be tested, determine the first quartile, the third quartile and the interquartile range corresponding to the data to be tested;

[0057] Based on the first quartile, the third quartile and the interquartile range, a third boundary value and a fourth boundary value are determined respectively using a preset interquartile range expression to determine a second numerical interval consisting of the third boundary value and the fourth boundary value;

[0058] Determine respectively whether each value in the to-be-tested data belongs to the second value interval, and obtain a third determination result;

[0059] Based on the third judgment result, the values ​​that do not belong to the second value interval are eliminated from the data to be checked to obtain the data to be processed corresponding to the data to be checked.

[0060] Furthermore, when the data processing module is used to perform a data processing operation of non-negative processing on the data to be processed to obtain target data corresponding to the data to be tested, the data processing module is used to:

[0061] Arrange the values ​​included in the data to be processed in ascending order according to the values, and determine a first value of the values ​​arranged in a first order based on the ascending order;

[0062] Adding each value included in the data to be processed to the absolute value of the first value to obtain accumulated data corresponding to the data to be processed;

[0063] Each value included in the accumulated data is added to a preset value to obtain target data corresponding to the data to be tested.

[0064] Furthermore, when the result analysis module is used to perform threshold detection and non-parametric test on the distribution test result to obtain at least one test probability value, and to sort the test probability values ​​according to the number of the test probability values ​​to determine the target distribution type corresponding to the data to be tested, the result analysis module is used to:

[0065] Determine whether each of the distribution test results is greater than a preset threshold value to obtain a fourth determination result;

[0066] Based on the fourth judgment result, at least one target distribution test result greater than a preset threshold is screened out from the distribution test results, and a non-parametric test is performed on the target distribution test result to obtain at least one test probability value;

[0067] Arrange the test probability values ​​in descending order according to their numerical values, and determine a first test probability value arranged in a first order among the test probability values ​​based on the descending order;

[0068] A target distribution test result corresponding to the first test probability value is determined, and a distribution type corresponding to the target distribution test result is determined as the target distribution type corresponding to the data to be tested.

[0069] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the data distribution verification method as described above are performed.

[0070] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data distribution verification method as described above are executed.

[0071] The data distribution verification method, device, electronic device and storage medium provided in the embodiments of the present application, the distribution verification method includes: based on preset distribution verification configuration information, determining at least one group of data to be tested in a preset data set, and setting at least one distribution type for each group of data to be tested that needs to be distributed verified; for each group of data to be tested, judging whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing to obtain a first judgment result, and performing the data processing operation corresponding to the first judgment result on the data to be tested to obtain target data corresponding to the data to be tested; based on the target data and the corresponding distribution type, performing at least one distribution test corresponding to the distribution type on the target data using a preset verification tool to obtain at least one distribution test result corresponding to the target data; performing threshold detection and non-parametric test on the distribution test result to obtain at least one test probability value, and determining the target distribution type corresponding to the data to be tested according to the number of the test probability values.

[0072] Compared with the prior art method of performing distribution test on data in a statistical analysis system which reduces the test efficiency and the method of directly using data for distribution test, by configuring at least one group of data to be tested and setting at least one distribution type for each group of data to be tested, the data is preprocessed by removing outliers and performing non-negative processing, thereby avoiding the problem of the test service refusing to accept data and the test result deviation caused by the serious deviation of the data from the mean. The processed data to be tested is subjected to distribution test in a batch processing manner, and the distribution test results are subjected to threshold detection and non-parametric test to determine the target distribution type that the data to be tested is relatively close to, thereby improving the distribution test efficiency of the data and further improving the accuracy of the distribution test on the data.

[0073] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 One of the flow charts of a data distribution verification method provided in an embodiment of the present application;

[0076] Figure 2 A second flowchart of a data distribution verification method provided in an embodiment of the present application;

[0077] Figure 3 One of the structural schematic diagrams of a data distribution inspection device provided in an embodiment of the present application;

[0078] Figure 4 A second structural diagram of a data distribution inspection device provided in an embodiment of the present application;

[0079] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0081] Research has found that distribution test is a type of method used in statistics to determine whether a set of data follows a specific probability distribution. It generally includes conventional distribution test methods and machine learning-based methods or fitting methods based on specific distributions. In practical applications, it is necessary to select appropriate distribution test methods based on specific problems and data characteristics. At the same time, in order to draw more reliable conclusions, it is necessary to combine multiple distribution test methods for comprehensive data analysis.

[0082] Among them, distribution test technology mainly exists as a function of large-scale statistical analysis software. Specifically, the data is distributed and tested through the following steps: first, prepare the data that may be generated in the statistical analysis system or imported into the statistical analysis system after being output from other systems; then, set the distribution type and other information to be tested; finally, perform a distribution test on the data and analyze the test results.

[0083] Regarding the above-mentioned distribution test steps, distribution test of data is generally set up in a statistical analysis system. Distribution test of data in this system reduces the test efficiency and reduces the user experience. In addition, directly using data for distribution test will produce test bias, so that the distribution test results have a significant impact on the statistical analysis results, reducing the accuracy of distribution test of data.

[0084] Based on this, an embodiment of the present application provides a method for data distribution verification, which performs data preprocessing operations of outlier removal and non-negative processing on the data by configuring at least one group of data to be verified and setting at least one distribution type for each group of data to be verified, thereby avoiding the problem of the verification service refusing to accept data and the deviation of the test results due to the serious deviation of the data from the mean. The processed data to be verified is subjected to distribution verification in a batch processing manner, and the distribution verification results are subjected to threshold detection and non-parametric testing to determine the target distribution type that the data to be verified is relatively close to, thereby improving the efficiency of the distribution verification of the data, and thereby improving the accuracy of the distribution verification of the data.

[0085] See also Figure 1 , Figure 1 This is one of the flow charts of a data distribution verification method provided in an embodiment of the present application. Figure 1 As shown in , the data distribution verification method provided by the embodiment of the present application includes:

[0086] S100. Based on preset distribution test configuration information, determine at least one group of data to be tested in a preset data set, and set at least one distribution type for distribution test required for each group of the data to be tested.

[0087] In an embodiment of the present application, the WinForms interface for data distribution verification is developed using the C# language and the backend data processing is developed using the Java language; wherein the C# language is used for constructing the user interface and drawing charts; and the Java language is used for data query and interface calling.

[0088] In this step, based on the distributed inspection configuration information pre-configured by the user according to the distributed inspection requirements, at least one group of data to be inspected corresponding to the distributed inspection configuration information is determined in a preset data set including multiple groups of data to be inspected.

[0089] Furthermore, based on the distribution check configuration information, at least one distribution type for which distribution check needs to be performed for each group of the data to be checked is set.

[0090] Here, the distribution type may include but is not limited to normal distribution, logarithmic full-day distribution, Gamma distribution, Poisson distribution and exponential distribution, etc.

[0091] S200. For each group of the data to be tested, determine whether the data to be tested needs to undergo data processing operations including outlier removal and / or non-negative processing to obtain a first judgment result, and perform the data processing operations corresponding to the first judgment result on the data to be tested to obtain target data corresponding to the data to be tested.

[0092] In an embodiment of the present application, outlier data in each group of data to be tested are eliminated, and existing negative numbers are non-negatively processed to avoid the problem of the test service refusing to accept data and the problem of incorrect test results due to some serious deviations from the mean.

[0093] The data processing operation for outlier removal may include two processing methods: standard score method and interquartile range method, which may be selected according to the actual situation of the data and the actual needs of the user.

[0094] Here, the data processing operation of the non-negative processing uses the translation value accumulation method to perform non-negative processing on the negative numbers existing in each group of data to be tested.

[0095] In one implementation of the present application, during specific implementation, step S200 may include:

[0096] S210: For each group of the data to be tested, determine whether the data to be tested contains outlier data.

[0097] In the embodiment of the present application, it is determined whether there are outlier data in each group of data to be tested, so as to exclude the influence of some obviously abnormal values ​​on the distribution performance of the data set.

[0098] In this step, a preset data identification tool (eg, a box plot, etc.) is used to identify outliers in each group of data to be tested, so as to determine whether there are outlier data in each group of data to be tested.

[0099] Here, the outlier data refers to data that is significantly different from other data, which may be caused by measurement errors, recording errors or other abnormal situations.

[0100] S220: If outlier data exists, a data processing operation of removing outliers is performed on the data to be checked to obtain data to be processed corresponding to the data to be checked.

[0101] In the embodiment of the present application, there are two methods for performing data processing operations to remove outliers on the data to be inspected, and the data processing operations to remove outliers can be performed by any one of the two methods.

[0102] Among them, one way is to determine the first numerical interval through a standard score expression, so as to eliminate the data that does not belong to the first numerical interval through regular standardization processing; another way is to determine the second numerical interval through the interquartile range and quartiles, so as to eliminate the data that does not belong to the second numerical interval through the interquartile range method.

[0103] In one embodiment of the present application, in a specific implementation, the step of performing a data processing operation of removing outliers on the data to be inspected in step S220 to obtain the data to be processed corresponding to the data to be inspected may include:

[0104] S2211. Based on the mean and standard deviation corresponding to the data to be tested, a first boundary value and a second boundary value are determined respectively using a preset standard score expression to determine a first numerical interval consisting of the first boundary value and the second boundary value.

[0105] In the examples of the present application, the standard score expression is as follows.

[0106] B1 = μ-n×σ.

[0107] B2=μ+n×σ.

[0108] Among them, B1 and B2 are the first boundary value and the second boundary value respectively; μ and σ are the mean and standard deviation corresponding to the data to be tested respectively; n is the standard score coefficient value (usually set to 3).

[0109] In this way, according to the first boundary value (B1) and the second boundary value (B2), the first numerical interval is determined to be (B1, B2).

[0110] S2212. Determine whether each value in the to-be-checked data belongs to the first value interval, and obtain a second determination result.

[0111] S2213. Based on the second judgment result, remove the values ​​that do not belong to the first numerical range from the data to be checked, and obtain the data to be processed corresponding to the data to be checked.

[0112] In this step, based on the second judgment result obtained by judging whether each value in the data to be tested belongs to the first value interval, the values ​​that do not belong to the first value interval in each group of data to be tested that need to be eliminated of outliers are eliminated to obtain the data to be processed corresponding to the data to be tested.

[0113] In one embodiment of the present application, in a specific implementation, the step of performing a data processing operation of removing outliers on the data to be inspected in step S220 to obtain the data to be processed corresponding to the data to be inspected may also include:

[0114] S2221. Based on the numerical values ​​included in the data to be tested, determine the first quartile, the third quartile and the interquartile range corresponding to the data to be tested.

[0115] In an embodiment of the present application, the first quartile refers to the lower quartile, that is, 25% of the values ​​included in the values ​​of the data to be tested correspond to values ​​lower than the lower quartile; the third quartile refers to the upper quartile, that is, 75% of the values ​​included in the values ​​of the data to be tested correspond to values ​​lower than the upper quartile; the interquartile range refers to the difference between the third quartile and the first quartile.

[0116] S2222. Based on the first quartile, the third quartile and the interquartile range, respectively determine the third boundary value and the fourth boundary value using a preset interquartile range expression to determine a second numerical interval consisting of the third boundary value and the fourth boundary value.

[0117] In the examples of the present application, the interquartile range expression is as follows.

[0118] B3 = Q1-m×IQR.

[0119] B4=Q3+m×IQR.

[0120] Among them, B3 and B4 are the third boundary value and the fourth boundary value respectively; Q1 and Q3 are the first quartile and the third quartile respectively; IQR is the interquartile range; m is the interquartile range coefficient value (usually set to 1.5).

[0121] In this way, according to the third boundary value (B3) and the fourth boundary value (B4), the second numerical interval is determined to be (B3, B4).

[0122] S2223. Determine whether each value in the to-be-checked data belongs to the second value interval, and obtain a third determination result.

[0123] S2224. Based on the third judgment result, remove the values ​​that do not belong to the second numerical range from the data to be tested, and obtain the data to be processed corresponding to the data to be tested.

[0124] In this step, based on the third judgment result obtained by judging whether each value in the data to be tested belongs to the second value interval, the values ​​that do not belong to the second value interval in each group of data to be tested that need to be eliminated of outliers are eliminated to obtain the data to be processed corresponding to the data to be tested.

[0125] S230: If there is no outlier data, the data to be checked is determined as data to be processed.

[0126] In this way, after obtaining or determining the data to be processed, it is further determined whether at least one value of the data to be processed is a negative value, so as to perform non-negative processing on the data to be processed.

[0127] S240: Determine whether there is at least one negative value in the data to be checked.

[0128] In the embodiment of the present application, if there is data with a negative value in the data to be checked, then the inspection service will refuse to accept the data when performing the distribution inspection.

[0129] For example, taking the lognormal distribution as an example, assuming that a set of data to be tested conforms to the lognormal distribution, it is necessary to calculate the logarithm of the data in the test data when performing a distribution test. However, the logarithm of data with negative values ​​cannot be calculated, which makes it impossible to perform a distribution test on the set of data to be tested.

[0130] In the embodiment of the present application, the data processing operation for non-negative processing is performed on the data to be processed corresponding to the test data. After the data to be processed is translated to a range where the minimum value starts from greater than 0, the data to be processed can be tested separately to solve the problem that numbers less than zero cannot be accepted and cannot be tested during the conversion process.

[0131] In this step, the minimum value in the data to be processed is determined. If the minimum value is less than 0, the data to be processed needs to be shifted to perform a data processing operation of non-negative processing.

[0132] S250: If there is no negative value, the data to be processed is determined as the target data corresponding to the data to be checked.

[0133] S260: If there is at least one negative value, a data processing operation of non-negative processing is performed on the data to be processed to obtain target data corresponding to the data to be tested.

[0134] In this step, the absolute value of the minimum value in the data to be processed is added to each data to be processed. At the same time, in order to achieve better stability, a non-negative preset value (for example, 0.1) is added to each data plus the absolute value of the minimum value in the data to be processed, so that the data in the data to be processed are shifted to the positive numerical range.

[0135] In one embodiment of the present application, in a specific implementation, the data processing operation of performing non-negative processing on the data to be processed in step S260 to obtain the target data corresponding to the data to be tested may include:

[0136] S261. Arrange the numerical values ​​included in the data to be processed in ascending order according to their numerical values, and determine a first numerical value arranged in a first order among the numerical values ​​based on the ascending order.

[0137] In this step, the data to be processed are sorted according to the size of the values, and the first value in the data to be processed is found, that is, the first value is the minimum value in the data to be processed.

[0138] S262: Add each value included in the data to be processed to the absolute value of the first value to obtain accumulated data corresponding to the data to be processed.

[0139] S263, adding each value included in the accumulated data to a preset value to obtain target data corresponding to the data to be tested.

[0140] In this step, the absolute value of the minimum value in the data to be processed is added to each data in the data to be processed to obtain the accumulated data corresponding to the data to be processed, and each value included in the accumulated data is added to the preset value to obtain the target data corresponding to the data to be tested.

[0141] In this way, after each group of data to be tested is processed by outlier removal and / or non-negative processing, target data is obtained, so as to further perform distribution test on the target data under the corresponding distribution type.

[0142] S300. Based on the target data and the corresponding distribution type, use a preset inspection tool to perform at least one distribution inspection corresponding to the distribution type on the target data to obtain at least one distribution inspection result corresponding to the target data.

[0143] In this step, the distribution test mainly performs the distribution test operations in the configuration content on the target data according to the distribution type of the distribution test configuration. For example, the two groups of target data are respectively subjected to the normal distribution, logarithmic full-day distribution and gamma distribution distribution tests. Then, according to this configuration, the preset test tools are used to perform the above three distribution tests on the group target data to obtain the three distribution test results corresponding to each group of target data.

[0144] Here, the preset test tool may include a statistical test tool under the R language platform, such as nortest and fitdistrplus, etc.; the preset test tool is a lightweight tool that can be used as a separate software or embedded in other systems.

[0145] S400, performing threshold detection and non-parametric test on the distribution test result to obtain at least one test probability value, and determining the target distribution type corresponding to the data to be tested according to the number of the test probability values.

[0146] In the embodiment of the present application, the threshold detection is used to determine whether the distribution test result meets the distribution test requirements corresponding to the distribution type.

[0147] Here, at least one test probability value is obtained through a non-parametric test to determine the actual fit of the distribution test result under the distribution type.

[0148] Among them, the non-parametric test (Kolmogorov-Smirnov, KS test) is a non-parametric test method used to compare the cumulative distribution function (CDF) of a sample with a reference probability distribution (single-sample KS test), and the test probability value (P value) represents the probability of observing the current sample data or more extreme situations when the null hypothesis (that is, the population distribution from which the sample comes is the same as the reference distribution, or the two samples come from the same population distribution) is true.

[0149] In one implementation of the present application, during specific implementation, step S400 may include:

[0150] S410, respectively determine whether each distribution test result in the distribution test results is greater than a preset threshold, and obtain a fourth determination result.

[0151] In this step, the distribution test results of at least one distribution type corresponding to each group of target data are compared with the preset threshold value to obtain a fourth judgment result of whether each distribution test result is greater than the preset threshold value.

[0152] The preset threshold value may be specifically calibrated according to the distribution requirements of the distribution type and the actual needs of the distribution test.

[0153] S420. Based on the fourth judgment result, at least one target distribution test result greater than a preset threshold is screened out from the distribution test results, and a non-parametric test is performed on the target distribution test result to obtain at least one test probability value.

[0154] In this step, at least one target distribution test result greater than a preset threshold is screened out from the distribution test results under at least one distribution type, and a non-parametric test is performed on the target distribution test result to obtain at least one test probability value; or a test probability value is determined in the target distribution test results where a non-parametric test exists.

[0155] S430, arranging the inspection probability values ​​in descending order according to numerical values, and determining a first inspection probability value arranged in a first order among the inspection probability values ​​based on the descending order.

[0156] In this step, a maximum value is determined among at least one test probability value, so as to determine the maximum value as a first test probability value.

[0157] S440: Determine a target distribution test result corresponding to the first test probability value, and determine a distribution type corresponding to the target distribution test result as a target distribution type corresponding to the data to be tested.

[0158] In this step, the target distribution test result corresponding to the first test probability value is determined as a distribution type that is closest to the distribution type corresponding to the target distribution test result, and the distribution type is determined as the target distribution type corresponding to the data to be tested.

[0159] Optional, see Figure 2 , Figure 2 This is a second flow chart of a data distribution verification method provided in an embodiment of the present application. Figure 2 As shown in, in addition to the data distribution verification method described in steps S100 to S400, step S500 is also included. Specifically, step S500 is used to illustrate a method for visualizing the distribution characteristics and distribution relationships of the target distribution type based on the target data to improve the user experience.

[0160] S500. Based on the distribution test result and the target distribution type, a distribution test chart corresponding to the data to be tested is generated using a preset chart generation method to visualize the distribution characteristics and distribution relationship corresponding to the data to be tested.

[0161] In an embodiment of the present application, the preset chart generation method may include but is not limited to maximum likelihood estimation and least squares method, etc.; the distribution test chart may include but is not limited to PP chart (Probability-Probability Plot) and QQ chart (Quantile-Quantile Plot), etc.

[0162] In this step, in the specific implementation, first, based on the target data, the distribution test results and the target distribution type, the preset linear regression model is used to fit the data to perform the least squares (OLS) fitting; then, the parameters are estimated by maximizing the likelihood function to perform the maximum likelihood estimation (MLE) fitting; finally, the data points of the target data and the two fitting lines are plotted on the same chart to generate a distribution test chart corresponding to the data to be tested, so as to visualize the distribution characteristics and distribution relationships corresponding to the data to be tested.

[0163] The data distribution verification method provided in the embodiment of the present application configures at least one group of data to be verified and sets at least one distribution type for each group of data to be verified, performs data preprocessing operations of outlier removal and non-negative processing on the data, thereby avoiding the problem of the verification service refusing to accept data and the deviation of the test results due to the serious deviation of the data from the mean. The processed data to be verified are subjected to distribution verification in a batch processing manner, and the distribution verification results are subjected to threshold detection and non-parametric test to determine the target distribution type that the data to be verified is relatively close to, thereby improving the distribution verification efficiency of the data and further improving the accuracy of the distribution verification of the data.

[0164] See also Figure 3 , Figure 4 , Figure 3 This is one of the structural schematic diagrams of a data distribution inspection device provided in an embodiment of the present application. Figure 4 This is a second structural diagram of a data distribution inspection device provided in an embodiment of the present application. Figure 3 As shown in , the distribution inspection device 300 includes:

[0165] The inspection configuration module 310 is used to determine at least one group of data to be inspected in the preset data set based on the preset distribution inspection configuration information, and set at least one distribution type of distribution inspection required for each group of the data to be inspected;

[0166] The data processing module 320 is used to determine, for each group of the data to be tested, whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing, obtain a first judgment result, and perform the data processing operation corresponding to the first judgment result on the data to be tested to obtain target data corresponding to the data to be tested;

[0167] A distribution inspection module 330 is used to perform at least one distribution inspection corresponding to the distribution type on the target data based on the target data and the corresponding distribution type using a preset inspection tool to obtain at least one distribution inspection result corresponding to the target data;

[0168] The result analysis module 340 is used to perform threshold detection and non-parametric test on the distribution test results to obtain at least one test probability value, and sort the test probability values ​​according to the number to determine the target distribution type corresponding to the data to be tested.

[0169] Further, such as Figure 4 As shown in , the distribution inspection device 300 further includes a chart generating module 350, and the chart generating module 350 is used to:

[0170] Based on the distribution test result and the target distribution type, a distribution test chart corresponding to the data to be tested is generated using a preset chart generation method to visualize the distribution characteristics and distribution relationships corresponding to the data to be tested.

[0171] Further, when the data processing module 320 is used to determine, for each group of the data to be tested, whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing, to obtain a first judgment result, and to perform the data processing operation corresponding to the first judgment result on the data to be tested to obtain target data corresponding to the data to be tested, the data processing module 320 is used to:

[0172] For each group of the data to be tested, determining whether the data to be tested contains outlier data;

[0173] If there are outlier data, a data processing operation of removing outliers is performed on the data to be tested to obtain the data to be processed corresponding to the data to be tested;

[0174] If there is no outlier data, the data to be tested is determined as data to be processed;

[0175] Determine whether at least one value of the data to be checked is a negative number;

[0176] If there is no negative value, the data to be processed is determined as the target data corresponding to the data to be checked;

[0177] If there is at least one negative value, a data processing operation of non-negative processing is performed on the data to be processed to obtain target data corresponding to the data to be tested.

[0178] Furthermore, when the data processing module 320 is used to perform a data processing operation of removing outliers on the data to be inspected and obtain the data to be processed corresponding to the data to be inspected, the data processing module is used to:

[0179] Based on the mean and standard deviation corresponding to the data to be tested, a first boundary value and a second boundary value are determined respectively using a preset standard score expression to determine a first numerical interval consisting of the first boundary value and the second boundary value;

[0180] Determine whether each value in the to-be-tested data belongs to the first value interval, and obtain a second determination result;

[0181] Based on the second judgment result, the values ​​that do not belong to the first value interval are eliminated from the data to be checked to obtain the data to be processed corresponding to the data to be checked.

[0182] Furthermore, when the data processing module 320 is used to perform a data processing operation of removing outliers on the data to be inspected to obtain the data to be processed corresponding to the data to be inspected, the data processing module 320 is used to:

[0183] Based on the values ​​included in the data to be tested, determine the first quartile, the third quartile and the interquartile range corresponding to the data to be tested;

[0184] Based on the first quartile, the third quartile and the interquartile range, a third boundary value and a fourth boundary value are determined respectively using a preset interquartile range expression to determine a second numerical interval consisting of the third boundary value and the fourth boundary value;

[0185] Determine respectively whether each value in the to-be-tested data belongs to the second value interval, and obtain a third determination result;

[0186] Based on the third judgment result, the values ​​that do not belong to the second value interval are eliminated from the data to be checked to obtain the data to be processed corresponding to the data to be checked.

[0187] Furthermore, when the data processing module 320 is used to perform a data processing operation for non-negative processing on the data to be processed to obtain target data corresponding to the data to be tested, the data processing module 320 is used to:

[0188] Arrange the values ​​included in the data to be processed in ascending order according to the values, and determine a first value of the values ​​arranged in a first order based on the ascending order;

[0189] Adding each value included in the data to be processed to the absolute value of the first value to obtain accumulated data corresponding to the data to be processed;

[0190] Each value included in the accumulated data is added to a preset value to obtain target data corresponding to the data to be tested.

[0191] Further, when the result analysis module 340 is used to perform threshold detection and non-parametric test on the distribution test result to obtain at least one test probability value, and to sort the test probability values ​​according to the number of the test probability values ​​to determine the target distribution type corresponding to the data to be tested, the result analysis module 340 is used to:

[0192] Determine whether each of the distribution test results is greater than a preset threshold value to obtain a fourth determination result;

[0193] Based on the fourth judgment result, at least one target distribution test result greater than a preset threshold is screened out from the distribution test results, and a non-parametric test is performed on the target distribution test result to obtain at least one test probability value;

[0194] Arrange the test probability values ​​in descending order according to their numerical values, and determine a first test probability value arranged in a first order among the test probability values ​​based on the descending order;

[0195] A target distribution test result corresponding to the first test probability value is determined, and a distribution type corresponding to the target distribution test result is determined as the target distribution type corresponding to the data to be tested.

[0196] The data distribution inspection device provided in the embodiment of the present application configures at least one group of data to be inspected and sets at least one distribution type for each group of data to be inspected, performs data preprocessing operations of outlier removal and non-negative processing on the data, thereby avoiding the problem of the inspection service refusing to accept data and the inspection result deviation caused by the serious deviation of the data from the mean. The distribution inspection is performed on the processed data to be inspected in a batch processing manner, and the distribution inspection results are subjected to threshold detection and non-parametric testing to determine the target distribution type that the data to be inspected is relatively close to, thereby improving the distribution inspection efficiency of the data and further improving the accuracy of the distribution inspection on the data.

[0197] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5As shown in , the electronic device 500 includes a processor 510 , a memory 520 and a bus 530 .

[0198] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 as well as Figure 2 The steps of the data distribution verification method in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0199] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The steps of the data distribution verification method in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0200] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0201] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0202] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0203] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0204] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0205] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data distribution inspection method, characterized in that: The distribution inspection method comprises: Based on the preset distribution test configuration information, at least one group of data to be tested is determined in the preset data set, and at least one distribution type of distribution test is set for each group of the data to be tested; For each group of the to-be-tested data, determine whether the to-be-tested data needs to be subjected to a data processing operation including outlier removal and / or non-negative processing, obtain a first determination result, and perform the data processing operation corresponding to the first determination result on the to-be-tested data to obtain target data corresponding to the to-be-tested data; Based on the target data and the corresponding distribution type, using a preset inspection tool to perform at least one distribution inspection corresponding to the distribution type on the target data to obtain at least one distribution inspection result corresponding to the target data; Threshold detection and non-parametric test are performed on the distribution test results to obtain at least one test probability value, and the test probability values ​​are sorted according to the number to determine the target distribution type corresponding to the data to be tested.

2. The method according to claim 1, characterized in that The distribution inspection method further comprises: Based on the distribution test result and the target distribution type, a distribution test chart corresponding to the data to be tested is generated using a preset chart generation method to visualize the distribution characteristics and distribution relationships corresponding to the data to be tested.

3. The method according to claim 1, characterized in that The step of determining, for each group of the data to be tested, whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing to obtain a first determination result, and performing the data processing operation corresponding to the first determination result on the data to be tested to obtain target data corresponding to the data to be tested includes: For each group of the data to be tested, determining whether the data to be tested contains outlier data; If there are outlier data, a data processing operation of removing outliers is performed on the data to be tested to obtain the processed data corresponding to the data to be tested; If there is no outlier data, the data to be tested is determined as data to be processed; Determine whether at least one value of the data to be checked is a negative number; If there is no negative value, the data to be processed is determined as the target data corresponding to the data to be checked; If there is at least one negative value, a data processing operation of non-negative processing is performed on the data to be processed to obtain target data corresponding to the data to be tested.

4. The method according to claim 3, characterized in that The data processing operation of removing outliers from the data to be inspected to obtain the data to be processed corresponding to the data to be inspected includes: Based on the mean and standard deviation corresponding to the data to be tested, a first boundary value and a second boundary value are determined respectively using a preset standard score expression to determine a first numerical interval consisting of the first boundary value and the second boundary value; Determine whether each value in the to-be-tested data belongs to the first value interval, and obtain a second determination result; Based on the second judgment result, the values ​​that do not belong to the first value interval are eliminated from the data to be checked to obtain the data to be processed corresponding to the data to be checked.

5. The method according to claim 3, characterized in that: The data processing operation of removing outliers from the data to be inspected to obtain the data to be processed corresponding to the data to be inspected includes: Based on the values ​​included in the data to be tested, determine the first quartile, the third quartile and the interquartile range corresponding to the data to be tested; Based on the first quartile, the third quartile and the interquartile range, a third boundary value and a fourth boundary value are determined respectively using a preset interquartile range expression to determine a second numerical interval consisting of the third boundary value and the fourth boundary value; Determine respectively whether each value in the to-be-tested data belongs to the second value interval, and obtain a third determination result; Based on the third judgment result, the values ​​that do not belong to the second value interval are eliminated from the data to be checked to obtain the data to be processed corresponding to the data to be checked.

6. The method according to claim 3, characterized in that The data processing operation of performing non-negative processing on the data to be processed to obtain target data corresponding to the data to be tested includes: Arrange the values ​​included in the data to be processed in ascending order according to the magnitude of the values, and determine a first value of the values ​​arranged in a first order based on the ascending order; Adding each value included in the data to be processed to the absolute value of the first value to obtain accumulated data corresponding to the data to be processed; Each value included in the accumulated data is added to a preset value to obtain target data corresponding to the data to be tested.

7. The method according to claim 1, characterized in that The step of performing threshold detection and non-parametric testing on the distribution test results to obtain at least one test probability value, and determining the target distribution type corresponding to the data to be tested according to the number of the test probability values, includes: Determine whether each of the distribution test results is greater than a preset threshold value to obtain a fourth determination result; Based on the fourth judgment result, at least one target distribution test result greater than a preset threshold is screened out from the distribution test results, and a non-parametric test is performed on the target distribution test result to obtain at least one test probability value; Arrange the test probability values ​​in descending order according to their numerical values, and determine a first test probability value arranged in a first order among the test probability values ​​based on the descending order; A target distribution test result corresponding to the first test probability value is determined, and a distribution type corresponding to the target distribution test result is determined as the target distribution type corresponding to the data to be tested.

8. A data distribution verification device, characterized in that: The distribution inspection device comprises: A test configuration module, used to determine at least one group of data to be tested in a preset data set based on preset distribution test configuration information, and set at least one distribution type for each group of data to be tested that needs to be tested; A data processing module is used to determine, for each group of the data to be tested, whether the data to be tested needs to be subjected to a data processing operation including outlier removal and / or non-negative processing, to obtain a first judgment result, and to perform the data processing operation corresponding to the first judgment result on the data to be tested, to obtain target data corresponding to the data to be tested; A distribution inspection module, configured to perform at least one distribution inspection corresponding to the distribution type on the target data using a preset inspection tool based on the target data and the corresponding distribution type, and obtain at least one distribution inspection result corresponding to the target data; The result analysis module is used to perform threshold detection and non-parametric test on the distribution test results to obtain at least one test probability value, and sort the test probability values ​​according to the quantity to determine the target distribution type corresponding to the data to be tested.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the data distribution verification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data distribution verification method according to any one of claims 1 to 7 are executed.

Images