A data validity verification method and system for a big data platform
By analyzing the data to be detected and historical data in the big data platform, combining the attribute degree and degree of difference of attribute values, and calculating the data validity score, the problem of difficulty in adapting to data patterns based on rule verification in the existing technology is solved, and data calibration is improved accuracy and reliability of the test.
Patent Information
- Application Number
- CN202411261838.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-10
AI Technical Summary
The data validity verification methods of existing big data platforms mainly rely on rule-based verification, making it difficult to dynamically adapt to potential patterns in the data, resulting in large errors in the verification results.
By reading the data to be detected and historical data of various attribute types, determining key attributes, analyzing the distribution and randomness of the data on key attributes, combining the attribute value attributes and degree of difference, calculating the data validity score to achieve data validity check.
It improves the efficiency and accuracy of data validity verification, reduces the risk of error caused by changes in data sources, and enhances the reliability of judgment.
Smart Images

Figure CN118797387B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of big data analysis and processing, and in particular to a data validity verification method and system for a big data platform. Background Art
[0002] As a platform for enterprises to conduct analysis and decision-making in a big data environment, the big data platform can help enterprises analyze massive data sets, and validity verification can avoid deviations in analysis results or decision-making errors caused by data errors, ensure the validity of the big data platform data set, and improve the efficiency of data management.
[0003] At present, the data validity verification method mainly adopts rule-based verification method, which requires a large number of preset rules for judgment. Due to the characteristics of multi-dimensional and large sample size of data on big data platforms, the rule-based verification method has the problem that the preset rules are difficult to dynamically adapt to the potential data patterns contained in the data, resulting in large errors in the results of data validity verification. Summary of the invention
[0004] In order to solve the above technical problems, the purpose of this application is to provide a data validity verification method and system for a big data platform. The technical solutions adopted are as follows:
[0005] In a first aspect, an embodiment of the present application provides a data validity verification method for a big data platform, the method comprising the following steps:
[0006] Read the data to be tested and historical data of various attribute types;
[0007] Determining key attributes in all attribute types based on the distribution of the historical data of various attribute types;
[0008] Analyze the distribution of the data to be detected and the historical data in each key attribute to obtain the attribute value attribution degree of each data to be detected on the corresponding key attribute; analyze the distribution of the historical data of each key attribute to obtain the value randomness of each key attribute; combine the attribute value attribution degree and the value randomness to obtain the attribute value validity degree of each data to be detected on the corresponding key attribute;
[0009] Analyze the difference between the distribution of the data to be tested and the distribution of historical data on each key attribute to obtain the data difference degree of all the data to be tested under each key attribute; combine the validity degree of the attribute value with the data difference degree to obtain the validity score of all the data to be tested under each key attribute, and obtain the data validity verification result.
[0010] In one embodiment, the key attribute is determined as follows:
[0011] The richness of attribute details of each attribute type is determined based on the degree of chaos of the distribution of all historical data of various attribute types; a clustering algorithm is used to cluster the richness of attribute details of all attribute types to obtain clusters, and the key attributes of all attribute types are obtained based on the clusters.
[0012] In one embodiment, the attribute detail richness is the information entropy of all historical data of each attribute type.
[0013] In one embodiment, the key attributes are all attribute types corresponding to the cluster with the maximum attribute detail richness mean value.
[0014] In one embodiment, the process of determining the attribute value attribution degree is as follows:
[0015] The value range of all historical data of each key attribute is evenly divided into a preset number of value intervals, and the ratio of the number of historical data in each value interval of each key attribute to the number of all historical data is determined as the interval projection quantity ratio of each value interval of each key attribute;
[0016] If the data to be detected is in the value interval, the sum of the interval projection quantity ratio of the corresponding value interval and the preset value is used as the attribute value attribution degree of the data to be detected; otherwise, the preset value is used as the attribute value attribution degree of the data to be detected.
[0017] In one embodiment, the process of determining the randomness of the value is:
[0018] All the interval projection number ratios of each key attribute are combined into a projection number distribution sequence, the concentration degree of the projection number distribution sequence is calculated, and the difference between the value 1 and the concentration degree is used as the value randomness of each key attribute.
[0019] In one embodiment, the process of determining the degree of data difference is as follows:
[0020] The ratio of the number of data to be detected in each value interval of each key attribute to the number of all data to be detected is calculated and recorded as the first ratio. All the first ratios of each key attribute are combined into a distribution sequence of data to be checked. The measured distance between the distribution sequence of data to be checked and the projection number distribution sequence of each key attribute is determined as the data difference degree of all data to be detected under each key attribute.
[0021] In one embodiment, the validity score is a normalized result of the product of the data difference degree of each key attribute and the validity degree of the attribute value.
[0022] In one embodiment, the data validity check result is:
[0023] If the validity score of all the data to be tested under the key attribute is greater than the preset threshold, then all the data to be tested under the corresponding key attribute is valid data; otherwise, all the data to be tested under the corresponding key attribute is invalid data, and the data to be tested except all the data to be tested under all the key attributes is recorded as valid data.
[0024] In a second aspect, an embodiment of the present application also provides a data validity verification system for a big data platform, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above-mentioned methods when executing the computer program.
[0025] This application has at least the following beneficial effects:
[0026] The present application reads the data to be detected and historical data of various attribute types, obtains the attribute detail richness of each attribute type based on the distribution of the historical data of various attribute types, obtains the key attributes of all attribute types based on the attribute detail richness, and screens all attribute types, thereby improving the efficiency of data validity verification; spatially maps the data to be detected and historical data of all key attributes, analyzes the distribution of the data to be detected and historical data in the spatial mapping results, and obtains the attribute value attribution of each data to be detected on the corresponding key attribute. The beneficial effect is to deeply mine the characteristics of historical data, avoid the interference of the discrete data of the historical data itself on the value of the data to be detected, and improve the accuracy of subsequent data validity analysis; analyzes the distribution of the historical data of each key attribute in the spatial mapping results, and obtains the value of each key attribute. randomness of values; combining the attribute value attribution degree with the value randomness, the validity degree of the attribute value of each data to be detected on the corresponding key attribute is obtained, which reflects the possibility that the data to be detected is valid data under the corresponding key attribute, and improves the reliability of the validity judgment of the data to be detected; analyzing the difference between the distribution of the data to be detected on each key attribute and the distribution of historical data, and obtaining the data difference degree of all the data to be detected under each key attribute; combining the attribute value validity degree with the data difference degree, obtaining the validity score of all the data to be detected under each key attribute, obtaining the data validity verification result, reducing the risk of errors in data validity judgment due to changes in data sources, and improving the accuracy of data validity verification by analyzing the distribution characteristics of a large amount of data to be detected and the comparative relationship with historical data. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0028] Figure 1 A flowchart of the steps of a data validity verification method for a big data platform provided in this application;
[0029] Figure 2 It is a schematic diagram of the ratio of the sequence number of the value interval to the number of interval projections;
[0030] Figure 3 Schematic diagram for comparing the distribution sequence of the data to be verified and the distribution sequence of the projection number. DETAILED DESCRIPTION
[0031] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following is a detailed description of the data validity verification method and system of a big data platform proposed by the present application, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0032] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
[0033] The following is a detailed description of a data validity verification method and system for a big data platform provided by the present application in conjunction with the accompanying drawings.
[0034] See also Figure 1 , which shows a flowchart of a method for verifying data validity on a big data platform provided by an embodiment of the present application, the method comprising the following steps:
[0035] Step S001, reading medical data of various attribute types and performing pre-processing.
[0036] This embodiment reads medical data of various attribute types of all hospitals in the municipal area from the database of the medical big data platform, including medical data that has undergone data validity verification, which is recorded as historical data, and medical data that has not undergone data validity verification, which is recorded as data to be tested.
[0037] In this embodiment, the attribute types include: entity type, number of beds, number of doctors, quarterly time, quarterly prescription volume, quarterly sales volume, and drug name; among them, the entity types include community health centers, township health centers, county hospitals, and core hospitals; the quarterly time is year + {first quarter, second quarter, third quarter, fourth quarter}, the quarterly prescription volume is the number of all prescriptions issued by the hospital in a quarter, the quarterly sales volume is the total amount of drugs sold by the hospital in a quarter, and the drug name is the names of all drugs contained in the hospital.
[0038] Data preprocessing operations are performed on the data to be detected and the historical data respectively, and the data preprocessing operations include data logic inspection and data repeatability inspection, wherein the data logic inspection and data repeatability inspection are both existing publicly known technologies, and this application will not be described in detail here. For the numerical data in the preprocessed data to be detected and the historical data, Z-Score normalization is used to eliminate the dimension, and for the non-numeric data in the preprocessed data to be detected and the historical data, the data is converted into decimal data through the Unicode encoding method, and then the Z-Score algorithm is used for normalization to obtain all the normalized data to be detected and the historical data.
[0039] Step S002, based on the distribution of historical data of various attribute types, obtain the attribute detail richness of each attribute type; obtain key attributes in all attribute types based on the attribute detail richness; and spatially map the data to be detected and the historical data of all key attributes.
[0040] In the big data platform, attribute types are used as a measure of a subject in a certain direction. In this embodiment, the subject is the hospital, which provides data support for platform management and decision-making. Different attribute types have different contributions to analytical decision-making. Attribute types with higher richness often contain a large amount of data details and are more likely to be key attributes for analytical decision-making.
[0041] First, the information entropy of all historical data under each attribute type is calculated as the attribute detail richness of each attribute type, which reflects the richness of the information contained in the attribute type. Secondly, this application takes the attribute detail richness of all attribute types as input, and adopts the K-means clustering algorithm. The number of clusters K in this embodiment is 2, and the implementer can set it according to the actual situation. This application does not limit this. The cluster cluster with the largest mean attribute detail richness in the output result is recorded as the key attribute class. All attribute types in the key attribute class are regarded as key attributes. The K-means clustering algorithm is an existing well-known calculation, and this application will not be described in detail here. The implementer can choose other existing feasible clustering algorithms according to the actual situation, and this application does not limit this.
[0042] Due to the comprehensiveness of the data source and the large sample size, there is a correlation between the historical data and the data to be detected in the big data platform. In this embodiment, each key attribute is used as a feature dimension, and the data to be detected and historical data under all key attributes are mapped to the corresponding feature dimension, and the obtained spatial mapping result is used as a feature space.
[0043] The stronger the validity of the data to be tested is, the more similar the key attributes of the data to be tested and the historical data should be. The more likely the key attributes of the data to be tested are consistent with the key attributes of the historical data, the more likely the data attribute values of the data to be tested are valid data.
[0044] Step S003, analyze the distribution of the data to be detected and the historical data in the feature space to obtain the attribute value attribution degree of each data to be detected on the corresponding key attribute; analyze the distribution of the historical data of each key attribute in the feature space to obtain the value randomness of each key attribute; combine the attribute value attribution degree and the value randomness to obtain the attribute value validity degree of each data to be detected on the corresponding key attribute.
[0045] This embodiment takes the key attribute j as an example to obtain all projection points of the historical data on the key attribute j in the feature space. The value range of the historical data on the key attribute j is evenly divided into a preset number of value intervals. The preset number is 10 in this embodiment, and the implementer can set it according to the actual situation. This application does not impose any restrictions on this. The ratio of the number of projection points of the historical data in each value interval to the number of all historical data is counted as the ratio of the number of interval projections on the key attribute j for each value interval, and the projection number distribution sequence of the historical data on the key attribute j is formed according to the results of the ascending sorting of the value intervals. The schematic diagram of the ratio of the sequence number of the value interval to the number of interval projections is shown in the figure. Figure 2 shown.
[0046] Based on the above analysis, this embodiment calculates the attribute value attribution degree of the to-be-detected data a on the key attribute j :
[0047] ; In the formula, is the attribute value attribution degree of the detected data a on the key attribute j, is the value of the key attribute j of the data to be detected a, is the value range of historical data on key attribute j, is the ratio of the number of interval projections of the value interval of the key attribute j in the data a to be detected. A preset value greater than 0.
[0048] when , indicating that the value of the key attribute j of the data to be tested a belongs to the value range of the historical data. The more consistent the value of the key attribute of the data to be tested is with that of the key attribute of the historical data, the higher the degree of attribute value belonging. The higher it is, the more likely it is a valid key attribute value; when , indicating that the value of the key attribute j of the data to be detected a does not belong to the value range of the historical data, and the degree of attribution of the attribute value Pick , the preset value in this embodiment , implementers can set it according to actual conditions, and this application does not impose any restrictions on this.
[0049] In the big data platform, there may be both invalid data and valid data in the data to be tested, while all historical data are valid data. For key attribute j with a large concentration of values, the value of the key attribute j of the data to be tested should be closer to the value of the key attribute j of the historical data, so that the randomness of the value of the key attribute of the data to be tested is lower.
[0050] Based on the above analysis, this embodiment can obtain the randomness of the value of the key attribute j: :
[0051] ; In the formula, is the randomness of the value of key attribute j, is the Herfindahl index of the projection number distribution sequence of the key attribute j, representing the concentration of the projection number distribution sequence. The calculation of the Herfindahl index is a well-known technology, and this application will not be described in detail here.
[0052] In another embodiment, the Herfindahl index may be replaced by an entropy index to calculate the randomness of values.
[0053] When the concentration of the projection number distribution sequence is lower, that is, The larger the value is, the lower the concentration of the key attribute j of the historical data is, and the more likely the value of the key attribute j of the data to be tested is to be highly random. The higher.
[0054] This embodiment takes key attribute j as an example, and obtains the validity of the attribute value of the key attribute j of the entity data to be verified based on the attribute value attribution degree of the key attribute j and the value randomness of the key attribute j. Specifically, the product of the attribute value attribution degree and the value randomness of the key attribute j is calculated as the validity of the attribute value of the data to be detected on the key attribute j.
[0055] The greater the randomness of the key attribute j, the greater the error in the attribute value attribution degree of the key attribute j of the data to be tested, and the higher the attribute value validity. The greater the attribute value validity, the more the key attribute value of the data to be tested conforms to the value distribution characteristics of the historical data, and the more likely it is to be valid key attribute data.
[0056] Step S004, analyzing the difference between the distribution of the data to be tested and the distribution of the historical data on each key attribute to obtain the data difference degree of each key attribute; combining the validity degree of the attribute value and the data difference degree to obtain the validity score of each data to be tested on the corresponding key attribute, and obtaining the data validity verification result.
[0057] In a big data platform, when the data source undergoes significant changes, the key attribute values of the overall data to be tested are highly deviated from the key attribute values of the historical data. These data to be tested are often actually beneficial for analytical decision-making, and help the big data platform gain insight into the underlying rules of changes in data sources. They are valid data, but there are a large number of key attribute values of the data to be tested that are too low in effectiveness.
[0058] In this embodiment, in each value interval of each key attribute, the ratio of the number of projection points of the data to be detected to the number of all the data to be detected is calculated, recorded as the first ratio, and all the first ratios of each key attribute are sorted in ascending order according to the value interval to form a distribution sequence of the data to be detected under each key attribute. A schematic diagram of the comparison between the distribution sequence of the data to be detected and the distribution sequence of the number of projections under a certain key attribute is shown in FIG. Figure 3 As shown, Figure 3 The middle oblique line fill represents the first ratio of the data to be detected, and the pure gray fill represents the interval projection quantity ratio of the historical data.
[0059] Taking key attribute j as an example, based on the overall value difference between the data to be tested and the historical data under key attribute j, the data difference degree of all the data to be tested under key attribute j can be obtained. :
[0060] ; In the formula, is the degree of data difference of all the data to be tested under the key attribute j, is the distribution sequence of the projection number of all historical data under the key attribute j, is the distribution sequence of all the data to be tested under the key attribute j, It is a function for calculating the EMD (Earth Mover's Distance) distance.
[0061] In another embodiment, the EMD distance may be replaced by the Mahalanobis distance to calculate the degree of data difference.
[0062] The greater the difference between the projection number distribution sequence and the distribution sequence of the data to be verified, that is, The larger the value is, the greater the overall difference between the data to be tested and the historical data is, and the more significantly the data to be tested is affected by the change in data source. The larger it is, the more we should calibrate the validity of the attribute value in order to avoid the interference of changes in the data source itself on the validity of the attribute value.
[0063] The same calculation method as that for the data difference degree under key attribute j is adopted to obtain the data difference degree of all the data to be detected under each key attribute.
[0064] Based on the data difference degree and the validity degree of the attribute value, the validity of the data to be tested under each key attribute is corrected, specifically: the normalized result of the product of the data difference degree and the validity degree of the attribute value of each key attribute is calculated as the validity score of all the data to be tested under each key attribute. The method for obtaining the normalized result here is to use the Sigmoid function. The implementer can choose other normalization methods according to the actual situation, and this application does not limit this.
[0065] For all the data to be tested under each key attribute, if the corresponding validity score is greater than the preset threshold, all the data to be tested under the corresponding key attribute will be judged as valid data, otherwise, all the data to be tested under the corresponding key attribute will be judged as invalid data, and the invalid data will be removed from the database of the big data platform. In this implementation, the preset threshold is 0.8, and the implementer can set it according to the actual situation. This application does not impose any restrictions on this. For all the data to be tested under non-key attributes, due to their low richness, it means that there has been no significant change compared with historical data, and they may be fixed data in a short period of time. Therefore, all the data to be tested under non-key attributes are regarded as valid data.
[0066] Based on the same inventive concept as the above method, an embodiment of the present application also provides a data validity verification system for a big data platform, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above-mentioned methods for verifying data validity for a big data platform when executing the computer program.
[0067] In summary, the present application reads the data to be detected and historical data of various attribute types, obtains the attribute detail richness of each attribute type based on the distribution of the historical data of various attribute types, obtains the key attributes in all attribute types based on the attribute detail richness, and screens all attribute types, thereby improving the efficiency of data validity verification; spatially maps the data to be detected and historical data of all key attributes, analyzes the distribution of the data to be detected and historical data in the spatial mapping results, and obtains the attribute value attribution of each data to be detected on the corresponding key attribute, which has the beneficial effect of deeply mining the characteristics of historical data, avoiding the interference of the data discreteness of the historical data itself on the value of the data to be detected, and improving the accuracy of subsequent data validity analysis; analyzes the distribution of the historical data of each key attribute in the spatial mapping result, and obtains the attribute value attribution of each key attribute. randomness of values; combining the attribute value attribution degree with the randomness of values, the validity degree of the attribute value of each data to be detected on the corresponding key attribute is obtained, which reflects the possibility that the data to be detected is valid data under the corresponding key attribute, and improves the reliability of the validity judgment of the data to be detected; analyzing the difference between the distribution of the data to be detected and the distribution of historical data on each key attribute, and obtaining the data difference degree of all the data to be detected under each key attribute; combining the attribute value validity degree with the data difference degree, obtaining the validity score of all the data to be detected under each key attribute, obtaining the data validity verification result, reducing the risk of errors in data validity judgment caused by changes in data sources, and improving the accuracy of data validity verification by analyzing the distribution characteristics of a large amount of data to be detected and the comparative relationship with historical data.
[0068] It should be noted that the above sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0069] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0070] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Modifications to the technical solutions recorded in the aforementioned embodiments, or equivalent replacement of some of the technical features therein, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A data validity verification method for a big data platform, characterized in that: The method comprises the following steps: Read the data to be tested and historical data of various attribute types; Determining key attributes in all attribute types based on the distribution of the historical data of various attribute types; Analyze the distribution of the data to be detected and the historical data in each key attribute to obtain the attribute value attribution degree of each data to be detected on the corresponding key attribute; analyze the distribution of the historical data of each key attribute to obtain the value randomness of each key attribute; combine the attribute value attribution degree and the value randomness to obtain the attribute value validity degree of each data to be detected on the corresponding key attribute; Analyze the difference between the distribution of the data to be tested and the distribution of historical data on each key attribute, and obtain the data difference degree of all the data to be tested under each key attribute; combine the validity degree of the attribute value and the data difference degree to obtain the validity score of all the data to be tested under each key attribute, and obtain the data validity verification result; The process of determining the attribute value attribution degree is as follows: The value range of all historical data of each key attribute is evenly divided into a preset number of value intervals, and the ratio of the number of historical data in each value interval of each key attribute to the number of all historical data is determined as the interval projection quantity ratio of each value interval of each key attribute; If the data to be detected is within the value interval, the sum of the ratio of the number of interval projections corresponding to the value interval and the preset value is used as the attribute value attribution degree of the data to be detected; otherwise, the preset value is used as the attribute value attribution degree of the data to be detected; The process of determining the randomness of the value is as follows: All the interval projection number ratios of each key attribute are used to form a projection number distribution sequence, the concentration degree of the projection number distribution sequence is calculated, and the difference between the value 1 and the concentration degree is used as the value randomness of each key attribute; The process of determining the degree of data difference is as follows: The ratio of the number of data to be detected in each value interval of each key attribute to the number of all data to be detected is calculated and recorded as the first ratio. All the first ratios of each key attribute are combined into a distribution sequence of data to be checked. The measured distance between the distribution sequence of data to be checked and the projection number distribution sequence of each key attribute is determined as the data difference degree of all data to be detected under each key attribute.
2. A data validity verification method for a big data platform as claimed in claim 1, characterized in that: The process of determining the key attributes is as follows: The richness of attribute details of each attribute type is determined based on the degree of chaos of the distribution of all historical data of various attribute types; a clustering algorithm is used to cluster the richness of attribute details of all attribute types to obtain clusters, and the key attributes of all attribute types are obtained based on the clusters.
3. A data validity verification method for a big data platform as claimed in claim 2, characterized in that: The attribute detail richness is the information entropy of all historical data of each attribute type.
4. A data validity verification method for a big data platform as claimed in claim 2, characterized in that: The key attributes are all attribute types corresponding to the cluster with the maximum attribute detail richness mean.
5. The data validity verification method of a big data platform according to claim 1, characterized in that: The validity score is a normalized result of the product of the data difference degree of each key attribute and the validity degree of the attribute value.
6. A data validity verification method for a big data platform as claimed in claim 1, characterized in that: The data validity check result is: If the validity score of all the data to be tested under the key attribute is greater than the preset threshold, then all the data to be tested under the corresponding key attribute is valid data; otherwise, all the data to be tested under the corresponding key attribute is invalid data, and the data to be tested except all the data to be tested under all the key attributes is recorded as valid data.
7. A data validity verification system for a big data platform, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Test data verification method and device, storage medium and electronic equipment
CN114706771A
Data verification method and device, computer equipment and storage medium
CN116737708A
Method for verification of data and metadata in a data repository
US20090171991A1
