Data verification method and device, electronic equipment, storage medium and program product
By comparing the feature values of the target business data with historical feature values, and automatically generating verification rules, the problem of low efficiency in identifying field errors in business data is solved, and efficient and accurate field verification is achieved.
Patent Information
- Application Number
- CN202511087488.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-04
- Publication Date
- 2025-11-21
AI Technical Summary
In existing technologies, errors in business data can occur due to changes in business rules and mistakes in the creation of business data. Existing validation methods suffer from problems such as time lag, high development costs, low efficiency, and outdated business rules.
By acquiring the feature values of the target business data, and using historical feature values to query the target field and its historical field features in the target storage space, comparisons are made to identify field errors, and validation rules are automatically generated and field validation is performed.
It effectively identifies field errors in target business data, improves validation efficiency, shortens processing time from hours to minutes, and reduces manual intervention and development costs.
Smart Images
Figure CN120994649A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of data processing, and in particular, to a data verification method and device, an electronic device, a storage medium, and a program product. BACKGROUND
[0002] At present, due to business rule changes and business data creation errors, etc., field errors may exist in business data. For example, the field value of field field1 in the business data of a certain platform should theoretically be value1, but the field value of field field1 is value2, indicating that field field1 in the business data is incorrect. Therefore, how to effectively identify field errors becomes a problem to be solved. SUMMARY
[0003] Therefore, the present disclosure provides a data verification method and device, an electronic device, a storage medium, and a program product to solve the problem of identifying field errors in business data.
[0004] In a first aspect, the present disclosure provides a data verification method, which comprises:
[0005] obtaining target business data;
[0006] extracting the field value of a first field in the target business data;
[0007] performing feature extraction based on the first field and the field value of the first field to obtain a target feature value of the target business data;
[0008] querying a target field and a historical field feature of the target field that match the target feature value in a target storage space, the target storage space being used to store a second field and a historical field feature of the second field corresponding to at least one historical feature value; the second field including the first field; the data in the target storage space being obtained based on first historical business data;
[0009] obtaining a field verification result of the target business data based on a comparison result of the field feature of the target field in the target business data and the corresponding historical field feature.
[0010] In a second aspect, the present disclosure provides a data verification device, which comprises:
[0011] a first obtaining module configured to obtain target business data;
[0012] a first processing module configured to extract the field value of a first field in the target business data;
[0013] The second processing module is configured to perform feature extraction based on the first field and the field value of the first field, to obtain a target feature value of the target service data.
[0014] The third processing module is configured to query a target field and a historical field feature of the target field that match the target feature value in a target storage space, the target storage space being configured to store a second field corresponding to at least one historical feature value and a historical field feature of the second field; the second field includes the first field; and data in the target storage space is obtained based on first historical service data.
[0015] The fourth processing module is configured to obtain a field verification result of the target service data based on a comparison result of the field feature of the target field in the target service data and the corresponding historical field feature.
[0016] In a third aspect, the present disclosure provides an electronic device, including a memory and a processor, which are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the data verification method in the first aspect or any of the corresponding embodiments thereof.
[0017] In a fourth aspect, the present disclosure provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the data verification method in the first aspect or any of the corresponding embodiments thereof.
[0018] In a fifth aspect, the present disclosure provides a computer program product, which includes computer instructions, and the computer instructions are used to make a computer execute the data verification method in the first aspect or any of the corresponding embodiments thereof.
[0019] The data verification method provided by the embodiments of the present disclosure is used to divide different types of historical service data according to historical feature values of historical service data, to extract second fields and historical feature values of the second fields under the same historical feature value, according to the matching result of the target feature value of the target service data and the historical feature value in the target storage space, to find the second fields and the historical field features of the second fields of the historical service data of the same type as the target service data, to obtain the target field and the historical field feature of the target field. Then, according to the comparison result of the field feature of the target field in the target service data and the historical field feature, it is determined whether the field features of the target service data and the historical service data of the same type are consistent, to realize the field verification of the target service data, and effectively identify the field error in the target service data.
[0020] The advantageous effects of the data verification device, the electronic device, the storage medium, and the program product correspond to the advantageous effects of the data verification method, and will not be described here again. BRIEF DESCRIPTION OF DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the specific embodiments of the present disclosure or the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.
[0022] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure;
[0023] Figure 2 is a flowchart of a data verification method according to an embodiment of the present disclosure;
[0024] Figure 3 is a flowchart of another data verification method according to an embodiment of the present disclosure;
[0025] Figure 4 is a flowchart of still another data verification method according to an embodiment of the present disclosure;
[0026] Figure 5 is a structural block diagram of a data verification device according to an embodiment of the present disclosure;
[0027] Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0028] In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present disclosure.
[0029] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.
[0030] For example, in response to receiving an active request of a user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can autonomously select whether to provide personal information to the software or hardware, such as an electronic device, an application program, a server, or a storage medium, performing the operation of the technical solution of the present disclosure according to the prompt information.
[0031] As an optional but non-limiting implementation, in response to receiving an active request of a user, the prompt information can be sent to the user in the form of a pop-up window, in which the prompt information can be presented in the form of text. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.
[0032] It can be understood that the above notification and obtaining of user authorization process is only illustrative and does not limit the implementation of the present disclosure. Other ways that meet relevant laws and regulations can also be applied to the implementation of the present disclosure.
[0033] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the obtaining or use of the data) should comply with the requirements of relevant laws and regulations and relevant provisions.
[0034] At present, due to reasons such as business rule changes and business data creation errors, there may be field errors in business data. For example, the field value of a field field1 in the business data of a certain platform should theoretically be value1, but the field value of the field field1 is value2, indicating that the field field1 in the business data is incorrect. Therefore, how to effectively identify the field error becomes a problem to be solved.
[0035] In the related art, the following two schemes are mainly used to check the fields of business data:
[0036] The first scheme: manually export multiple data tables of business data for Excel comparison at regular intervals. However, this scheme has time lag, incomplete field coverage, and business rules need to be manually identified and sorted. If updated in real time, there may be problems of old business rules.
[0037] The second scheme: using code hardcoding to check and verify the fields of business data one by one for each business scenario. However, this scheme has high development cost and long cycle, and new business rules need to go through the whole cycle of development, testing, and online. In addition, when the business rules are upgraded, the old business rules need to be modified synchronously to avoid false positives, so the code development cycle needs to be repeated. In addition, the fields checked generally only cover key fields and cannot cover all business data.
[0038] In conclusion, the related art has the problems of low efficiency of manual checking of field differences of multiple data sources, and the problems of extra development cost of checking scripts caused by non-uniform field logic of business rules across platforms, and the problems of extra modification to maintain accuracy caused by business rule changes in a huge amount of data scenarios.
[0039] Therefore, according to an embodiment of the present disclosure, a data verification method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0040] As an optional application scenario of the embodiment of the present disclosure, as shown in the figure, Figure 1 The entire data verification scenario includes multiple business platforms 1 (such as business platform 11, business platform 12, and business platform 13) and a data verification platform 2. The data verification platform 2 automatically generates field verification rules based on historical business data of the business platform 1 (see the second field corresponding to the historical feature value and the historical field feature of the second field described below), and performs field verification on the target business data to be verified according to the generated field verification rules to obtain the field verification result of the target business data.
[0041] In this embodiment, a data verification method is provided, which can be used in the data verification platform described above, Figure 2 is a flowchart of a data verification method according to an embodiment of the present disclosure, as shown in the figure, Figure 2 The flowchart includes the following steps:
[0042] Step S201, obtaining target business data.
[0043] Specifically, the target business data can be obtained in a Binlog triggered and / or manually triggered manner. The Binlog trigger is to obtain the changed business data in real time when the business data is changed by subscribing to data change messages to obtain the target business data.
[0044] Manual triggering is to configure data filtering conditions according to data verification requirements, and to filter the target business data according to the data filtering conditions. For example, according to the start time, end time, and business identifier, the range of business data that needs to be verified is given, and the target business data is filtered.
[0045] Step S202, extracting the field value of the first field in the target business data.
[0046] The first field is a field that exists in different service platforms and service scenarios and has a fixed field value in the same type of service data.
[0047] In step S203, a target feature value of the target service data is obtained by performing feature extraction based on the first field and the field value of the first field.
[0048] Specifically, the hash algorithm can be used to perform feature extraction on the first field and the field value of the first field to obtain the target feature value. The neural network, natural language processing model, etc. can also be used to perform feature extraction on the first field and the field value of the first field to obtain the target feature value. The feature extraction method can be selected according to the actual situation.
[0049] In step S204, a target field and a historical field feature of the target field that match the target feature value are queried in a target storage space. The target storage space is used to store a second field and a historical field feature of the second field corresponding to at least one historical feature value. The second field includes the first field. The data in the target storage space is obtained based on the first historical service data.
[0050] It should be noted that the data in the target storage space includes the second field and the historical field feature of the second field corresponding to at least one historical feature value.
[0051] Specifically, the historical feature value that matches the target feature value is found in the target storage space. The second field of the matched historical feature value is taken as the target field of the target feature value, and then the historical field feature of the target field is obtained.
[0052] In step S205, a field verification result of the target service data is obtained based on the comparison result of the field feature of the target field in the target service data and the corresponding historical field feature.
[0053] Specifically, if the field feature of the target field in the target service data does not match the corresponding historical field feature, it is determined that the field verification of the target service data fails. If the field feature of the target field in the target service data matches the corresponding historical field feature, it is determined that the field verification of the target service data passes.
[0054] The data verification method provided in the embodiment is characterized in that the first fields of the same type of service data and the feature values of the fields are consistent, and therefore, the historical feature values of the historical service data are extracted in advance in the embodiment, so as to divide the historical service data of different types by using the historical feature values, to extract the second fields and the historical feature values of the second fields under the same historical feature value. Then, according to the matching result of the target feature value of the target service data and the historical feature values in the target storage space, the second fields of the historical service data of the same type as the target service data and the historical field features of the second fields are searched, to obtain the target field and the historical field features of the target field. Further, according to the comparison result of the field features of the target field in the target service data and the historical field features, it is determined whether the field features of the target service data and the historical service data of the same type are consistent, so as to realize the field verification of the target service data and effectively identify the field error in the target service data.
[0055] In some optional embodiments, the step S201 comprises: in response to the data change message, obtaining the changed service data as the target service data.
[0056] The data verification method provided in the embodiment is characterized in that the data change message is subscribed to obtain the target service data, and therefore, the changed target service data can be obtained in time when the target service data is changed, and the field verification of the changed target service data can be performed in time.
[0057] In some optional embodiments, the step S201 comprises: displaying a target page; obtaining a data filtering condition in response to the data input in the target page; and querying the target service data corresponding to the data filtering condition in the service database based on the data filtering condition.
[0058] Optionally, the data filtering condition comprises a start time, an end time, a service identifier, and the like of the service data, which can be adjusted according to actual conditions.
[0059] The data verification method provided in the embodiment is characterized in that the target service data to be verified is obtained by inputting the data filtering condition, and therefore, the target service data to be verified can be obtained flexibly according to actual requirements.
[0060] In some optional embodiments, the step S203 comprises:
[0061] In step a1, the first fields and the field values of the first fields are fused to obtain a fusion result.
[0062] Specifically, the first fields and the field values of the first fields are spliced to obtain a splicing result of the first fields. Then, the splicing results of the first fields are merged to obtain the fusion result.
[0063] Specifically, the first fields and their field values are spliced in the manner of "field name + field value" to obtain the spliced results of the first fields. If a first field has no field value, the field value of the first field is denoted as "-". Then, the spliced results of the first fields are merged according to a preset field order to obtain a fusion result. The different fields can be sorted and stored in advance to obtain a dictionary. The field order between the first fields is determined by using the dictionary.
[0064] In step a2, the fusion result is subjected to feature extraction to obtain a target feature value.
[0065] Specifically, the step a2 includes performing a hash operation on the fusion result to obtain the target feature value.
[0066] Specifically, the hash operation is performed on the fusion result to obtain a hash fingerprint. The hash fingerprint is taken as the target feature value.
[0067] For example, in an advertisement creation scenario, the business data all contain gd_send_type (representing a delivery target), ad_send_type (representing a rule for various businesses), and cpt_sequence (representing a brush time), and the field values of the three fields are fixed. Therefore, the gd_send_type, the ad_send_type, and the cpt_sequence can be taken as the first fields. The field values of the first fields are as follows: gd_send_type = 0, ad_send_type = 1, and the cpt_sequence has no field value. The fusion result of the three first fields and their field values is "gd_send_type0ad_send_type1cpt_sequence-". The message digest algorithm MD5 in the hash algorithm is used to extract features from the fusion result to obtain a hash fingerprint, which is taken as the target feature value.
[0068] The data verification method provided in this embodiment can effectively represent the features of the target business data by extracting features from the fusion result of the first fields and their field values to obtain the target feature value, because the same type of business data has consistent features for the same first field and its field value.
[0069] In some optional embodiments, the step S205 includes:
[0070] In step b1, all fields in the target business data are compared with all target fields corresponding to the target feature value in the target storage space to obtain a first comparison result.
[0071] Specifically, the step b1 comprises: comparing the total quantity of fields in the target business data with the total quantity of fields of the target field corresponding to the target feature value in the target storage space to obtain a first comparison result.
[0072] Specifically, the total quantity of fields of the target field in the target business data is compared with the total quantity of fields of the target field corresponding to the target feature value in the target storage space to obtain a first comparison result. That is, the target field in the target business data is compared with the target field corresponding to the target feature value in the target storage space for full-field comparison.
[0073] It should be noted that the full-field comparison is mainly used to determine whether the target business data is missing a target field that must exist. If yes, it indicates that the target business data is abnormal. If the target business data contains other fields in addition to the target field that must exist, it indicates that the target business data is still in line with the expected condition in the dimension of the field, which can be ignored.
[0074] Step b2, comparing the field characteristics of the target field in the target business data with the corresponding historical field characteristics to obtain a second comparison result.
[0075] That is, the field characteristics of the target field in the target business data are compared with the corresponding historical field characteristics for difference set analysis to obtain a second comparison result.
[0076] If the field characteristics of any target field in the target business data do not match the corresponding historical field characteristics, the target field is regarded as an abnormal field, and it is determined that the field verification of the target business data fails.
[0077] Step b3, fusing the first comparison result and the second comparison result to obtain a field verification result of the target business data.
[0078] That is, it is determined whether the target business data is missing a target field based on the first comparison result, whether the missing target field is a field that must exist, and whether the field characteristics of other target fields match the field characteristics of the similar business data based on the second comparison result.
[0079] The data verification method provided in the embodiment determines the field verification result of the target business data based on the fusion result of the first comparison result of the full-field comparison and the second comparison result of the field characteristics of the target field. Therefore, the accuracy of the field verification result can be further improved.
[0080] In some optional embodiments, the data verification method of the present disclosure further comprises:
[0081] Step c1, obtaining a plurality of first historical business data.
[0082] The first historical business data is obtained from one or more data sources. For example, the first historical business data includes business data of the business platform 11, business data of the business platform 12, and business data of the business platform 13.
[0083] Optionally, the step c1 includes:
[0084] In step c11, historical business data in a first preset time range is obtained, and an end time of the first preset time range is a current time.
[0085] In step c12, a first data total amount of the historical business data in the first preset time range is determined.
[0086] In step c13, if the first data total amount does not reach a first preset quantity requirement, the historical business data in the first preset time range is determined as the first historical business data.
[0087] In actual application, the historical business data can be obtained from the data sources associated with the data verification platform according to a preset period, for example, the historical business data of different data sources is synchronized every day. Alternatively, the historical business data is obtained from the data sources associated with the data verification platform when a preset synchronization rule is met.
[0088] Optionally, the first preset time range is the last 6 months, which can be adjusted according to actual conditions, such as the last 5 months.
[0089] In actual application, the number of historical business data required for calculating the historical feature value can be determined according to actual needs, and the first preset quantity requirement is determined. For example, the first preset quantity requirement is 5000 historical business data.
[0090] For example, if 5000 historical business data is required for calculating the historical feature value, but the historical business data in the last 6 months obtained does not reach 5000. The historical business data in the last 6 months is determined as the first historical business data. Thus, the first historical business data can ensure that the amount of business data required for calculating the historical feature value is met, so as to improve the accuracy of the historical feature value and cover different types of business data as much as possible.
[0091] Further, the step c1 includes:
[0092] In step c14, if the first data total amount reaches the first preset quantity requirement, a generation time of the historical business data is obtained.
[0093] In step c15, the first historical business data of the first preset quantity requirement is filtered from the historical business data in the first preset time range based on the generation time.
[0094] For example, if 5000 pieces of historical service data need to be selected, and there are 1w pieces of historical service data in the last 6 months, the latest 5000 pieces of historical service data are selected as the first historical service data according to the generation time of the historical service data. Thus, the data amount of a single calculation of the historical characteristic value can be avoided, and the calculation efficiency of the historical characteristic value is improved.
[0095] Step c2, extracting the field value of the first field in the first historical service data.
[0096] The first field is a field that exists in different service data and has a fixed value in the same type of service data. For example, a field for representing the target of promotion data, a field for representing the promotion plan.
[0097] Step c3, performing feature extraction based on the first field and the field value of the first field in the first historical service data to obtain the historical characteristic value of the first historical service data.
[0098] Specifically, the first field and the field value in the first historical service data are fused, and the fusion result is subjected to feature extraction to obtain the historical characteristic value.
[0099] The fusion mode of the first field and the field value in the first historical service data can refer to the fusion mode of the first field and the field value in the target service data, which will not be described in detail here.
[0100] Further, the fusion result of the first field and the field value in the first historical service data is subjected to a hash operation to obtain the historical characteristic value.
[0101] Step c4, analyzing the field information in the first historical service data of the same historical characteristic value to obtain the second field corresponding to the historical characteristic value and the historical field characteristic of the second field.
[0102] It can be understood that if the historical characteristic values of the first historical service data are the same, it indicates that the first historical service data belongs to the same type or is derived from the same data source. Therefore, the field information in the first historical service data of the same historical characteristic value can be analyzed to determine the second field in the first historical service data of the same type corresponding to the historical characteristic value and the historical field characteristic of the second field in the first historical service data of the same type.
[0103] It should be noted that the second fields corresponding to different historical characteristic values can be the same or different.
[0104] Optionally, the above step c4 includes:
[0105] In step c41, the second historical service data in the first historical service data within the second preset time range is filtered based on the generation time of the first historical service data, to obtain the target historical service data, and the end time of the second preset time range is the current time.
[0106] Optionally, the second preset time range is the last 6 months, which can be adjusted according to actual conditions, such as the last 1 month, the last 7 days, etc.
[0107] Further, the above step c41 includes:
[0108] In step c411, the second historical service data in the first historical service data within the second preset time range is filtered based on the generation time of the first historical service data.
[0109] In step c412, the second data total amount of the second historical service data is determined.
[0110] In step c413, if the second data total amount does not meet the second preset quantity requirement, the third historical service data in the first historical service data within the third preset time range is filtered based on the generation time of the first historical service data, the start time of the third preset time range is earlier than the start time of the second preset time range, and the end time of the third preset time range is the current time.
[0111] In step c414, the third historical service data is taken as the target historical service data.
[0112] It can be understood that if the obtained target historical service data is too little, it may affect the judgment of the second field and the historical field characteristics of the second field in the same type of historical service data. Therefore, the second preset quantity requirement needs to be set according to actual conditions to ensure that the data amount of the selected target historical service data can accurately analyze the second field and the historical field characteristics of the second field of the historical service data of this type.
[0113] Optionally, the second preset quantity requirement is 100 historical service data, which can be adjusted according to actual conditions.
[0114] It should be noted that the above steps c411-c414 are all filtering of historical service data (such as first historical service data, second historical service data, third historical service data) of the same historical characteristic value.
[0115] For example, the second preset time range is the last 1 month, the third preset time range is the last 6 months, the second preset quantity requirement is 100, and assuming that the second data total amount of the second historical service data of the same historical feature value in the last 1 month does not reach 100, the first historical service data of the historical feature value in the last 6 months is taken as the target service data.
[0116] Step c42, analyzing the field information in the target historical service data of the same historical feature value to obtain the second field corresponding to the historical feature value and the historical field feature of the second field.
[0117] Specifically, the above step c42 includes: analyzing the field existence of different fields in the target historical service data of the same historical feature value and the field value change of the corresponding field to obtain the second field corresponding to the historical feature value and the historical field feature of the second field.
[0118] Specifically, the full-amount field of the target historical service data under the same historical feature value is analyzed to obtain the second field corresponding to the historical feature value and the historical field feature of the second field.
[0119] Optionally, referring to Table 1, the historical field feature is mainly divided into five categories. The first field feature: the field may exist or may not exist. The second field feature: the field must exist, but the field value is not fixed. The third field feature: the field must exist and the field value is a fixed value (the fixed value needs to be recorded). The fourth field feature: special logic when the field is an enumeration value, that is, the field must exist and the field value is an enumeration value in a specific range. The fifth field feature: the field does not meet the first to fourth field features, and an ignore label is added. Among them, for the third field feature, a first preset field is needed to record the fixed field value of the field. For the fourth field feature, a second preset field is needed to record the enumeration value range corresponding to the field.
[0120] Table 1: Information of historical field feature
[0121] Name of the historical field feature Meaning First field feature Field can or can not exist Second field feature Field must exist, but field value is not fixed Third field feature Field must exist and field value is a fixed value (the fixed value needs to be recorded) Fourth field feature Field must exist and field value is an enumerated value in a specific range Fifth field feature Field does not satisfy the first to fourth field features described above
[0122] It should be noted that the above five field features are only an optional embodiment, and in actual application, the above five field features can be adjusted and extended according to actual conditions.
[0123] Step c5, writing the second field corresponding to the historical feature value and the historical field feature of the second field into the target storage space.
[0124] The data checking method provided by the embodiment can filter out historical service data of the same type through historical feature values of the historical service data, because the first fields of the historical service data of the same type and the feature of the field values thereof are consistent. Furthermore, the second field of the historical service data of the corresponding type is analyzed, and the historical field feature of the second field under the type is determined, for the historical service data of the same historical feature value, so that the differentiated field checking rule of the historical feature value can be automatically generated from the historical service data, that is, the second field corresponding to each historical feature value and the historical field feature of the second field. Furthermore, the field checking rule is determined by searching for a matching historical feature value through a target feature value of target service data to be checked, so as to perform field checking without manual intervention, thereby effectively improving the field checking efficiency. Moreover, when a new service rule appears, the field checking rule divided according to the historical feature value can be automatically updated by using the historical service data of the new service rule, without manual intervention to change the field checking rule.
[0125] In some optional embodiments, the data checking method of the present disclosure further comprises:
[0126] Step d1, if the field checking result indicates that there is an abnormal field in the target service data, obtaining abnormal processing information, and the abnormal field belongs to the target field; the abnormal processing information includes an abnormal processing action corresponding to at least one field feature.
[0127] Step d2, determining a target abnormal processing action matching the historical field feature corresponding to the abnormal field in the abnormal processing information, and executing the target abnormal processing action.
[0128] Wherein, the abnormal field is a target field whose field feature in the target service data does not match the corresponding historical field feature.
[0129] Specifically, taking the above five types of historical field features as an example, the first type of field feature is impossible to be inconsistent with the expectation, so the corresponding abnormal processing information can not be configured. The second type of field feature can be configured with abnormal processing information to send first preset alarm information, such as the abnormal field should exist, which is inconsistent with the expectation. The third type of field feature can be configured with abnormal processing information to send second preset alarm information, such as the field value of the abnormal field is incorrect, and the field value is a fixed value X, which is inconsistent with the expectation. The fourth type of field feature can be configured with abnormal processing information to send third preset alarm information, such as the field value of the abnormal field is incorrect, and the enumeration range of the field value is [X1, X2, X3, …], which is inconsistent with the expectation. The fifth type of field feature can be ignored, and no abnormal processing information is configured.
[0130] In actual application, the matching target abnormal processing action can be determined and executed according to the historical field feature corresponding to the abnormal field.
[0131] The data verification method provided by the embodiment configures different exception handling actions for different historical field characteristics to obtain exception handling information. Therefore, when there is an exception field, the target exception handling action that is adapted to the historical field characteristic of the exception field can be quickly queried according to the preconfigured exception handling information, so that the adapted target exception handling action is executed.
[0132] As a specific application example, refer to Figure 3 The overall flow of the data verification method of the present disclosure mainly includes historical business data processing, historical field characteristic generation, and field data comparison. In the historical business data processing flow, the characteristics of the historical business data in the last 6 months are extracted, historical characteristic values (such as hash fingerprints) are generated, and are stored in a preset first database, to prepare for subsequent field verification rule generation. The specific implementation is as follows: synchronize the historical business data every day, and establish a first database for recording historical characteristic values. Obtain the historical business data in the last 6 months, record the first field in the historical business data and the field value of the first field. Generate historical characteristic values based on the first field of the historical business data and the field value of the first field, and record the historical characteristic values in the first database.
[0133] In the historical field characteristic generation flow, under the premise that the same historical characteristic values represent the same type of business data, the historical business data under each historical characteristic value is extracted, and the fields in each type of historical business data are analyzed to determine the field verification rule under each historical characteristic value. The specific implementation is as follows: a second database is established for recording the second field of different historical characteristic values and the historical field characteristics of the second field. For the historical business data of the same historical characteristic value, the last 100 historical business data X days ago are selected. If the selected historical business data does not meet 100, the historical business data of the last 6 months X days ago is selected. The field characteristics of the historical business data of the same historical characteristic value are analyzed to obtain the second field and the historical field characteristics of the second field. The second field of different historical characteristic values and the historical field characteristics of the second field are recorded in the second database.
[0134] In the flow of the field data comparison stage, the target characteristic values (such as hash fingerprints) of the target business data of two sources are calculated, the field verification rules of the matching historical characteristic values are queried from the above-mentioned first database and second database, to perform automatic field verification on the target business data, and if the field verification rule is not met, an alarm is performed. The specific implementation is as follows: the target business data is obtained by triggering the subscription data change message through the incremental binlog. Alternatively, the target business data is obtained by manually triggering the user to manually input the data filtering conditions. Then, refer to Figure 4According to the first field and the field value of the first field in the target business data, a target feature value is calculated. According to the target feature value, a second field and a historical field feature of a matching historical feature value are queried in the first database and the second database (i.e., the target storage space in the figure), so as to perform field checking on the target business data, and a field checking result of the target business data is obtained. The field checking result of the target business data that does not pass the field checking is recorded in the third database. Feedback is performed based on the field checking result in the third database.
[0135] Actual operation data shows that the data checking method of the present disclosure can effectively identify more than 80% of field errors in business data, shorten the processing time of field checking from the hour level to the minute level, and effectively improve the efficiency of field checking.
[0136] In the present embodiment, a data checking device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.
[0137] The present embodiment provides a data checking device, as shown in Figure 5 , comprising:
[0138] The first acquisition module 501 is configured to acquire target business data.
[0139] The first processing module 502 is configured to extract a field value of a first field in the target business data.
[0140] The second processing module 503 is configured to perform feature extraction based on the first field and the field value of the first field, and obtain a target feature value of the target business data.
[0141] The third processing module 504 is configured to query a target field and a historical field feature of the target field that match the target feature value in a target storage space. The target storage space is configured to store at least one second field corresponding to a historical feature value and a historical field feature of the second field. The second field includes the first field. The data in the target storage space is obtained based on the first historical business data.
[0142] The fourth processing module 505 is configured to obtain a field checking result of the target business data based on a comparison result of the field feature of the target field in the target business data and the corresponding historical field feature.
[0143] In some optional embodiments, the first acquisition module 501 comprises:
[0144] The first obtaining unit is configured to obtain the changed service data as target service data in response to the data change message.
[0145] In some optional embodiments, the first obtaining module 501 includes:
[0146] The display unit is configured to display the target page.
[0147] The input unit is configured to obtain a data filtering condition in response to data input on the target page.
[0148] The second obtaining unit is configured to query target service data corresponding to the data filtering condition in the service database based on the data filtering condition.
[0149] In some optional embodiments, the second processing module 503 includes:
[0150] The fusion unit is configured to fuse the first field and the field value of the first field to obtain a fusion result.
[0151] The feature extraction unit is configured to perform feature extraction on the fusion result to obtain a target feature value.
[0152] In some optional embodiments, the feature extraction unit includes:
[0153] The feature extraction subunit is configured to perform a hash operation on the fusion result to obtain the target feature value.
[0154] In some optional embodiments, the fourth processing module 505 includes:
[0155] The first comparison unit is configured to compare all fields in the target service data with all target fields corresponding to the target feature value in the target storage space to obtain a first comparison result.
[0156] The second comparison unit is configured to compare the field feature of the target field in the target service data with the corresponding historical field feature to obtain a second comparison result.
[0157] The comprehensive analysis unit is configured to fuse the first comparison result and the second comparison result to obtain a field verification result of the target service data.
[0158] In some optional embodiments, the first comparison unit includes:
[0159] The field total amount comparison subunit is configured to compare the total amount of fields in the target service data with the total amount of target fields corresponding to the target feature value in the target storage space to obtain the first comparison result.
[0160] In some optional embodiments, the data verification apparatus of the present disclosure further includes:
[0161] a second obtaining module, configured to obtain a plurality of first historical service data;
[0162] a fifth processing module, configured to extract a field value of a first field in the first historical service data;
[0163] a sixth processing module, configured to perform feature extraction based on the first field and the field value of the first field in the first historical service data, to obtain a historical feature value of the first historical service data;
[0164] a seventh processing module, configured to analyze field information in the first historical service data of the same historical feature value, to obtain a second field corresponding to the historical feature value and a historical field feature of the second field;
[0165] an eighth processing module, configured to write the second field corresponding to the historical feature value and the historical field feature of the second field into a target storage space.
[0166] In some optional embodiments, the second obtaining module comprises:
[0167] a first screening unit, configured to obtain historical service data in a first preset time range, an end time of the first preset time range being a current time;
[0168] a first processing unit, configured to determine a first data total amount of the historical service data in the first preset time range;
[0169] a second processing unit, configured to determine the historical service data in the first preset time range as the first historical service data if the first data total amount does not reach a first preset quantity requirement.
[0170] In some optional embodiments, the second obtaining module further comprises:
[0171] a first time obtaining unit, configured to obtain a generation time of the historical service data if the first data total amount reaches the first preset quantity requirement;
[0172] a second screening unit, configured to screen out the first historical service data reaching the first preset quantity requirement from the historical service data in the first preset time range based on the generation time.
[0173] In some optional embodiments, the seventh processing module comprises:
[0174] a third screening unit, configured to screen out second historical service data in a second preset time range from the first historical service data of the same historical feature value based on a generation time of the first historical service data in the first historical service data, to obtain target historical service data, an end time of the second preset time range being a current time.
[0175] a field analysis unit, configured to analyze field information in the target historical service data of the same historical feature value, to obtain a second field corresponding to the historical feature value and a historical field feature of the second field.
[0176] In some optional embodiments, the third screening unit comprises:
[0177] a first screening sub-unit, configured to screen, based on generation time of the first historical service data, second historical service data in the first historical service data that is within a second preset time range;
[0178] a data amount acquisition sub-unit, configured to determine a second total data amount of the second historical service data;
[0179] a second screening sub-unit, configured to, if the second total data amount does not meet a second preset quantity requirement, screen, based on generation time of the first historical service data, third historical service data in the first historical service data that is within a third preset time range, a start time of the third preset time range being earlier than a start time of the second preset time range, and an end time of the third preset time range being a current time;
[0180] a third screening sub-unit, configured to take the third historical service data as the target historical service data.
[0181] In some optional embodiments, the field analysis unit comprises:
[0182] a field analysis sub-unit, configured to analyze field existence of different fields in the target historical service data of the same historical feature value and field value change of the corresponding fields, to obtain the second field corresponding to the historical feature value and the historical field feature of the second field.
[0183] The data verification device provided in the embodiments of the present disclosure can perform the data verification method provided in any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method. The data verification device of the present disclosure divides different types of historical service data in advance according to historical feature values of historical service data, to extract a second field under the same historical feature value and a historical feature value of the second field. Then, according to a matching result of a target feature value of target service data and historical feature values in a target storage space, the second field and the historical field feature of the historical service data of the same type as the target service data are found, to obtain a target field and a historical field feature of the target field. Further, according to a comparison result of field features of the target field in the target service data and the historical field feature, it is determined whether the field features of the target service data and the historical service data of the same type are consistent, so as to realize field verification of the target service data and effectively identify field errors in the target service data.
[0184] Further function description of each module and unit above is the same as the corresponding embodiment above, and will not be repeated here.
[0185] Figure 6 A structural block diagram of an electronic device is provided for the embodiments of the present disclosure.
[0186] Reference will now be made in detail to Figure 6 which shows a structural block diagram suitable for implementing an electronic device in the embodiments of the present disclosure. The electronic device can include a processor (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a memory 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for operation of the electronic device are also stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0187] Generally, the following devices can be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a memory 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device with various devices is shown, but it should be understood that it is not required to implement or have all the shown devices, and more or less devices can be implemented or had instead.
[0188] In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer readable medium, the computer program containing program codes for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 609, or installed from the memory 608, or installed from the ROM 602. When the computer program is executed by the processor 601, the above-mentioned functions defined in the data verification method of the embodiments of the present disclosure are performed.
[0189] Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.
[0190] The embodiments of the present disclosure further provide a computer readable storage medium, and the method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer code originally stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded through a network and stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk or a solid state disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor or hardware, the data verification method shown in the above embodiments is implemented.
[0191] Part of the present disclosure can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present disclosure can be called or provided. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file, etc., accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.
[0192] Although the embodiments of the present disclosure are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present disclosure, and such modifications and changes fall within the scope defined by the appended claims.
Claims
1. A data checking method, characterized by, The method comprises: acquiring target business data; extracting a field value of a first field in the target business data; performing feature extraction based on the first field and the field value of the first field to obtain a target feature value of the target business data; querying a target field matching the target feature value and historical field features of the target field in a target storage space, the target storage space being configured to store a second field corresponding to at least one historical feature value and historical field features of the second field, the second field including the first field, and data in the target storage space being obtained based on first historical business data; obtaining a field verification result of the target business data based on a comparison result of field features of the target field in the target business data and the corresponding historical field features.
2. The method of claim 1, wherein, The feature extraction based on the first field and the field value of the first field to obtain the target feature value of the target business data comprises: fusing the first field and the field value of the first field to obtain a fusion result; performing feature extraction on the fusion result to obtain the target feature value.
3. The method of claim 2, wherein, The feature extraction on the fusion result to obtain the target feature value comprises: performing a hash operation on the fusion result to obtain the target feature value.
4. The method of claim 1, wherein, The obtaining of the field verification result of the target business data based on the comparison result of the field features of the target field in the target business data and the corresponding historical field features comprises: comparing all fields in the target business data with all target fields corresponding to the target feature value in the target storage space to obtain a first comparison result; comparing the field features of the target field in the target business data with the corresponding historical field features to obtain a second comparison result; fusing the first comparison result and the second comparison result to obtain the field verification result of the target business data.
5. The method of claim 4, wherein, The comparison of all fields in the target business data with all target fields corresponding to the target feature value in the target storage space to obtain a first comparison result comprises: comparing a total amount of fields in the target business data with a total amount of fields of the target fields corresponding to the target feature value in the target storage space to obtain the first comparison result.
6. The method of claim 1, wherein, The method further comprises: acquiring a plurality of first historical business data; extracting a field value of the first field in the first historical business data; performing feature extraction based on the first field and the field value of the first field in the first historical business data to obtain a historical feature value of the first historical business data; analyzing field information of the first historical business data of the same historical feature value to obtain a second field corresponding to the historical feature value and historical field features of the second field; writing the second field corresponding to the historical feature value and the historical field features of the second field into the target storage space.
7. The method of claim 6, wherein, The acquisition of a plurality of first historical business data comprises: acquire historical service data in a first preset time range, an end time of the first preset time range being a current time; determine a first data total amount of the historical service data in the first preset time range; if the first data total amount does not reach a first preset quantity requirement, determine the historical service data in the first preset time range as the first historical service data.
8. The method of claim 7, wherein, The acquiring a plurality of first historical service data further comprises: if the first data total amount reaches the first preset quantity requirement, acquire a generation time of the historical service data; based on the generation time, filter out first historical service data of the first preset quantity requirement from the historical service data in the first preset time range.
9. The method according to any one of claims 6-8, characterized in that, The analysis of the field information in the first historical service data of the same historical characteristic value to obtain the second field corresponding to the historical characteristic value and the historical field characteristic of the second field comprises: for the first historical service data of the same historical characteristic value, based on the generation time of the first historical service data, filter out second historical service data in a second preset time range from the first historical service data to obtain target historical service data, an end time of the second preset time range being the current time; analyze the field information in the target historical service data of the same historical characteristic value to obtain the second field corresponding to the historical characteristic value and the historical field characteristic of the second field.
10. The method of claim 9, wherein, The filtering out of the second historical service data in the second preset time range from the first historical service data based on the generation time of the first historical service data to obtain target historical service data comprises: filtering out second historical service data in a second preset time range from the first historical service data based on the generation time of the first historical service data; determining a second data total amount of the second historical service data; if the second data total amount does not reach a second preset quantity requirement, based on the generation time of the first historical service data, filter out third historical service data in a third preset time range from the first historical service data, a start time of the third preset time range being earlier than a start time of the second preset time range, an end time of the third preset time range being the current time; take the third historical service data as the target historical service data.
11. The method of claim 9, wherein, The analysis of the field information in the target historical service data of the same historical characteristic value to obtain the second field corresponding to the historical characteristic value and the historical field characteristic of the second field comprises: analyze the field existence of different fields in the target historical service data of the same historical characteristic value and the field value change of the corresponding fields to obtain the second field corresponding to the historical characteristic value and the historical field characteristic of the second field.
12. The method of claim 1, wherein, The acquiring target service data comprises: in response to a data change message, acquiring changed service data as the target service data.
13. The method of claim 1, wherein, The acquiring target service data comprises: display a target page; In response to the data input in the target page, a data filtering condition is obtained; Based on the data filtering condition, the target business data corresponding to the data filtering condition is queried in a business database.
14. The method of claim 1, wherein, The method further comprises: If the field checking result represents that there is an abnormal field in the target business data, abnormal processing information is obtained, and the abnormal field belongs to the target field; the abnormal processing information comprises an abnormal processing action corresponding to at least one field feature; In the abnormal processing information, a target abnormal processing action matching a historical field feature corresponding to the abnormal field is determined, and the target abnormal processing action is executed.
15. A data checking apparatus, characterized by comprising: The device comprises: A first obtaining module is configured to obtain target business data; A first processing module is configured to extract a field value of a first field in the target business data; A second processing module is configured to perform feature extraction based on the first field and the field value of the first field, and obtain a target feature value of the target business data; A third processing module is configured to query a target field and a historical field feature of the target field matching the target feature value in a target storage space, the target storage space being configured to store a second field and a historical field feature of the second field corresponding to at least one historical feature value; the second field comprises the first field; and data in the target storage space is obtained based on first historical business data; A fourth processing module is configured to obtain a field checking result of the target business data based on a comparison result of a field feature of the target field in the target business data and a corresponding historical field feature.
16. An electronic device, comprising: It comprises: A memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the data checking method of any one of claims 1 to 14.
17. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the data checking method of any one of claims 1 to 14.
18. A computer program product, characterised in that, It comprises computer instructions, and the computer instructions are used to make a computer execute the data checking method of any one of claims 1 to 14.