A method, system, device and medium for automatically verifying the quality of structured data

Abstract

The present invention relates to a method, system, device and medium for automatically verifying the quality of structured data. The method comprises: extracting the field type of each field in a table to be verified; and selecting a corresponding verification method according to the field type of the field to verify the data of the field. The technical solution provided by the present invention is used for data quality verification of structured data, which reduces the workload of data quality verification and improves work efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, device and medium for automatically verifying the quality of structured data. Background Art

[0002] In structured data, the value of a field should be within a certain range or the format and type should be unified. However, in actual situations, a small part of the data may be inconsistent with other data formats or value ranges. In this case, the data is considered abnormal. In the prior art, for these abnormal data, it is necessary to manually understand the data content and format and formulate data quality rules separately so that the abnormal data can be automatically located. This method has a large amount of repetitive work and is relatively time-consuming, and has the problem of low inspection efficiency. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present invention provides a method, system, device and medium for automatically verifying the quality of structured data, which can realize automatic verification of data quality and greatly reduce the workload of manual matching.

[0004] According to a first aspect of an embodiment of the present invention, a method for automatically verifying the quality of structured data is provided, comprising:

[0005] Extract the field type of each field in the table to be verified;

[0006] Select the corresponding verification method according to the field type to verify the data of the field.

[0007] Further, according to the field type of the field, a corresponding verification method is selected to verify the data of the field, specifically including:

[0008] Extract a set amount of sample data from a field;

[0009] If the field type is numeric, the standard deviation σ and mean μ of the sample data are calculated;

[0010] Determine whether the data in the field is abnormal based on the 3sigma principle.

[0011] Further, according to the field type of the field, a corresponding verification method is selected to verify the data of the field, which specifically includes:

[0012] If the field type is time and date type, the sample data is converted to a timestamp, and the standard deviation σ and mean μ of the sample data are calculated according to the timestamp;

[0013] Determine whether the data in the field is abnormal based on the 3sigma principle.

[0014] Further, according to the field type of the field, a corresponding verification method is selected to verify the data of the field, which specifically includes:

[0015] If the field type is character type, calculate the proportion of numeric data and non-numeric data in the sample data;

[0016] If the proportion of numeric data in the sample data exceeds the threshold, the non-numeric data in the field is judged as abnormal data, and the numeric data in the field is verified according to the verification method for numeric fields;

[0017] If the proportion of numerical data in the sample data does not exceed the threshold, then determine whether the proportion of non-numerical data in the sample data exceeds the threshold;

[0018] If the proportion of non-numeric data in the sample data exceeds the threshold, the numeric data in the field is judged to be abnormal data, and whether the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold;

[0019] If the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold, then determine whether the proportion of time and date data in the sample data exceeds the threshold;

[0020] If the proportion of time and date data in the sample data exceeds the threshold, the non-time and date data in the field is judged as abnormal data, and the time and date data in the field are judged according to the verification method of the field type being time and date type;

[0021] If the proportion of time and date data in the sample data does not exceed the threshold, the data type combination of the sample data is determined based on the numerical type of each digit of the sample data. If the proportion of the sample data that conforms to the same data type combination exceeds the threshold, the data in the field that does not conform to the data type combination is judged to be abnormal data.

[0022] According to a second aspect of an embodiment of the present invention, a system for automatically verifying the quality of structured data is provided, comprising:

[0023] Extraction module, extracting the field type of each field in the table to be verified;

[0024] The verification module is used to select the corresponding verification method according to the field type of the field to verify the data of the field.

[0025] Furthermore, the verification module specifically includes:

[0026] An extraction unit, used to extract a set amount of sample data from a field;

[0027] The first verification unit is specifically used for:

[0028] If the field type is numeric, the standard deviation σ and mean μ of the sample data are calculated;

[0029] Determine whether the data in the field is abnormal based on the 3sigma principle.

[0030] Furthermore, the verification module further includes a second verification unit, which is specifically used to:

[0031] If the field type is time and date type, the sample data is converted to a timestamp, and the standard deviation σ and mean μ of the sample data are calculated according to the timestamp;

[0032] Determine whether the data in the field is abnormal based on the 3sigma principle.

[0033] Furthermore, the verification module further includes a third verification unit, which is specifically used to:

[0034] If the field type is character type, calculate the proportion of numeric data and non-numeric data in the sample data;

[0035] If the proportion of numeric data in the sample data exceeds the threshold, the non-numeric data in the field is judged as abnormal data, and the numeric data in the field is verified according to the verification method for numeric fields;

[0036] If the proportion of numerical data in the sample data does not exceed the threshold, then determine whether the proportion of non-numerical data in the sample data exceeds the threshold;

[0037] If the proportion of non-numeric data in the sample data exceeds the threshold, the numeric data in the field is judged to be abnormal data, and whether the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold;

[0038] If the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold, then determine whether the proportion of time and date data in the sample data exceeds the threshold;

[0039] If the proportion of time and date data in the sample data exceeds the threshold, the non-time and date data in the field is judged as abnormal data, and the time and date data in the field are judged according to the verification method of the field type being time and date type;

[0040] If the proportion of time and date data in the sample data does not exceed the threshold, the data type combination of the sample data is determined based on the numerical type of each digit of the sample data. If the proportion of the sample data that conforms to the same data type combination exceeds the threshold, the data in the field that does not conform to the data type combination is judged to be abnormal data.

[0041] According to a third aspect of an embodiment of the present invention, a terminal device is provided, including:

[0042] Processor; and

[0043] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0044] According to a fourth aspect of an embodiment of the present invention, there is provided a non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method as described above.

[0045] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0046] Used for data quality verification of structured data, it reduces the workload of data quality verification and improves work efficiency.

[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0049] Figure 1 The figure is a flowchart of a method for automatically verifying the quality of structured data according to an exemplary embodiment of the present invention.

[0050] Figure 2 The figure is a flow chart of a system for automatically verifying the quality of structured data according to an exemplary embodiment of the present invention.

[0051] Figure 3 It is a schematic diagram showing the structure of a computing device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0052] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0053] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0054] It should be understood that although the terms "first", "second", "third", etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0055] The technical solutions of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] Figure 1 The figure is a flowchart of a method for automatically verifying the quality of structured data according to an exemplary embodiment of the present invention.

[0057] See also Figure 1 , the method comprising:

[0058] 110. Extract the field type of each field in the table to be verified.

[0059] Specifically, in one embodiment, the field type of each field in the table is first extracted and divided into numeric type, character type, time type, date type, and binary type. Among them, the binary data cannot judge the data quality through the information of the data value, so no judgment is made and it is directly defined as abnormal data.

[0060] 120. Select the corresponding verification method according to the field type to verify the data of the field.

[0061] Optionally, in this embodiment, step 120 specifically includes:

[0062] 1201. Extract a set amount of sample data from a field.

[0063] Specifically, in order to reduce the amount of calculation, 1000 non-empty data can be selected from the field as sample data for calculation.

[0064] 1202. If the field type is numeric, calculate the standard deviation σ and mean μ of the sample data.

[0065] 1203. Determine whether the data in the field is abnormal based on the 3sigma principle.

[0066] Specifically, when the field type is numeric, the 3sigma principle is used to determine whether the data is abnormal. Specifically, the "3sigma principle" means that the probability of a value being distributed between (μ-3σ, μ+3σ) is 0.9974, that is, approximately 99.7% of the data will be distributed within three standard deviations around the mean. First, the standard deviation σ and the mean μ of the sample data are calculated, and the data that is not in the numerical range (μ-3σ, μ+3σ) is abnormal data.

[0067] Optionally, in this embodiment, step 120 further includes:

[0068] 1204. If the field type is time and date type, the sample data is converted into a timestamp, and the standard deviation σ and mean μ of the sample data are calculated according to the timestamp.

[0069] 1205. Determine whether the data in the field is abnormal based on the 3sigma principle.

[0070] Specifically, if the field type is time and date type, first convert the sample data into a timestamp, and judge whether the data is abnormal according to the 3sigma principle. Specifically, if the field type is time type, calculate the standard deviation σ and the average μ of the sample data according to the timestamp, calculate the (μ-3σ, μ+3σ) data range, and take the value upward, convert the calculated value into a time format, and obtain a time range. Data not in this time range is abnormal data. If the field type is date type, when converting the time range into a date, subtract one day from the start time and add one day to the end time.

[0071] Optionally, in this embodiment, step 120 further includes:

[0072] 1206. If the field type is character type, calculate the proportion of numeric data and non-numeric data in the sample data;

[0073] 1207. If the proportion of numeric data in the sample data exceeds a threshold, the non-numeric data in the field is determined to be abnormal data, and the numeric data in the field is verified according to the verification method for numeric field types.

[0074] Specifically, in one embodiment, if the field type is character type, the data stored in the field may be of various types such as Chinese, English, symbols, and numerical values. It is necessary to first determine the type of the sample data and perform corresponding processing according to the type. First, calculate the proportion of numerical data and non-numerical data in the sample data. If the proportion of numerical data in the sample data exceeds a threshold of 95%, the non-numerical data is considered to be abnormal data, and the numerical data in the field is determined according to the verification method for the field type being numerical type, otherwise proceed to the next step of determination.

[0075] 1208. If the proportion of numerical data in the sample data does not exceed the threshold, determine whether the proportion of non-numerical data in the sample data exceeds the threshold;

[0076] 1209. If the proportion of non-numeric data in the sample data exceeds a threshold, the numeric data in the field is determined to be abnormal data, and it is determined whether the proportion of data with the same length in the non-numeric data in the sample data exceeds a threshold.

[0077] Specifically, if the proportion of numerical data in the sample data does not exceed the threshold value of 95%, it is further determined whether the proportion of non-numerical data in the sample data exceeds the threshold value of 95%. If it does not exceed the threshold value of 95%, that is, the proportions of numerical data and non-numerical data in the sample data are both higher than 5%, it is considered that the data in this field contains both a large amount of numerical data and a large amount of non-numerical data, and the data stored in this field may be an irregular string, which cannot be automatically verified for data quality, and the data in this field is not judged; if it exceeds the threshold value of 95%, it is considered that the numerical data is abnormal data and the non-numerical data is normal data, and the next step of judgment is performed on the non-numerical data.

[0078] 1210. If the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold, determine whether the proportion of time and date data in the sample data exceeds the threshold.

[0079] 1211. If the proportion of time and date data in the sample data exceeds the threshold, the non-time and date data in the field is judged to be abnormal data, and the time and date data in the field are judged according to the verification method of the field type being time and date type.

[0080] Specifically, in one embodiment, step 1210 determines whether the lengths of the sample data are consistent. If the proportion of data with consistent lengths is greater than or equal to the threshold of 95%, the sample data is subjected to time and date format verification. The formats include "year-month-day hour:minute:second", "year-month-day", "year-month-day hour:minute:second", etc. The sample data that passes the verification is time and date type data. If the proportion of time and date type data in the sample data exceeds the threshold of 95%, the non-time and date type data is abnormal data, and the time and date type data in this field is judged according to the above verification method for time and date type data.

[0081] If the proportion of data with consistent lengths in the sample data is less than 95%, the data in this field may store an irregular string and cannot be automatically verified for data quality, so no judgment is made.

[0082] 1212. If the proportion of time and date type data in the sample data does not exceed the threshold, the data type combination of the sample data is determined according to the numerical type of each value of the sample data. If the proportion of data that conforms to the same data type combination in the sample data exceeds the threshold, the data that does not conform to this data type combination in the field is judged as abnormal data.

[0083] Specifically, in one embodiment, if the proportion of time and date type data in the data does not exceed the threshold of 95%, the type of each value of the sample data is judged in the order of numbers, English, Chinese, and symbols, and the data is identified according to the combination order of the types. If the proportion of data that conforms to the same identification in the sample data exceeds the threshold of 95%, the data that conforms to this identification is normal data, and the others are abnormal data, otherwise no judgment is made. For example, the identifications of "A12S3@" and "S32A3@" are "English + numerical value + numerical value + English + numerical value + symbol", and the identification of "啊12431" is "Chinese + numerical value + numerical value + numerical value + numerical value + numerical value". Since the two identifications are different, it is considered that "啊12431", "A12S3@", and "S32A3@" are not the same type of data.

[0084] Figure 2 It is a structural block diagram of a system for automatically verifying the quality of structured data shown according to an exemplary embodiment of the present invention.

[0085] See Figure 2 , the system includes:

[0086] An extraction module that extracts the field types of each field in the table to be verified;

[0087] A verification module for verifying the data of this field by selecting the corresponding verification method according to the field type of the field.

[0088] Optionally, in this embodiment, the verification module specifically includes:

[0089] An extraction unit, used to extract a set amount of sample data from a field;

[0090] The first verification unit is specifically used for:

[0091] If the field type is numeric, the standard deviation σ and mean μ of the sample data are calculated;

[0092] Determine whether the data in the field is abnormal based on the 3sigma principle.

[0093] Optionally, in this embodiment, the verification module further includes a second verification unit, which is specifically configured to:

[0094] If the field type is time and date type, the sample data is converted to a timestamp, and the standard deviation σ and mean μ of the sample data are calculated according to the timestamp;

[0095] Determine whether the data in the field is abnormal based on the 3sigma principle.

[0096] Optionally, in this embodiment, the verification module further includes a third verification unit, which is specifically configured to:

[0097] If the field type is character type, calculate the proportion of numeric data and non-numeric data in the sample data;

[0098] If the proportion of numeric data in the sample data exceeds the threshold, the non-numeric data in the field is judged as abnormal data, and the numeric data in the field is verified according to the verification method for numeric fields;

[0099] If the proportion of numerical data in the sample data does not exceed the threshold, then determine whether the proportion of non-numerical data in the sample data exceeds the threshold;

[0100] If the proportion of non-numeric data in the sample data exceeds the threshold, the numeric data in the field is judged to be abnormal data, and whether the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold;

[0101] If the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold, then determine whether the proportion of time and date data in the sample data exceeds the threshold;

[0102] If the proportion of time and date data in the sample data exceeds the threshold, the non-time and date data in the field is judged as abnormal data, and the time and date data in the field are judged according to the verification method of the field type being time and date type;

[0103] If the proportion of time and date data in the sample data does not exceed the threshold, the data type combination of the sample data is determined based on the numerical type of each digit of the sample data. If the proportion of the sample data that conforms to the same data type combination exceeds the threshold, the data in the field that does not conform to the data type combination is judged to be abnormal data.

[0104] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0105] Figure 3 It is a schematic diagram showing the structure of a computing device according to an exemplary embodiment of the present invention.

[0106] See also Figure 3 , the computing device 300 includes a memory 310 and a processor 320 .

[0107] The processor 320 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0108] The memory 310 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. Among them, ROM can store static data or instructions required by the processor 320 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 310 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 310 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0109] The memory 310 stores executable codes, and when the executable codes are processed by the processor 320 , the processor 320 can execute part or all of the methods described above.

[0110] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing part or all of the steps in the above method of the present invention.

[0111] Alternatively, the present invention may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present invention.

[0112] The scheme of the present invention has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art should also be aware that the actions and modules involved in the specification are not necessarily required by the present invention. In addition, it can be understood that the steps in the method of the embodiment of the present invention can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present invention can be merged, divided and deleted according to actual needs.

[0113] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0114] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for automatically verifying the quality of structured data, characterized in that: include: Extract the field type of each field in the table to be verified; Select a corresponding verification method according to the field type of the field to verify the data of the field; Select a corresponding verification method according to the field type of the field to verify the data of the field, including: extracting a set amount of sample data from the field; If the field type is numeric, the standard deviation σ and the mean value μ of the sample data are calculated; and whether the data of the field is abnormal is determined according to the 3sigma principle; If the field type is time and date type, convert the sample data into a timestamp, calculate the standard deviation σ and the average value μ of the sample data according to the timestamp; determine whether the data in the field is abnormal based on the 3sigma principle; If the field type is character type, then calculate the proportion of numerical data and non-numerical data in the sample data; If the proportion of numeric data in the sample data exceeds a threshold, the non-numeric data in the field is determined to be abnormal data, and the numeric data in the field is verified according to a verification method for a numeric field; If the proportion of numerical data in the sample data does not exceed the threshold, determining whether the proportion of non-numerical data in the sample data exceeds the threshold; If the proportion of non-numeric data in the sample data exceeds a threshold, the numeric data in the field is determined to be abnormal data, and whether the proportion of data with the same length in the non-numeric data in the sample data exceeds a threshold; If the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold, determining whether the proportion of time and date data in the sample data exceeds the threshold; If the proportion of time and date data in the sample data exceeds a threshold, the non-time and date data in the field is judged as abnormal data, and the time and date data in the field are judged according to the verification method for the field type being the time and date type; If the proportion of time and date type data in the sample data does not exceed the threshold, the data type combination of the sample data is determined according to the numerical type of each bit value of the sample data; if the proportion of the sample data that conforms to the same data type combination exceeds the threshold, the data in the field that does not conform to the data type combination is judged to be abnormal data.

2. A system for automatically verifying the quality of structured data, characterized in that: include: Extraction module, extracting the field type of each field in the table to be verified; A verification module, used to select a corresponding verification method according to the field type of the field to verify the data of the field; The verification module specifically includes: an extraction unit, used for extracting a set amount of sample data from the field; The first verification unit is specifically used for: If the field type is numeric, the standard deviation σ and the mean value μ of the sample data are calculated; and whether the data of the field is abnormal is determined according to the 3sigma principle; The verification module further includes a second verification unit, which is specifically used to: If the field type is time and date type, the sample data is converted into a timestamp, and the standard deviation σ and the average value μ of the sample data are calculated according to the timestamp; Determine whether the data in the field is abnormal based on the 3sigma principle; The verification module further includes a third verification unit, which is specifically used to: If the field type is character type, then calculate the proportion of numerical data and non-numerical data in the sample data; If the proportion of numeric data in the sample data exceeds a threshold, the non-numeric data in the field is determined to be abnormal data, and the numeric data in the field is verified according to a verification method for a numeric field; If the proportion of numerical data in the sample data does not exceed the threshold, determining whether the proportion of non-numerical data in the sample data exceeds the threshold; If the proportion of non-numeric data in the sample data exceeds a threshold, the numeric data in the field is determined to be abnormal data, and whether the proportion of data with the same length in the non-numeric data in the sample data exceeds a threshold; If the proportion of data with the same length in the non-numeric data in the sample data exceeds the threshold, determining whether the proportion of time and date data in the sample data exceeds the threshold; If the proportion of time and date data in the sample data exceeds a threshold, the non-time and date data in the field is judged as abnormal data, and the time and date data in the field are judged according to the verification method for the field type being the time and date type; If the proportion of time and date type data in the sample data does not exceed the threshold, the data type combination of the sample data is determined according to the numerical type of each bit value of the sample data; if the proportion of the sample data that conforms to the same data type combination exceeds the threshold, the data in the field that does not conform to the data type combination is judged to be abnormal data.

3. A terminal device, characterized in that: include: processor; as well as A memory having executable codes stored therein, which, when executed by the processor, causes the processor to execute the method according to claim 1.

4. A non-transitory machine-readable storage medium having executable code stored thereon, characterized in that: When the executable code is executed by a processor of an electronic device, the processor is caused to perform the method as claimed in claim 1.

Images