Data preprocessing method, system and equipment and readable storage medium

Abstract

The invention discloses a data preprocessing method, system and device and a readable storage medium. The method comprises the steps that a to-be-checked record is imported into a preset database table in a specified format; reading the credible table item, and obtaining a credible identification number and corresponding credible identification information; traversing the database table, and if the identification number in the to-be-checked record is consistent with the credible identification number, updating the identification information in the to-be-checked record based on the credible identification information; and writing the checked records with the same identification number into an independent table item, and recording the number of the checked records in the independent table item. Compared with the prior art, the identity information in the corresponding database table can be updated, the situation that name writing errors occur under the same identity document number is repaired, and it is guaranteed that the same identity document number corresponds to the same name; distinguishing the condition that the financial identification code is inconsistent with the area where the bank is located; and the bank names under the same financial institution identification code are standardized and unified.

Description

Technical Field

[0001] The present invention belongs to the technical field of network data preprocessing, and in particular relates to a data preprocessing method, system, device and readable storage medium. Background Art

[0002] Foreign exchange, in a broad sense, refers to the claims held by the monetary administration in the form of bank deposits, long-term and short-term government securities, and Treasury bills, which can be used when the balance of payments is in deficit. In a narrow sense, foreign exchange is a variety of payment methods expressed in foreign currencies and generally accepted by all countries for international settlement of debts and credits. When banks and individuals handle foreign exchange-related businesses, they should abide by the provisions of the "Personal Foreign Exchange Management Measures". The State Administration of Foreign Exchange and its branches manage and supervise individuals' domestic and cross-border foreign exchange businesses.

[0003] The original and real data of personal foreign exchange business is affected by factors such as policy changes, rule changes, writing habits, etc., and often leads to problems such as inconsistent ID card number lengths, coexistence of ID card and passport numbers, and irregular formats of payee / payee names.

[0004] In work practice, monitoring of personal foreign exchange business is a relatively weak link. The existing personal foreign exchange business system does not yet have a data pre-processing function. The monitoring of personal foreign exchange purchases and external payments is in a relatively scattered and unsystematic state, and there are certain loopholes.

[0005] Therefore, in view of the above technical problems, it is necessary to provide a data preprocessing method based on personal foreign exchange business data.

[0006] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0007] The object of the present invention is to provide a data preprocessing method, system, device and readable storage medium, which can realize faster and more efficient data normalization operation on personal foreign exchange data.

[0008] In order to achieve the above object, a technical solution provided by a specific embodiment of the present invention is as follows:

[0009] In a first aspect, the present invention provides a data preprocessing method, comprising:

[0010] Importing the record to be checked into a preset database table in a specified format, wherein the record to be checked includes an identification number and identification information;

[0011] Read the trusted entry to obtain the trusted identification number and the corresponding trusted identification information;

[0012] Traversing the database table, if the identification number in the record to be checked is consistent with the trusted identification number, updating the identification information corresponding to the identification number in the record to be checked that is the same as the trusted identification number based on the trusted identification information;

[0013] The verified records with the same identification number are written into an independent table entry, and the number of the verified records in the independent table entry is recorded.

[0014] In one or more embodiments of the present invention, the method further comprises:

[0015] Obtaining a set of trusted identification numbers and corresponding trusted identification information in the trusted entry;

[0016] Traversing the database table, if the identification number in the record to be checked is inconsistent with the trusted identification number; or the identification number and identification information in the record to be checked are consistent with the trusted identification number and trusted identification information, then skipping the record to be checked;

[0017] If the identification number in the record to be verified is consistent with the trusted identification number and the corresponding identification information in the record to be verified is inconsistent with the trusted identification information, then updating the identification information in the record to be verified to the trusted identification information;

[0018] Another group of trusted identification numbers and corresponding trusted identification information in the trusted table entry are obtained, and the database table is traversed and updated until all the data in the trusted table entry are involved in data preprocessing.

[0019] In one or more embodiments of the present invention, the method further comprises:

[0020] Based on the number of records to be checked in the database table, multiple counters are maintained, each counter corresponds to a unique record to be checked, and each counter is initialized to a first default value;

[0021] If, when traversing the database table, the identification number in the record to be checked is consistent with the trusted identification number in the acquired trusted table entry, the counter corresponding to the record to be checked is set from the first default value to the second default value.

[0022] In one or more embodiments of the present invention, the method further comprises:

[0023] When traversing the database table, obtaining the value of the counter corresponding to each record to be checked;

[0024] If the value of the counter is the second default value, skip checking the record to be verified.

[0025] In one or more embodiments of the present invention, the method further comprises:

[0026] Set an index for each record to be checked in the database table;

[0027] Based on the preset screening rules, the records to be checked corresponding to the abnormal identification number are marked and filtered so that the records to be checked corresponding to the abnormal identification number do not participate in the subsequent update of the database table.

[0028] In one or more embodiments of the present invention, the screening rules include:

[0029] Divide the identification numbers in the database table into segments, and set interval thresholds for each segment;

[0030] If each segment of the identification number in the record to be checked is within the corresponding interval threshold, the identification number is normal;

[0031] If there is at least one segment of the identification number in the record to be checked that is not within the corresponding interval threshold, the identification number is abnormal.

[0032] In one or more embodiments of the present invention, the method further comprises:

[0033] Generate a verification code corresponding to the identification number in the record to be verified based on a preset verification formula;

[0034] If the verification code is consistent with the initial verification code in the record to be verified, the identification number is normal;

[0035] If the verification code is inconsistent with the initial verification code in the record to be verified, the identification number is abnormal.

[0036] In a second aspect, the present invention provides a data preprocessing system, comprising:

[0037] An import module, used to import the record to be checked into a preset database table in a specified format, wherein the record to be checked includes an identification number and identification information;

[0038] A reading module is used to read the trusted table entry and obtain the trusted identification number and the corresponding trusted identification information;

[0039] An updating module, configured to traverse the database table, and if the identification number in the record to be checked is consistent with the trusted identification number, update the identification information corresponding to the identification number in the record to be checked that is the same as the trusted identification number based on the trusted identification information;

[0040] The recording module is used to write the verified records with the same identification number into an independent table item, and record the number of the verified records in the independent table item.

[0041] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the data preprocessing method by executing the computer instructions.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data preprocessing method.

[0043] Compared with the prior art, the data preprocessing method provided by the present invention, by exporting personal foreign exchange data, decouples the foreign exchange data from the platform / system where the foreign exchange data is stored, thereby avoiding the impact on system functions caused by data processing. During the import process, the identification number and identification information are matched with the preset header in the database table, which is convenient for subsequent operations, and at the same time enables the user terminal to clearly understand the meaning of each data in the database table during the visualization process. Based on the trusted table items, the identity information in the corresponding database table can be updated, and the situation where the name writing under the same identity document number has errors can be repaired to ensure that the same identity document number corresponds to the same name; the situation where the financial identification code is inconsistent with the bank's area can be distinguished; the bank names under the same financial institution identification code are standardized and unified. The setting of independent table items can facilitate user classification search and further improve the flexibility of this solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a schematic diagram of a data preprocessing scenario in one embodiment of the present invention;

[0046] Figure 2 is a flow chart of a data preprocessing method in one embodiment of the present invention;

[0047] Figure 3 is a structural block diagram of a data preprocessing system in one embodiment of the present invention;

[0048] Figure 4is a structural block diagram of an electronic device in one embodiment of the present invention;

[0049] Figure 5 It is a schematic diagram of database table division in one embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0052] Please refer to Figure 1 , which is a schematic diagram of an application scenario of the data preprocessing method provided by the present invention under one implementation mode, and the scenario specifically includes: a data import module 101, a data processing module 102 and a result display module 103.

[0053] It should be noted that communication connections are provided between the data import module 101, the data processing module 102 and the result display module 103, and the communication network derived from the above communication connection may include various connection types, including but not limited to: wired connection, wireless connection or optical fiber cable connection, etc. At the same time, the communication network may be a local area network, a metropolitan area network, a wide area network or any combination of the three.

[0054] In the specific implementation of the present invention, firstly, the records to be checked are imported into the warehouse through the data import module 101, and the processing results are returned after the data processing module 102 automatically processes the records, and the results and processing flow are visualized based on the result display module 103. Among them, the data import module 101 and the result display module 103 can be completed with the help of Navicat for MySQL, and the data processing module 102 is connected to the MySQL database through JDBC using the algorithm in Java to operate.

[0055] Specifically, the result display module 103 is used to form a visual view to display the preprocessing process and results of the present invention. The visual view may include but is not limited to: a login interface, a result display interface, a data processing interface, etc.

[0056] The login interface mainly implements the verification of username and password, as well as the random generation and verification of verification codes. At the same time, it supports clearing and resetting the information entered in the text input box. The focus is on implementing the random generation and verification of verification codes. When the user clicks the "Login" button, the system will call the doGet() and doPost() methods rewritten in the UserServlet class to receive the data passed from the view (login.jsp), and perform related operations through login(), logout(), and list(), and call the corresponding behavior from the model (UserDao) to process the data, and finally encapsulate the obtained results into logout() and return them to the view.

[0057] The data processing module 102 is used to perform database connection testing, conduct legality verification based on the field structure of the identification number, distinguish between different types of identification numbers, standardize the identification information, and classify the verified records.

[0058] It should be noted that, since the data processing module 102 is mostly operated by connecting to the MySQL database through JDBC using the algorithm in Java, in the application scenario of the present invention, the program code written by the user should also be translated into bytecode based on the compiler and loaded into the memory, and the platform operating environment is carried out to wait for execution. The platform operating environment consists of a class loader and a virtual machine. The class loader receives the bytecode and then sends it to the virtual machine. After the bytecode enters the virtual machine, it will be translated into machine code by the code interpreter and the just-in-time compiler and handed over to the platform operating system, and then transmitted to the hardware level for execution through the operating system.

[0059] It should be noted that the embodiments of the present invention do not limit the type of computer language used. Considering security, robustness, and portability (cross-platform), the Java language can be preferably used. Compared with C++, Java has absorbed various advantages of C++ while abandoning the difficult-to-understand concepts such as multiple inheritance and pointers in C++. This gives Java new features that are simple and easy to use while ensuring powerful functions: Java provides break and continue statements, as well as exception handling, to replace the go to statement; Java has no structure, and arrays and strings are designed as objects, so pointers are not needed; Java eliminates the characteristics of C++ operator overloading and multiple inheritance, and does not use master files, which eliminates the steps of preprocessing programs and simplifies operations; Java can automatically handle object references and indirect references, and automatically collect useless units, which makes users do not need to worry about storage management issues, so that they can focus more on program development.

[0060] It should also be noted that the data preprocessing method of the embodiment of the present invention can be applied to the data preprocessing system of the embodiment of the present invention. The data preprocessing system can be configured in a terminal. The terminal can include but is not limited to a PC (Personal Computer), a PDA (tablet computer), a smart phone, a smart wearable device, and the like.

[0061] Please refer to Figure 2 FIG. 1 is a flow chart of a data preprocessing method according to an embodiment of the present invention. The data preprocessing method specifically comprises the following steps:

[0062] S201: Importing the record to be checked into a preset database table in a specified format, wherein the record to be checked includes an identification number and identification information;

[0063] It should be noted that the identification number in this article may be a number used to identify user information or transaction information in personal foreign exchange data, which may include but is not limited to: ID number, passport number, financial institution identification code, etc. The identification information is also the user information and / or transaction information involved in personal foreign exchange business, which may include but is not limited to: name, region where the bank handling the business is located, name of the bank handling the business, etc.

[0064] The identification number in the record to be checked may correspond to at least one item of identification information, for example, the ID number as the identification number and the user name as the identification information; the financial institution identification code as the identification number and the region where the bank that handles the business is located and the name of the bank that handles the business as the identification information.

[0065] It should also be noted that the embodiment of the present invention does not limit the source of the records to be checked, and the source of the records to be checked may include but is not limited to: the State Administration of Foreign Exchange online service platform (ASOne), the balance of payments reporting system and the account management system.

[0066] The preset database table in the present invention should be able to be completely written with all the records to be checked. It is understandable that if there are n records to be checked that need to be written into the table, and each record to be checked has m items of data. Since it is also necessary to set the table header corresponding to each data in the record to be checked, the number of rows and columns of the preset database table should satisfy: one of the number of rows and the number of columns is greater than m+1, and the other is greater than n. The embodiment of the present invention does not limit other settings of the preset database table.

[0067] It is understandable that the data imported into the database table may have occasional defects other than the format, such as the ID number area is the passport number, the ID number is missing digits, etc. Such data will inevitably not be able to match the corresponding trusted identification number and trusted identification information in the subsequent data format unification. Therefore, in an exemplary embodiment of the present invention, it also includes: setting an index for each record to be verified in the database table; based on the preset screening rules, marking and filtering the records to be verified corresponding to the abnormal identification number, so that the records to be verified corresponding to the abnormal identification number do not participate in the subsequent update of the database table. Before starting to update the data of the records to be verified, first update the abnormal data, which can reduce the workload of subsequent data processing and improve data monitoring efficiency.

[0068] Using indexes can greatly improve the efficiency of queries. An index is a special data structure that can quickly locate specific records in a data table, thereby speeding up queries. To use indexes in Java, you can specify the index to be used through SQL statements. For example, you can use a SELECT statement in conjunction with the INDEX keyword to specify the use of an index, or you can create an index for a specific column in the definition of a table so that the database system can locate relevant data more quickly when executing a query. Using an index is usually faster than traversing without an index, especially when dealing with large amounts of data. When there is no index, the database system needs to scan the entire table row by row to find records that meet the criteria, while using an index allows the database system to directly locate records that meet the criteria, greatly reducing query time.

[0069] Specifically, the screening rules include: dividing the identification number in the database table into segments and setting an interval threshold for each segment; if each segment of the identification number in the record to be verified is within the corresponding interval threshold, the identification number is normal; if there is at least one segment of the identification number in the record to be verified that is not within the corresponding interval threshold, the identification number is abnormal.

[0070] It should be noted that the identification numbers related to personal foreign exchange business are almost all composed of segmented splicing. For example, the 1st and 2nd digits of the ID card represent the code of the province, autonomous region, or municipality directly under the central government; the 3rd and 4th digits represent the code of the prefecture-level city or autonomous prefecture; the 5th and 6th digits represent the code of the county, autonomous county, county-level city, or district; the 7th and 14th digits represent the date of birth; the 15th and 16th digits represent the code of the local police station; the 17th digit represents the gender, with an odd number for males and an even number for females; and the 18th digit is the check code. Therefore, based on different types of identification numbers, the interval thresholds should be defined based on the meaning of each segment, so as to achieve the screening of abnormal information.

[0071] For example, based on the administrative division code, the interval set for the first two digits of the ID card number is set to {11,12,13,14,15,21,22,23,31,.....,91}. If the first two digits of the ID card number in the record to be verified are not in the above set, then the record is an abnormal record.

[0072] Similarly, taking the year 2025 as an example, based on the normal life span of humans and the age of customers who can handle business independently recognized by banks, ID card numbers with birth years of digits 7-10 less than 1900 and greater than 2009 are excluded. At the same time, ID card numbers with digits 11-12 less than 01 or greater than 12 are excluded; ID card numbers with digits 13-14 greater than 31 or less than 01 are excluded.

[0073] It should also be noted that the above-mentioned screening rules can only screen out the records to be verified that are obviously unreasonable. In order to further determine the correctness of the identification numbers in each record to be verified, the above-mentioned screening rules also include: based on a preset verification formula, generating a verification code corresponding to the identification number in the record to be verified; if the verification code is consistent with the initial verification code in the record to be verified, the identification number is normal; if the verification code is inconsistent with the initial verification code in the record to be verified, the identification number is abnormal.

[0074] It is understandable that the verification formulas used for different types of identification numbers may be different, and the embodiment of the present invention does not limit the selection of the verification formula.

[0075] For example, the following verification formula can be used for the ID card number:

[0076] S=Sum(Ai*Wi)

[0077] Wi=2^(i-1)(MOD11)

[0078] Y=mod(S,11)

[0079] Among them, i represents the serial number of each digit in the ID card number array: 0, 1, 2, ..., 16; Ai represents the number corresponding to the serial number position i.

[0080] Based on the calculated value of Y, the check value corresponding to the current ID number can be obtained. In the ID card, the original check value is the number or letter recorded in the 18th position. Therefore, if the check value obtained based on the calculated Y value is consistent with the original check value, the current ID number is considered normal; conversely, if the check value obtained based on the calculated Y value is inconsistent with the original check value, the current ID number is considered abnormal.

[0081] S202: Read the trusted entry to obtain the trusted identification number and the corresponding trusted identification information;

[0082] It should be noted that before starting this step, the database connection should also be tested. This mainly tests whether the configuration of MySQL, JDBC, etc. is complete, and whether there are errors in the code related to the database connection.

[0083] When interacting with a JDBC data source, an exception named SQLException will be thrown when an error occurs. If there is no exception handling statement in the code, the IDE will report an error. Therefore, in one embodiment of the present invention, an exception handling class is imported to capture and handle exceptions related to SQL operations in the code.

[0084] Specifically, importing the SQLException class allows programmers to more easily capture and handle possible database exceptions when writing code that interacts with the database, such as database connection timeout, SQL syntax errors, insufficient database access permissions, etc. By capturing these exceptions, the program can handle problems accordingly, such as outputting error information, rolling back transactions, closing connections, etc., to ensure the stability and reliability of the program.

[0085] It should also be noted that, in one embodiment, the trusted table items can be obtained based on networking with the public security system, judicial system, household registration management system, customs system, etc. The data types in the trusted table items should at least include the data types in the records to be verified.

[0086] S203: traversing the database table, if the identification number in the record to be checked is consistent with the trusted identification number, updating the identification information corresponding to the identification number in the record to be checked that is the same as the trusted identification number based on the trusted identification information;

[0087] In an exemplary embodiment, updating the database table based on the trusted identification number and the corresponding trusted identification information includes: obtaining a trusted identification number and the corresponding trusted identification information in the trusted table item; traversing the database table, if the identification number in the record to be verified is inconsistent with the trusted identification number; or the identification number and identification information in the record to be verified are consistent with the trusted identification number and the trusted identification information, then skipping the record to be verified; if the identification number in the record to be verified is consistent with the trusted identification number and the identification information in the record to be verified is inconsistent with the trusted identification information, then updating the identification information in the record to be verified to the trusted identification information; obtaining another set of trusted identification numbers and the corresponding trusted identification information in the trusted table item, traversing and updating the database table until all the data in the trusted table item are updated.

[0088] It is understandable that updating the data in the database table based on each piece of trusted data can indeed complete the update of the data format and data content. However, corresponding to any piece of trusted data, all the records to be verified in the database table need to be traversed, which will greatly affect the verification efficiency. Therefore, the present invention further provides the following technical solutions:

[0089] Specifically, in another exemplary embodiment of the present invention, based on the number of records to be verified in the database table, multiple counters are maintained, each counter corresponds to a unique record to be verified, and each counter is initialized to a first default value; if, when traversing the database table, the identification number in the record to be verified is consistent with the trusted identification number in the obtained trusted table item, then the counter corresponding to the record to be verified is set from the first default value to the second default value.

[0090] The counter is used here as the identifier of the record. Since the default value of the counter will change when the identification number in the record to be verified is consistent with the trusted identification number in the obtained trusted table item, if a record corresponds to the first default value, it means that the record has not been verified, so it is necessary to traverse it to confirm whether the identification number of the record to be verified is consistent with the current trusted identification number. On the other hand, if the counter corresponding to the currently traversed record to be verified is the second default value, it means that the record has been verified at this time, so the identification number and identification information corresponding to the record must be different from the currently obtained feasible identification number and trusted identification information. The record can be skipped directly.

[0091] S204: Write the verified records with the same identification number into an independent table entry, and record the number of the verified records in the independent table entry.

[0092] It should be noted that in the embodiment of the present invention, the database table space can be further allocated before the records to be verified are imported into the database, so as to reduce the computing pressure brought by the large amount of data, subdivide the data to be processed, facilitate debugging, and provide reference for subsequent work. The allocation method can be dynamically adjusted based on the actual usage scenario, and the embodiment of the present invention does not limit this. Specifically, the database table items can be divided into Figure 5 As shown, it includes: original data table, transaction data under Chinese ID card number, transaction data under passport number, non-standard data in CNIDCARD table, non-standard data in PASSPORT table, normalized data and transaction record statistics in the same time period.

[0093] Furthermore, the data at different stages before and after processing are written into different table items to achieve partitioned storage of data for easy statistical query. For example, in the data processing process, the "declaration number", "payer ID number", and "payer name" fields are obtained, and the names under the same ID number are judged to be consistent (standardized). If not, the entire record is output to the new table PASSPORT1 / CNIDCARD1 for processing.

[0094] By writing the verified records with the same identification number into an independent table item, a list of key personnel with frequent transactions can be screened out to provide a reference for subsequent analysis. In a specific embodiment, the step can be: adding a counter count = 0; traversing the verified database table, when the ID card number is the same, outputting all transaction records under the ID card to an independent table item, and returning a count value at the same time, so that the number of records can be intuitively seen.

[0095] In another embodiment, the scheme can also be combined with other restrictions to write independent table items. For example, set one month as a reference standard, and traverse all transaction records in a certain month in chronological order and in the order of ID number. Add a counter count = 0. When the ID numbers are the same, compare whether the payee name, card number and other information are consistent. If there is a consistent situation, output all transaction records of the same payee under the ID card to the OnetoOne table, and return a count value at the same time.

[0096] Please refer to Figure 3 As shown, based on the same inventive concept as the aforementioned data preprocessing method, an embodiment of the present invention provides a data preprocessing system 300 , which includes: an import module 301 , a reading module 302 , an update module 303 and a recording module 304 .

[0097] Specifically, the import module 301 is used to import the record to be verified into a preset database table in a specified format, and the record to be verified includes an identification number and identification information; the reading module 302 is used to read the trusted table item, obtain the trusted identification number and the corresponding trusted identification information; the update module 303 is used to traverse the database table, and if the identification number in the record to be verified is consistent with the trusted identification number, the identification information in the record to be verified is updated based on the trusted identification information; the recording module 304 is used to write the verified record with the same identification number into an independent table item, and record the number of the verified records in the independent table item.

[0098] Please refer to Figure 4As shown, an embodiment of the present invention further provides an electronic device 400, which includes at least one processor 401, a memory 402 (for example, a non-volatile memory), a memory 403, and a communication interface 404, and the at least one processor 401, the memory 402, the memory 403, and the communication interface 404 are connected together via an internal bus 405. The at least one processor 401 is used to call at least one program instruction stored or encoded in the memory 402, so that the at least one processor 401 performs various operations and functions of the data preprocessing method described in various embodiments of this specification.

[0099] In an embodiment of the present specification, the electronic device 400 may include, but is not limited to, a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile electronic device, a smart phone, a tablet computer, a cellular phone, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable electronic device, a consumer electronic device, and the like.

[0100] An embodiment of the present invention further provides a computer-readable medium carrying computer-executable instructions. When the computer-executable instructions are executed by a processor, they can be used to implement various operations and functions of the data preprocessing method described in various embodiments of this specification.

[0101] The computer-readable medium in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0102] In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0103] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, systems, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

[0106] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0107] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A data preprocessing method, characterized in that: include: Importing the record to be checked into a preset database table in a specified format, wherein the record to be checked includes an identification number and identification information; Read the trusted entry to obtain the trusted identification number and the corresponding trusted identification information; Traversing the database table, if the identification number in the record to be checked is consistent with the trusted identification number, updating the identification information corresponding to the identification number in the record to be checked that is the same as the trusted identification number based on the trusted identification information; The verified records with the same identification number are written into an independent table entry, and the number of the verified records in the independent table entry is recorded.

2. The data preprocessing method according to claim 1, characterized in that: The method further comprises: Obtaining a set of trusted identification numbers and corresponding trusted identification information in the trusted entry; Traversing the database table, if the identification number in the record to be checked is inconsistent with the trusted identification number; or the identification number and identification information in the record to be checked are consistent with the trusted identification number and trusted identification information, then skipping the record to be checked; If the identification number in the record to be verified is consistent with the trusted identification number and the corresponding identification information in the record to be verified is inconsistent with the trusted identification information, then updating the identification information in the record to be verified to the trusted identification information; Another group of trusted identification numbers and corresponding trusted identification information in the trusted table entry are obtained, and the database table is traversed and updated until all the data in the trusted table entry are involved in data preprocessing.

3. The data preprocessing method according to claim 2, characterized in that: The method further comprises: Based on the number of records to be checked in the database table, multiple counters are maintained, each counter corresponds to a unique record to be checked, and each counter is initialized to a first default value; If, when traversing the database table, the identification number in the record to be checked is consistent with the trusted identification number in the acquired trusted table entry, the counter corresponding to the record to be checked is set from the first default value to the second default value.

4. The data preprocessing method according to claim 3, characterized in that: The method further comprises: When traversing the database table, obtaining the value of the counter corresponding to each record to be checked; If the value of the counter is the second default value, skip checking the record to be verified.

5. The data preprocessing method according to claim 1, characterized in that: The method further comprises: Set an index for each record to be checked in the database table; Based on the preset screening rules, the records to be checked corresponding to the abnormal identification number are marked and filtered so that the records to be checked corresponding to the abnormal identification number do not participate in the subsequent update of the database table.

6. The data preprocessing method according to claim 5, characterized in that: The screening rules include: Divide the identification numbers in the database table into segments, and set interval thresholds for each segment; If each segment of the identification number in the record to be checked is within the corresponding interval threshold, the identification number is normal; If there is at least one segment of the identification number in the record to be checked that is not within the corresponding interval threshold, the identification number is abnormal.

7. The data preprocessing method according to claim 5, characterized in that: The method further comprises: Generate a verification code corresponding to the identification number in the record to be verified based on a preset verification formula; If the verification code is consistent with the initial verification code in the record to be verified, the identification number is normal; If the verification code is inconsistent with the initial verification code in the record to be verified, the identification number is abnormal.

8. A data preprocessing system, characterized in that: include: An import module, used to import the record to be checked into a preset database table in a specified format, wherein the record to be checked includes an identification number and identification information; A reading module is used to read the trusted table entry and obtain the trusted identification number and the corresponding trusted identification information; An updating module, configured to traverse the database table, and if the identification number in the record to be checked is consistent with the trusted identification number, update the identification information corresponding to the identification number in the record to be checked that is the same as the trusted identification number based on the trusted identification information; The recording module is used to write the verified records with the same identification number into an independent table item, and record the number of the verified records in the independent table item.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data preprocessing method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data preprocessing method according to any one of claims 1 to 7.

Images