Mass data storage and statistics method

Abstract

The application provides a mass data storage and statistics method, which uploads data and stores data in a database, and is divided into two stages according to different types of operations on the data, and an original database and a database are respectively set for each stage, so as to reduce the probability of database error caused by reading and writing the same database table in the same database during mass data storage; the method uses an Oracle database storage process Stored Procedure to realize a statistics device to complete the statistical operation on the data, and the statistics device realized based on the storage process is directly stored in the database, and a connection with the database does not need to be specially established from the outside each time the data statistical operation is performed, so that the statistical efficiency is improved, and the error probability is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mass data processing, in particular to a mass data storage and statistical method. BACKGROUND

[0002] In the collection and sharing of mass data of some large scale, possibly across provinces or countries, the data reported by the sub-departments needs to be statistically analyzed in real time. The statistics needs to summarize and analyze all the data reported by all the sub-departments involved, and the amount of data involved is very large. For example, as shown in the embodiments of Figure 1 and Figure 2 , the current system has 200 sub-departments, and the department codes and department names in Figure 2 are used to mark the sub-departments. Each department will submit data to the database according to the contents specified in different requirement files, and each requirement file corresponds to a data table, Figure 2 , which is a data table established according to requirement file 1 in Figure 1 . Each department can only submit data for one requirement file at a time, and the system will generate a unique batch number to mark the data each time the department submits data, as shown in Figure 2 , where the "task batch number" indicates which batch the data belongs to.

[0003] After storage, each requirement file corresponds to a library table, and all data submitted based on the requirement file is stored in the library table. Assuming that each department submits 1000 pieces of data for a requirement file at a time, then 200 departments submitting data for a requirement file will have 200*1000 pieces of data. If each department reports 5 times a day, then 200*1000*5 pieces of data will be generated every day. Once some departments upload data incorrectly or update data, they will upload data twice or three times to update the corresponding data in the database. During the data submission process, the data will be checked for rules, that is, the data content of the uploaded data will be checked according to the pre-set rules, and the data that does not meet the checking rules will be marked. The checked data is then cleaned, such as converting some date fields to a pre-set format, and then storing the cleaned data in the database. In the existing submission method, the Kettle tool (KDE Extraction, Transportation, Transformation and Loading Environment) is used, which is an open source ETL (Extract-Load-Transform, data warehouse) tool. The data is checked for rules and cleaned.

[0004] From the perspective of management, the data uploaded every day needs to be counted, and then the statistical data is reflected on the statistical page. The data that needs to be counted includes: reporting quantity, doubtful quantity, warehousing quantity, and updating quantity. Among them, the reporting quantity is the amount of data successfully submitted to the database, the doubtful quantity is the number of data that does not meet the verification rules in the process of submitting data to the database. The warehousing quantity is the number of new data stored in the library after the data meeting the verification rules are cleaned. The updating quantity is the data amount of updating the data that has been submitted to the business library. The data relationship is: reporting quantity=doubtful quantity+warehousing quantity+updating quantity.

[0005] The current data counting method adopts the log recording method, that is, the data amount of each reporting is recorded in real time through the log method in the process of executing the reporting process: reporting quantity, warehousing quantity, updating quantity, doubtful quantity, and reporting once. When the statistical data needs to be updated on the statistical page, the data recorded in the daily data table is queried and summarized again.

[0006] However, based on the existing data counting method, it is more suitable for the case of less reporting data and less frequent reporting. However, if the reporting data amount is huge, each data needs to modify the log data table, so frequent modification of the log data table will often cause table locking or reporting interruption, and the data cannot be normally inserted into the database. Moreover, the existing statistical method depends on the interface or third-party data cleaning tool. When the interface or third-party data cleaning tool is abnormal, the problem that the data may have been reported but the log is not updated correctly may occur, resulting in an error result when counting the data according to the log data table. SUMMARY

[0007] In order to solve the problem that the existing third-party tool-based statistical method in the uploading process will cause database errors or statistical result errors when counting the massive and frequently modified data, the present application provides a data counting method, which separates the data reporting and data amount recording steps to ensure the accuracy of the statistical result and will not cause database errors.

[0008] The technical scheme of the present application is as follows: a mass data storage and counting method, comprising the following steps:

[0009] S1: constructing an original library and a business library;

[0010] S2: creating a storage and counting tool;

[0011] A cleaning timer is created, and a data cleaning tool is started based on the cleaning timer to perform a data cleaning operation on the data in the original library;

[0012] A statisticator is created, which is implemented based on a storage procedure of an Oracle database and operates on data in the original library and the business library, including query, statistics and summary operations;

[0013] A statistics timer is created, and the statisticator is started periodically based on the statistics timer;

[0014] The statisticator and the statistics timer are stored in the business library, and the business library and the original library are connected through DBlink to complete data transmission;

[0015] S3: Confirm the number of requirement files, and establish corresponding original library tables and business library tables in the original library and the business library respectively for each requirement file;

[0016] The original library table includes: original library table data ID, department name, department code, verification state, batch number and data content;

[0017] The batch number corresponding to the data content uploaded by the same sub-department each time is the same; each piece of data content corresponds to a data state; the original library table data ID uniquely marks each piece of data content in the original library table; the verification state records the result of this data in the rule verification, including: compliance and non-compliance;

[0018] The business library table includes: business library table data ID, department name, department code, batch number and data content;

[0019] The business library table data ID uniquely marks each piece of data content in the business library table;

[0020] S4: Real-time receive data uploaded by each batch of sub-departments to the original library, and generate the batch number for this batch of uploaded data;

[0021] S5: Based on the verification tool, the uploaded data is subjected to rule verification, and after the verification state corresponding to the uploaded data is generated, the data is saved to the corresponding original library table according to the requirement file corresponding to this batch of uploaded data;

[0022] The verification state corresponding to the data content that meets the verification rule is marked as: compliance;

[0023] The verification state corresponding to the data content that does not meet the verification rule is marked as: non-compliance;

[0024] S6: Based on the cleaning timer, start the data cleaning tool according to the preset data cleaning operation start time;

[0025] In all the newly added data of the original library table, find the data with the qualified verification state, perform the data cleaning operation on these data, and obtain the cleaned data;

[0026] S7: confirming a positioning field;

[0027] The data contents corresponding to the positioning field are the same, indicating that they are the same data content;

[0028] S8: finding a corresponding business library table in the business library according to the cleaned data and the requirement file corresponding thereto, and querying in the business library table based on a preset positioning field to confirm whether it already exists;

[0029] If it already exists, it is determined to be data that has been stored, and is recorded as stored data;

[0030] Otherwise, it is determined to be data that has not been stored, and is recorded as new data;

[0031] S9: storing the cleaned data into the corresponding business library table in the business library;

[0032] Overwriting and updating the corresponding entry of the business library table with the stored data;

[0033] Inserting the new data into the business library table;

[0034] S10: starting the statistics of the data by the statistics according to the statistics timer and the preset statistics time, specifically including the following steps:

[0035] a1: obtaining a name set of all original library tables in the original library, recorded as an original library table name set;

[0036] And putting the original library table name set into a cursor;

[0037] a2: circulating the cursor to obtain each table name, and based on the original library table name, counting all data of each table;

[0038] According to the batch number, the reporting quantity of each batch is obtained;

[0039] The data quantity of all data in each original library table is calculated, that is, the reporting quantity corresponding to the original library table;

[0040] a3: finding all data with an unqualified verification state in each original library table;

[0041] According to the batch number, the corresponding question quantity of each batch number is obtained;

[0042] According to the original library table, the corresponding amount of doubt is obtained;

[0043] a4: obtain the name set of all the business library tables in the business library, denoted as: business library table name set;

[0044] And put the business library table name set into the cursor;

[0045] a5: loop the cursor to obtain each table name, and based on the business library table name, count all the data in each business library table;

[0046] According to the batch number, the corresponding warehouse quantity of each batch number is obtained;

[0047] According to the business library table name, the corresponding warehouse quantity of the business library table is obtained;

[0048] a6: calculate the update amount according to the batch number and the business library table respectively:

[0049] Update amount = reported amount - doubt amount - warehouse quantity;

[0050] S11: display the data after statistics in the statistics page;

[0051] S12: loop the steps S4~S11.

[0052] Further characterized in that:

[0053] The business library and the original library are built on different servers;

[0054] When the corresponding relationship between the original library table and the business library table is N:1, N≥2;

[0055] The business library table includes a field: type, which corresponds to different original library table names with different type values;

[0056] When calculating the update amount in the statistics calculator, according to the type field in the business library table, all the original library tables corresponding to the business library table are determined, and the reported amount and the amount of doubt in the original library table are obtained.

[0057] The application provides a mass data storage and statistics method, which uploads data and stores data in a database, and is divided into two stages according to different types of data operations, a database is set for each stage, and the probability of database error caused by reading and writing the same database table in the same database during mass data storage is reduced; the method uses an Oracle database storage process Stored Procedure to realize a statistics device to complete the statistics operation on the data, the statistics device realized based on the storage process is directly stored in the database, and a connection between the database and the outside is not needed each time the data statistics operation is performed, so that the statistics efficiency is improved, and the error probability is reduced; because the statistics of mass data needs to involve a large number of original library table and business library table operations, if the retrieval and statistics function is realized based on a SQL statement, a large number of SQL statements are needed, the statistics timer and the statistics device are stored in the business library in the method, the statistics device is awakened in the database through the statistics timer when the statistics device needs to be started, a large number of SQL statements do not need to be transmitted to the database from the outside for running, the error probability and the database access frequency are reduced, and the query performance is improved; meanwhile, through DBlink, the statistics device stored in the business library can realize the retrieval of the original library, the efficiency is high, the data does not need to be accessed from the outside, and the data security can be ensured; in the method, the data in different stages is recorded and stored through setting original library tables and business library tables, only when the data needs to be statistically based on the statistics device, the data states in different stages are statistically separated, compared with the method of recording the real-time statistics in the prior art based on a log data table, the error probability of the database table is greatly reduced, the data uploading and statistics accuracy are improved, and the method is especially suitable for the scene of mass data, frequent data reporting, irregular data, large data volume, frequent database table modification and the like. In the method, a storage process with a cursor is used, the database table to be statistically read into the cursor cache according to a preset statistics time point, the traversal of the database table based on the cursor greatly improves the access speed and efficiency of the database table, and then the data statistics speed is improved, the probability of simultaneously accessing the database table with the user operation and the storage operation after data cleaning is reduced, and then the error probability of the database table is reduced, and the method is especially suitable for the operation of the database table storing mass data. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 An example of a submission process in the prior art;

[0059] Figure 2 An example of a database in the prior art;

[0060] Figure 3 A schematic diagram of a database structure in the method;

[0061] Figure 4 An example of a data submission process in the method;

[0062] Figure 5 The original library in the present method is represented by example 1.

[0063] Figure 6 The original library in the present method is represented by example 2.

[0064] Figure 7 The business library in the present method is represented by example.

[0065] Figure 8 Part of the code example in the timer. DETAILED DESCRIPTION

[0066] The present application includes a mass data storage and statistical method, which includes the following steps.

[0067] S1: Constructing the original library and the business library.

[0068] In specific applications, in order to reduce the database burden and reduce the requirements on the server performance, the business library and the original library can be built on different servers.

[0069] S2: Creating a storage and statistical tool.

[0070] A cleaning timer is created, and a data cleaning tool is started periodically based on the cleaning timer to perform data cleaning operations on the data in the original library.

[0071] The cleaning timer can be implemented based on the timer of the Oracle database, or can be implemented using a third-party tool. In the present method, the Kettle tool is used to clean the data, and the Kettle tool itself has a timer function. In order to reduce the code complexity, the Kettle self-timer is used to implement the cleaning timer in the present application.

[0072] A statisticator is created, which is implemented based on the storage process of the Oracle database, and which operates on the data in the original library and the business library, including query, statistics and summary operations; a statistical timer is created, and the statisticator is started periodically based on the statistical timer; the statisticator and the statistical timer are stored in the business library, and the business library and the original library are connected through DBlink to complete data transmission.

[0073] Because the number of sub-departments is large in the use scenario of the present method, the data submission time is uncertain, the data amount submitted each time is large, and the number of database tables in the original library and the database is also large. The read and write operations on the database tables caused by the submission are already very frequent, so in order to reduce the error rate of the database tables, it is necessary to reduce the probability of errors caused by simultaneous operations on the database tables. The present method reduces the frequency of read and write operations on the data in the database by external commands, so the statisticator implemented based on the storage process is used to implement the statistical operations on the database tables.

[0074] A stored procedure is a set of SQL statements designed to perform a specific function; it is a named block of statements written in SQL and consisting of flow control rules. For example... Figure 3 As shown, this method uses a stored procedure with a cursor to implement the statistical analyzer. The compiled statistical analyzer is stored in the business database. During data statistics, access to the data in the original database is achieved through DBlink. A statistical timer is used to wake up the statistical analyzer from within the database. Compared with existing methods, this not only results in faster access to the database but also significantly improves statistical speed and efficiency by placing the database table into a cursor for traversal each time, reducing the probability of simultaneous database table access with other operations. Furthermore, no external connection to the database is needed for each data statistical operation, further improving efficiency. It also avoids frequent database access by external users, enhancing data security.

[0075] The specific method for implementing the statistical analyzer based on stored procedures can be implemented using existing Oracle technology. Figure 8 This is a code example of a timer.

[0076] S3: Confirm the number of required documents, and create corresponding original database tables and business database tables in the original database and business database for each required document.

[0077] For example: the required document name is: Administrative License. Figure 5 and Figure 7 The original database tables and business database tables established for "administrative licensing";

[0078] Figure 5 The original database table name is: Original Database Natural Person Administrative Permit - JC_ZRR_SGS_XZXK;

[0079] Figure 7 The business database table name is: Business Database-YW_L_SGSXZXK;

[0080] The original database table includes: original database table data ID, department name, department code, verification status, batch number, and data content. Figure 5 In this embodiment, the fields in the data content include: name, ID card number, document number, license certificate, and date of license decision.

[0081] The batch number corresponding to each data content uploaded by the same sub-department is the same; each data content corresponds to a data status; the original database table data ID uniquely identifies each data content in the original database table; the verification status records the result of this data in the rule verification, including: compliant and non-compliant.

[0082] The business library table includes: business library table data ID, department name, department code, batch number and data content.

[0083] The business library table data ID uniquely marks each data content in the business library table.

[0084] When the correspondence between the original library table and the business library table is N:1, where N≥2;

[0085] The business library table includes the field: type, which corresponds to different original library table names with different type values.

[0086] When calculating the update amount in the statistics, all original library tables corresponding to the business library table are counted according to the type field, and the reporting amount and the doubtful amount corresponding to the original library table are obtained by aggregation.

[0087] Generally, each requirement file has a corresponding database table in the original library and the business library, but in some cases, the requirement file may have two or more corresponding original library tables in the original library table. That is, the correspondence between the original library table and the business library table is not necessarily 1:1.

[0088] The name of the requirement file is: administrative license, which corresponds to the original library table in the original library Figure 5 in addition to the original library table Figure 6 , Figure 6 The original library table name is: original library administrative license JC_FR_SGS_XZXK. That is, Figure 5 and Figure 6 The data in and is submitted separately according to the identity of the submitter, with the data of legal persons and natural persons separated. However, once submitted to the database, the data of the two types of submitters is processed together for later statistics, so the data statistics is also done together.

[0089] Figure 5 and Figure 6 The data needs to be merged into the same business library table in the business library, that is, Figure 7 the business library table shown. Figure 7 In the business library table, the data source is distinguished by the field "type".

[0090] S4: Real-time reception of data uploaded to the original library by each batch of sub-departments, generation of a batch number for the data uploaded this time, corresponding to Figure 5 "task batch number" in

[0091] S5: Based on the verification tool, the uploaded data is checked according to the rules, and the verification state corresponding to the uploaded data is generated. According to the requirement file corresponding to the batch of uploaded data, the data is saved to the corresponding original library table;

[0092] Mark the check status corresponding to the data content meeting the check rule as: compliant;

[0093] Mark the check status corresponding to the data content not meeting the check rule as: non-compliant.

[0094] Referring to Figure 5 The column of "status" in the middle, the compliant data record is 0, and the non-compliant data record is 999. In actual work, the data submitter will be reminded by the system for non-compliant data to ensure that he can know the data status.

[0095] Steps S4-S5 complete the process of uploading a batch of original data to the original library. The uploading time of the sub-department is random, so the database will start the subsequent uploading process according to the uploading action of the sub-department. Data cleaning is carried out according to the preset time and frequency based on the cleaning timer.

[0096] S6: Start the data cleaning tool based on the cleaning timer according to the preset data cleaning operation start time.

[0097] Among all the new data of the original library table, find the data with a check status of qualified, and perform data cleaning operation on these data to obtain cleaned data. In this method, the third-party tool Kettle is used to realize data cleaning. First, according to the type of the above data, set the required data format in the Kettle tool. When Kettle cleans, read the data content of each data, and convert the data that does not meet the preset format to the preset data format according to the type of different fields. For example, convert the date data in the format of September 8, 2022 to 2022 / 09 / 08.

[0098] S7: Confirm the positioning field;

[0099] The data content corresponding to the positioning field is the same, which means it is the same data content.

[0100] According to the submitted data content, the positioning field will have different settings, and adaptive settings will be made according to different data content.

[0101] As shown in the embodiment, the positioning field is set as: ID card, document number, and permit decision date. Figure 5

[0102] As shown in the embodiment, the positioning field is set as: unified social credit code, document number, and permit decision date. Figure 6

[0103] As shown in the embodiment, the positioning field is set as: unified social credit code / ID card, document number, and permit decision date. Figure 7 As shown in the embodiment, the positioning field is set as: unified social credit code / ID card, document number, and permit decision date.

[0104] S8: After cleaning, locate the corresponding business database table in the business database according to the corresponding requirement file, and query the business database table based on the preset positioning field to confirm whether it already exists;

[0105] If it already exists, it is determined that the data has been entered into the database and is recorded as: stored data;

[0106] Otherwise, the data is determined to be data that has not been entered into the database and recorded as: new data.

[0107] S9: Store the cleaned data into the corresponding business database table in the business database;

[0108] Overwrite and update the corresponding entries in the business database table with existing data;

[0109] Insert the new data into the business database table.

[0110] Regarding data updates in the business database, in actual work, the original data and the subsequent updated data may be several days apart, or they may be very close together. For example... Figure 5 In the illustrated embodiment, based on this method, data is periodically cleaned and stored in the business database. The original data and subsequent updated data are stored in the same storage task. That is, when storing the business database table, for the same data, insertion is performed first, followed by an update operation. According to this method, data updates can be performed accurately regardless of the time interval between the original data and the updated data.

[0111] Figure 5 In the data shown, the data with IDs zrr_001 and zrr_002, and the data with IDs zrr_031 and zrr_032, are data submitted by the same department with the same document number. The data with batch number 202005020616, compared with the previous batch number "202001020001", is a supplementary submission of the "Fisheries Permit".

[0112] In the storage to the business library, the "position field" in the embodiment is set to "unified social credit code / ID card", document number and license date / permit decision date", in the submission process, the data in the same original library table is stored according to the ID number sequence, and the data to be stored is judged according to the position field before storage, so it is known that the data with ID zrr_001 and zrr_002 is "new data", and the new data is inserted into the business library table; and the position field of the data with ID zrr_031 and zrr_032 is the same as that of the data with ID zrr_001 and zrr_002, so the data with ID zrr_031 and zrr_032 is judged as "stored data", and when storing, the data content of the data with ID zrr_031 and zrr_032 is overwritten on the data with ID zrr_001 and zrr_002, but the state, department name, department code and batch number are kept as the content of the first original data without updating, and the data updating operation is completed.

[0113] Steps S6-S9 complete the process of uploading data in the original library to the business library. In the method, the data uploading is divided into two stages based on the original library and the business library, wherein the action of uploading the original library by the sub-department is random and frequent, and the process of transferring the original library data to the business library can be controlled by the cleaning timer, and the data statistical time can also be controlled based on the statistical timer.

[0114] Therefore, in specific implementation, the time period with low original data uploading frequency can be counted according to the uploading time of historical data, the start time of the cleaning timer and the statistical timer is set in the time period, the probability of simultaneous reading and writing of the original library table by the uploading operation and the cleaning operation is reduced, and then the efficiency of mass data processing is improved and the probability of error of the original library table is reduced.

[0115] S10: According to the statistical timer, the statistical timer is started at the preset statistical time to count the data, specifically including the following steps.

[0116] a1: Obtain the name set of all original library tables in the original library, denoted as: original library table name set;

[0117] And put the original library table name set into the cursor.

[0118] a2: Loop the cursor to obtain each table name, and count all data of each table based on the original library table name;

[0119] According to the batch number, the reporting amount of each batch is obtained;

[0120] The data amount of all data in each original library table is calculated, that is, the reporting amount corresponding to the original library table.

[0121] Because each sub-department reports the original data, each submission can only be submitted to one original library table, and each submission will generate a unique batch number. The reporting quantity can be counted according to the batch number or the original library table.

[0122] a3: In each original library table, find all data with unqualified verification status;

[0123] According to the batch number, the corresponding quantity of doubts for each batch number is obtained;

[0124] According to the original library table, the corresponding quantity of doubts for the original library table is obtained.

[0125] Figure 5 In the embodiment, the number of values of 999 in the "state" column is counted, that is, the quantity of doubtful data that does not meet the verification rule is obtained.

[0126] a4: Obtain the name set of all business library tables in the business library, denoted as: business library table name set;

[0127] And put the business library table name set into the cursor;

[0128] a5: Loop the cursor to obtain each table name, and based on the business library table name, count all the data in each business library table;

[0129] According to the batch number, the corresponding quantity of doubts for each batch number is obtained;

[0130] According to the business library table name, the corresponding quantity of doubts for the business library table is obtained.

[0131] a6: Calculate the update quantity according to the batch number and the business library table respectively:

[0132] Update quantity = reporting quantity - doubtful quantity - storage quantity.

[0133] Suppose that for two data, the sub-department stores them into the business library after the first submission, assuming that the doubtful quantity is 0. Then it is updated twice, so for these two data, there are three batches in the original library, and 6 reporting quantities. The storage quantity stored into the business library is 2, and the final update quantity is:

[0134] 6-0-2=4.

[0135] In this method, the update data quantity is calculated based on other statistical data. Compared with the real-time statistical method based on log data in the prior art, this method reduces the database table operation time, and is faster, and also conforms to the data storage and update rule.

[0136] S11: display the statistical data in the statistics page.

[0137] When the correspondence between the original library table and the business library table is N:1, where N≥2;

[0138] The business library table includes a field: type, which corresponds to different original library table names with different type values; in the statistics calculator, when calculating the update amount, find all original library tables corresponding to the business library table according to the type field, and get the reporting amount and the doubtful amount of the original library table.

[0139] Because multiple original library tables are merged into one business library table, different original library tables are distinguished in the business library table through the "type" field. For example, Figure 7 The value of the type field in the business library table is 1, indicating that the data comes from the natural person original library table of Figure 5 The value of the type field is 0, indicating that the data comes from the legal person original library table of Figure 6 The "task batch number" in the business library table will not be repeated, so when calculating the update amount based on the "task batch number", only the original library table corresponding to the task batch number needs to be found, and then the update amount can be directly calculated according to the calculation formula.

[0140] Only when the overall update amount of the business library table is calculated, the reporting amount and the doubtful amount of the two original library tables need to be added, and then the total amount of the business library table is subtracted, to get the overall update amount of the business library table.

[0141] S12: loop steps S4-S11 to ensure uninterrupted data upload and statistics.

[0142] After using the technical solution of the present application, the method of recording various data amounts at the same time as data reporting is no longer used, but the data reporting and data amount recording steps are separated, the user first completes the reporting process according to the normal process, and then the data of each database table is summarized by batch after the user has finished reporting the data, and the statistical results are recorded in the summary statistics table. The page only needs to display the reporting amount, the amount of data, the doubtful amount, and the update amount according to the statistics table. This greatly reduces the probability of data reporting interruption and data amount statistical error.

Claims

1. A mass data storage and statistics method comprising the following steps: S1: constructing an original library and a business library; S2: creating a storage and statistics tool; creating a cleaning timer, based on which a data cleaning tool is periodically started to perform data cleaning operations on data in the original library; creating a statistics processor, which is implemented based on a storage process of an Oracle database and operates on data in the original library and the business library, including query, statistics and summary operations; creating a statistics timer, based on which the statistics processor is periodically started; the statistics processor and the statistics timer are stored in the business library, and the business library and the original library are connected through a DBlink to complete data transmission; S3: confirming the number of requirement files, and establishing corresponding original library tables and business library tables in the original library and the business library respectively for each requirement file; the original library table includes original library table data ID, department name, department code, verification state, batch number and data content; the batch number corresponding to the data content uploaded by the same sub-department each time is the same; each piece of data content corresponds to a verification state; the original library table data ID uniquely marks each piece of data content in the original library table; the verification state records the result of this data in rule verification, including compliance and non-compliance; the business library table includes business library table data ID, department name, department code, batch number and data content; the business library table data ID uniquely marks each piece of data content in the business library table; S4: receiving data uploaded by each batch of sub-departments to the original library in real time, and generating a batch number for this batch of uploaded data; S5: performing rule verification on the uploaded data based on a verification tool, generating the verification state corresponding to this piece of uploaded data, and saving the data to the corresponding original library table according to the requirement file corresponding to this batch of uploaded data; marking the verification state corresponding to the data content that meets the verification rule as compliant; marking the verification state corresponding to the data content that does not meet the verification rule as non-compliant; S6: based on the cleaning timer, starting the data cleaning tool at a preset data cleaning operation start time; finding data with a verification state of qualified in all new data in the original library table, and performing the data cleaning operation on these data to obtain cleaned data; S7: confirming a positioning field; the same data content corresponds to the same data content corresponding to the positioning field; S8: finding the corresponding business library table in the business library according to the requirement file corresponding to the cleaned data, and querying in the business library table based on a preset positioning field to confirm whether it already exists; if it already exists, it is determined to be data that has been stored, and is recorded as stored data; otherwise, it is determined to be data that has not been stored, and is recorded as new data; S9: storing the cleaned data in the corresponding business library table in the business library; updating the corresponding entry in the business library table with the stored data. inserting the new data into the business library table; S10: According to the statistical timer, starting the statistics of the data by the statistics collector at the preset statistical time, specifically including the following steps: a1: Obtaining a name set of all original library tables in the original library, denoted as: original library table name set; and putting the original library table name set into the cursor; a2: Looping the cursor to obtain each table name, and based on the original library table name, counting all data of each table; According to the batch number statistics, the reporting amount of each batch is obtained; Calculate the data amount of all data in each original library table, which is the reporting amount corresponding to the original library table; a3: In each of the original library tables, find all data with the check state of unqualified; According to the batch number statistics, the corresponding amount of doubt of each batch number is obtained; According to the original library table statistics, the corresponding amount of doubt of the original library table is obtained; a4: Obtaining a name set of all business library tables in the business library, denoted as: business library table name set; and putting the business library table name set into the cursor; a5: Looping the cursor to obtain each table name, and based on the business library table name, counting all data in each business library table; According to the batch number statistics, the corresponding amount of doubt of each batch number is obtained; According to the original library table statistics, the corresponding amount of doubt of the original library table is obtained; a6: Respectively according to the batch number and the business library table, the update amount is calculated: update amount = reporting amount - doubt amount - storage amount; S11: Display the statistical data in the statistics page; S12: Looping the steps S4~S11.

2. The mass data storage and statistics method according to claim 1, characterized in that: The business library and the original library are built on different servers.

3. The method of claim 1, wherein: When the corresponding relationship between the original library table and the business library table is N:1, N≥2; The business library table includes a field: type, which corresponds to different original library table names with different type values; When calculating the update amount in the statistics collector, according to the type field in the business library table, all original library tables corresponding to the business library table are determined, and the reporting amount and the amount of doubt in the original library table are obtained.

Images