Data Import Verification Method, System, Device and Computer Readable Medium

Abstract

The present invention discloses a data import verification method, system, device and computer-readable medium, belonging to the technical field of data storage. The technical problem to be solved is how to achieve the integrity verification of data import and improve the reliability of data and the availability of the system. S100: Configure basic constraints and association constraints; S200: Build a data verification model based on the basic constraints and management constraints, establish a physical index corresponding to the data verification model in the Es search engine, and synchronously generate a corresponding excel template file; S300: Save the excel data that meets the constraints to the Es search engine, save the excel data that does not meet the constraints as failed data and prompt the reason for the detection failure; S400: Insert the excle data saved in the Es search engine into the database, export the failed data in excel and correct it, and execute steps S300-S400 for the failed data after correction.

Description

Technical Field

[0001] The present invention relates to the technical field of data warehousing, and more particularly to a data import verification method, system, device and computer-readable medium. Background Art

[0002] In the field of government service products, there are many scenarios where historical data or system data that cannot be docked needs to be imported. However, each business system has its own business rules, which leads to certain constraint requirements for the imported data. Otherwise, the imported data will be unqualified when imported into the database.

[0003] In view of the above situation, and considering the strong data correlation, strict format, and large data volume in the business scenario, it is inevitable to use a suitable model for data verification during data import, and it is necessary to achieve adaptation for the case of a large amount of imported data.

[0004] The integrity of a database refers to the correctness and compatibility of data. Data integrity control is to prevent data that does not conform to semantics in the database, that is, to prevent incorrect data from appearing. Currently, when verifying data in a database, there are mainly three methods to solve the data integrity constraint mechanism: front-end application control, database triggers, and declarative constraints.

[0005] Application control refers to verifying the data format of the input data in the developer application, including the type and content of the data. When inserting, each field of the data is verified through the program to avoid the insertion of dirty data. Database triggers refer to implementing various constraints on data table operations by defining trigger conditions and writing execution statements after triggering, and can reference fields in other tables. Triggers can reference other tables and can contain complex SQL statements. When a table is modified, other tables are modified through the trigger according to relevant business rules. Once a situation that violates the business rules is found during the modification process, the data can be restored to the state before the modification through a rollback statement. Declarative constraints mainly include three aspects: entity integrity verification, that is, the data primary key is not null and unique; referential integrity, that is, the verification of the associated primary and foreign keys between tables; user-defined integrity, that is, declaring that the column data needs to meet semantic requirements (such as a value meeting a certain range or not being null).

[0006] Setting more constraints in the database will affect the performance of the database to a certain extent and is rarely applied in the real environment. Instead, it is more often processed in the program logic. It is possible that when facing business changes or system expansions, database constraints will make the processing less convenient.

[0007] Based on the above analysis, combined with the application scenario of the government service system, how to implement the integrity verification of data import and improve the reliability of data and the availability of the system are technical problems that need to be solved. Summary of the Invention

[0008] The technical task of the present invention is to address the above deficiencies by providing a data import verification method, system, device, and computer-readable medium to solve the technical problems of how to implement the integrity verification of data import and improve the reliability of data and the availability of the system.

[0009] In a first aspect, the data import verification method of the present invention includes the following steps:

[0010] S100. Configure basic constraints and association constraints. The basic constraints are used to constrain the fields in a single table, and the association constraints cooperate with the Es search engine to determine the index relationship of the associated tables in the Es search engine based on the table structure relationship between the associated tables, and import the table data into the Es search engine in sequence according to the index relationship;

[0011] S200. Build a data verification model based on the basic constraints and management constraints, establish a physical index corresponding to the data verification model in the Es search engine, and synchronously generate an excel template file corresponding to the data verification model. The data verification model can export an excel file and can save and import the excel file for use. The excel template file is used to fill in data and import it into the corresponding data verification model;

[0012] S300. Fill the data into the excel template file to obtain a to-be-verified excel file, and import the to-be-verified excel file into the corresponding data verification model for basic constraints and association constraints. Save the excel data that meets the constraints in the to-be-verified excel file to the Es search engine, and save the excel data that does not meet the constraints in the to-be-verified excel file as failed data and prompt the reason for the detection failure;

[0013] S400. Insert the excle data saved in the Es search engine into the database, export the failed data in excel and correct it, and execute steps S300 - S400 for the corrected failed data.

[0014] Preferably, the basic constraints include:

[0015] Basic constraints, which are used to constrain a single field in a single table, and are used to extract the basic constraints of the fields in the data table to be imported through table structure analysis, including non-null constraints, unique constraints, and primary key constraints;

[0016] Content constraints, which are used to set the content format of fields in a table, including time format, number format, and data range.

[0017] Preferably, based on the POI program, the data to be verified is imported into the data verification model through the following steps:

[0018] Match the corresponding data verification model for the excel file to be tested through the correspondence between the excel template file and the data verification model;

[0019] Extract the sheet pages of the excel file to be verified, so as to save each table of the excel file to be verified separately in a sheet page;

[0020] Extract the excel data in pages to ensure that the excel data can be imported into the corresponding data verification model quickly and efficiently.

[0021] Preferably, the format of the data verification model includes model name, model identifier, model creation time, model description, basic constraints, and association constraints;

[0022] The Excel template file is matched through the file name and the model identifier of the corresponding data verification model.

[0023] In a second aspect, the data import verification system of the present invention includes:

[0024] A constraint configuration module, which is used to configure basic constraints and association constraints. The basic constraints are used to constrain the fields in a single table, and the association constraints cooperate with the Es search engine to determine the index relationship of the associated tables in the Es search engine based on the table structure relationship between the associated tables, and import the table data into the Es search engine in sequence according to the index relationship;

[0025] A model construction module, which is used to construct a data verification model based on basic constraints and management constraints, establish a physical index corresponding to the data verification model in the Es search engine, and synchronously generate an excel template file corresponding to the data verification model. The data verification model can export an excel file and can save and import the excel file for use. The excel template file is used to fill in data and import it into the corresponding data verification model;

[0026] A data verification module, which is used to fill data into an excel template file to obtain a to-be-verified excel file, import the to-be-verified excel file into the corresponding data verification model for basic constraints and association constraints, save the excel data that meets the constraints in the to-be-verified excel file to the Es search engine, and save the excel data that does not meet the constraints in the to-be-verified excel file as failed data and prompt the reason for the detection failure;

[0027] A data storage module, which is used to insert the excle data saved in the Es search engine into the database, perform excel export and correction on the failed data, and import the corrected failed data into the data verification module.

[0028] Preferably, the basic constraints include:

[0029] Basic constraints, which are used to constrain a single field in a single table, and are used to extract the basic constraints of the fields in the data table to be imported through table structure analysis, including non-null constraints, unique constraints, and primary key constraints;

[0030] Content constraints, which are used to set the content format of the fields in the table, including time format, number format, and data range.

[0031] Preferably, the data verification module is used to import the to-be-verified excel file into the corresponding data verification model based on the POI program through the following steps:

[0032] Match the corresponding data verification model for the to-be-verified excel file through the correspondence between the excel template file and the data verification model;

[0033] Extract the sheet pages of the to-be-verified excel file to save each table of the to-be-verified excel file separately in a sheet page;

[0034] Extract the excel data in pages to ensure that the excel data can be imported into the corresponding data verification model quickly and efficiently.

[0035] Preferably, the format of the data verification model includes model name, model identifier, model creation time, model description, basic constraints, and association constraints;

[0036] The Excel template file is matched through the file name and the model identifier of the corresponding data verification model.

[0037] In a third aspect, the device of the present invention includes: at least one memory and at least one processor;

[0038] The at least one memory is configured to store machine-readable programs;

[0039] The at least one processor is configured to call the machine-readable programs to execute the method according to any one of the first aspect.

[0040] In a fourth aspect, a computer-readable medium of the present invention stores computer instructions thereon, and when the computer instructions are executed by a processor, the processor is caused to execute the method according to any one of the first aspect.

[0041] The data import verification method, system, device and computer-readable medium of the present invention have the following advantages:

[0042] 1. It has a fast response speed in a high-concurrency environment, can quickly perform data integrity constraint detection on data, decouples the front-end operation and the processing of part of the business logic through the cooperation of the data verification model and the Es search engine, simplifies the business processing process of requests, and fundamentally improves the response speed of requests;

[0043] 2. The model relies on the elasticsearch search engine to quickly query and verify data, improves the reliability and stability of the model, considers the accuracy of data and the stability of the database in data landing, and ensures the high reliability and high availability of system use and service provision;

[0044] 3. Data is imported through the integrated POI program, adapting to file formats of multiple versions such as wps and office;

[0045] 4. Data is imported and exported through excel files, improving operability and realizing data visualization processing. Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0047] The present invention will be further described below with reference to the drawings.

[0048] Figure 1 It is a flowchart of the data import verification method for Embodiment 1;

[0049] Figure 2 It is an architecture block diagram of the Es search engine in the data import verification method for Embodiment 1

[0050] Figure 3Schematic diagram of the format of the Excel template file in the data import verification method of Embodiment 1;

[0051] Figure 4 Flow chart of the constraint detection process in the data import verification method of Embodiment 1;

[0052] Figure 5 Schematic diagram of the format of the failed data in the data import verification method of Embodiment 1. Detailed implementation manners

[0053] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it. However, the specific embodiments cited are not intended to limit the present invention. Without conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0054] The embodiments of the present invention provide a data import verification method, system, device and computer-readable medium, which are used to solve the technical problem of how to implement the integrity verification of data import and improve the reliability of data and the availability of the system.

[0055] Embodiment 1:

[0056] The data import verification method of the present invention includes the following steps:

[0057] S100. Configure basic constraints and association constraints. The basic constraints are used to constrain the fields in a single table. The association constraints cooperate with the Es search engine to determine the index relationship of the associated tables in the Es search engine based on the table structure relationship between the associated tables, and import the table data into the Es search engine in sequence according to the index relationship;

[0058] S200. Build a data verification model based on the basic constraints and management constraints, establish a physical index corresponding to the data verification model in the Es search engine, and synchronously generate an excel template file corresponding to the data verification model. The data verification model can export an excel file and can save and import the excel file for use. The excel template file is used to fill in data and import it into the corresponding data verification model;

[0059] S300. Fill the data into the excel template file to obtain a to-be-verified excel file, and import the to-be-verified excel file into the corresponding data verification model for basic constraints and association constraints. Save the excel data that meets the constraints in the to-be-verified excel file to the Es search engine, and save the excel data that does not meet the constraints in the to-be-verified excel file as failed data and prompt the reason for the detection failure;

[0060] S400. Insert the Excel data saved in the Es search engine into the database, export the failed data to Excel and correct it, and execute steps S300 - S400 for the failed data after correction.

[0061] Es (full English name: Elasticsearch) search engine (hereinafter referred to as Es) is a search server based on Lucene. It provides a distributed multi - user full - text search engine based on the RESTful web interface. Elasticsearch is developed in the Java language and released as open source under the Apache license terms. It is a popular enterprise - level search engine. Elasticsearch is used in cloud computing and can achieve real - time search, being stable, reliable, fast, and easy to install and use. The specific structure Figure 2 is shown as follows.

[0062] Gateway represents the persistent storage method of ES indexes. In Gateway, ES first stores the indexes in memory by default, and then when the memory is full, it persists to Gateway. When the ES cluster is shut down or restarted, it reads the index data from Gateway. Such as LocalFileSystem and HDFS, AS3, etc.

[0063] DistributedLucene Directory is a directory composed of a series of Lucene index files. It is responsible for managing these index files, including data reading, writing, and index addition and merging, etc. River represents the data source and exists in ES in the form of a plugin.

[0064] Mapping means mapping, which is very similar to data types in static languages. For example, when we declare an int - type variable, then this variable can only store int - type data in the future. For example, if we declare a mapping field of double type, then it can only store double - type data.

[0065] Search Moudle is the search module that supports some common search operations. Index Moudle is the index module that supports some common index operations. Disvcovery is mainly responsible for the discovery of the master node in the cluster. For example, when a certain node suddenly leaves or joins, it conducts a re - sharding of shards. There is a discovery mechanism here. RESTful StyleAPI realizes API programming in a RESTful way. 3rd plugins represent third - party plugins. Java (Netty) is the development framework and JMX is for monitoring.

[0066] For POI document import, Apache POI is an open-source project that processes various file formats based on the Office Open XML standard (OOXML) and Microsoft's OLE 2 Compound Document Format (OLE2). In short, you can read and write MS Excel files using Java, and you can also read and write MS Word and MS PowerPoint files using Java. It is divided into the following modules.

[0067] HSSF - Provides the function of reading and writing Microsoft Excel XLS format (Microsoft Excel 97(-2003)) files.

[0068] XSSF - Provides the function of reading and writing Microsoft Excel OOXML XLSX format (Microsoft Excel XML) files.

[0069] SXSSF - Provides the function of reading and writing Microsoft Excel OOXML XLSX format files with low memory consumption.

[0070] HWPF - Provides the function of reading and writing Microsoft Word DOC97 format (Microsoft Word 97(-2003)) files.

[0071] XWPF - Provides the function of reading and writing Microsoft Word DOC2003 format (WordprocessingML(2007+)) files.

[0072] HSLF / XSLF - Provides the function of reading and writing Microsoft PowerPoint format files.

[0073] HDGF / XDGF - Provides the function of reading Microsoft Visio format files.

[0074] HPBF - Provides the function of reading Microsoft Publisher format files.

[0075] HSMF - Provides the function of reading Microsoft Outlook format files.

[0076] In this embodiment, to establish a data verification model, it is necessary to configure and establish basic constraints and association constraints. The basic data includes basic constraints and content constraints.

[0077] The basic constraints mainly target the constraints of a single field in a single table. Through table structure analysis, the basic constraints of the fields in the data table to be imported are extracted, including three categories: NOT NULL constraint, UNIQUE constraint, and PRIMARY KEY constraint. These data can be obtained by connecting to the database without manual control.

[0078] The content constraints mainly set the content format (CHECK) of the fields in the table, including time format, number format, data range, etc.

[0079] The above two types of constraints are simple single-table constraints, and their storage format in the model is:

[0080]

[0081] The association constraints need to be combined with the Es search engine. First, organize the table structure relationships of multiple associated tables, determine the indexes in Es, and import the table data into Es in order according to the index relationships. This can achieve fast associated query and insertion of data, ensuring that data can be quickly detected and stored in the database in the case of a large amount of data, and avoiding the pressure on the database caused by directly connecting to the database for multi-table associated queries. The model format of the association constraints is:

[0082] Table Name Field Name Related Field Table2 Biz_id Table1.id Table3 Biz_id Table1.id Table3 Course_id Table2.id

[0083] After determining the basic constraints, content constraints, and association constraints, the model is basically determined. At the same time, a corresponding physical index is established in Es to ensure that the model can be exported as a file for storage and import use. A corresponding excel template will be generated synchronously, and data import can be achieved by filling in this template.

[0084] The basic format of the model is:

[0085]

[0086]

[0087] In step S300, the poi program is used to import the filled excel data. The import process is divided into three steps. First, the model corresponding to the excel is determined. Currently, this is done by matching the file name with the model id. Ensure that the imported excel format meets the requirements of the template and the model is valid. Second, the sheet pages of the excel are extracted. Since in the case of multiple sheets, the data needs to be saved separately in each sheet page. Finally, the data is extracted in pages to ensure that the data can be imported into the model quickly and efficiently. Poi can support the one-time processing of more than 100,000 pieces of data, avoiding the pressure on the model caused by too large data packets. The format of the Excel template is as Figure 3 shown.

[0088] The data verification model first detects the basic constraints and content constraints of the imported data. After passing the detection, it performs an associated comparison query through es. The inspection is carried out successively through the association relationship. If it meets the requirements, it is directly saved into es. The data that does not meet the requirements is saved and the reason for the detection failure is prompted. After all the data has been detected, the data that is completely in line with the requirements in es is inserted into the database, and the data that does not meet the requirements is exported to excel for the operator to rectify. The specific detection flow chart is as Figure 4 , and the exported excel format is as Figure 5 shown.

[0089] The method of this embodiment can be applied to government services to perform intelligent detection on the data during the data import process.

[0090] Embodiment 2:

[0091] The data import verification system of the present invention includes a constraint configuration module, a model construction module, a data verification module, and a data storage module. The constraint configuration module is used to configure basic constraints and association constraints. The basic constraints are used to constrain fields in a single table. The association constraints cooperate with the Es search engine to determine the index relationship of the associated tables in the Es search engine based on the table structure relationship between the associated tables, and sequentially import the table data into the Es search engine according to the index relationship. The model construction module is used to construct a data verification model based on the basic constraints and management constraints, establish a physical index corresponding to the data verification model in the Es search engine, and synchronously generate an excel template file corresponding to the data verification model. The data verification model can export an excel file and can save and import the excel file for use. The excel template file is used to fill in data and import it into the corresponding data verification model. The data verification module is used to fill in the data into the excel template file to obtain a to-be-verified excel file, and import the to-be-verified excel file into the corresponding data verification model for basic constraints and association constraints. The excel data that meets the constraints in the to-be-verified excel file is saved to the Es search engine, and the excel data that does not meet the constraints in the to-be-verified excel file is saved as failed data and the reason for the detection failure is prompted. The data storage module is used to insert the excle data saved in the Es search engine into the database, export the failed data in excel and correct it, and import the corrected failed data into the data verification module.

[0092] In this embodiment, the basic constraints include basic constraints and content constraints. The basic constraints are used to constrain a single field in a single table, and are used to extract the basic constraints of the fields in the data table to be imported through table structure analysis, including non-null constraints, unique constraints, and primary key constraints. The content constraints are used to set the content format of the fields in the table, including time format, number format, and data range.

[0093] The data verification module is used to import the to-be-verified excel file into the corresponding data verification model based on the POI program through the following steps: First, determine the model corresponding to the excel. Currently, this is matched by the file name corresponding to the model id. Ensure that the imported excel format meets the requirements of the template and the model is valid. Second, extract the sheet pages of the excel. Because in the case of multiple tables, the data needs to be saved separately in each sheet page for each table. Finally, extract the data in pages to ensure that the data can be imported into the model quickly and efficiently. Poi can support the one-time processing of more than 100,000 pieces of data, avoiding the pressure on the model caused by too large data packets.

[0094] The data verification module first detects the imported data through the data verification model for basic constraints and content constraints. After passing the detection, it conducts associated comparison queries through ES, and conducts inspections in sequence according to the association relationship. If it meets the requirements, it is directly saved into ES. The data that does not meet the requirements is saved and the reasons for the detection failure are prompted. After all the data has been detected, the data that is completely compliant in ES is inserted into the database, and the non-compliant data is exported to Excel for the operator to rectify.

[0095] The data import verification system of this embodiment can execute the method disclosed in the embodiment, and verify the data imported into the database based on the data constraint relationship.

[0096] Embodiment 3:

[0097] The device of the present invention includes: at least one memory and at least one processor; at least one memory for storing machine-readable programs; at least one processor for calling the machine-readable programs and executing the method disclosed in Embodiment 1.

[0098] Embodiment 4

[0099] The computer-readable medium of the present invention has computer instructions stored thereon. When the computer instructions are executed by a processor, the processor executes the method disclosed in Embodiment 1. Specifically, a system or device equipped with a storage medium can be provided. On this storage medium, software program codes for implementing the functions of any one of the above embodiments are stored, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0100] In this case, the program code read from the storage medium itself can implement the functions of any one of the above embodiments. Therefore, the program code and the storage medium storing the program code constitute a part of the present invention.

[0101] Embodiments of the storage medium for providing program codes include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program codes can be downloaded from a server computer through a communication network.

[0102] In addition, it should be clear that not only can the actual operations be completed in part or in whole by executing the program codes read by the computer, but also by means of instructions based on the program codes through an operating system operating on the computer, etc., so as to implement the functions of any one of the above embodiments.

[0103] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer. Subsequently, based on the instructions of the program code, the CPU or the like installed on the expansion board or the expansion unit is made to execute some or all of the actual operations, thereby implementing the functions of any one of the above embodiments.

[0104] It should be noted that not all steps and modules in the above-mentioned various processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted according to needs. The system structure described in the above-mentioned various embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities separately, or some components in multiple independent devices can be jointly implemented.

[0105] In the above embodiments, the hardware units can be implemented mechanically or electrically. For example, a hardware unit can include a permanently dedicated circuit or logic (such as a dedicated processor, FPGA or ASIC) to complete the corresponding operations. The hardware unit can also include programmable logic or circuits (such as a general-purpose processor or other programmable processors), which can be temporarily set by software to complete the corresponding operations. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0106] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above-mentioned multiple embodiments, those skilled in the art can know that more embodiments of the present invention can be obtained by combining the code review means in the above different embodiments, and these embodiments are also within the protection scope of the present invention.

Claims

1. A data import verification method, characterized in that It includes the following steps: S100. Configure basic constraints and association constraints. The basic constraints are used to constrain fields in a single table. The association constraints cooperate with the Es search engine to determine the index relationship of the associated tables in the Es search engine based on the table structure relationship between the associated tables, and import table data into the Es search engine in sequence according to the index relationship; S200. Build a data verification model based on the basic constraints and association constraints, establish a physical index corresponding to the data verification model in the Es search engine, and synchronously generate an excel template file corresponding to the data verification model. The data verification model can export an excel file and can save and import the excel file for use. The excel template file is used to fill in data and import it into the corresponding data verification model; S300. Fill the data into the excel template file to obtain the to-be-verified excel file, import the to-be-verified excel file into the corresponding data verification model for basic constraint and association constraint verification, save the excel data that meets the constraints in the to-be-verified excel file to the Es search engine, and save the excel data that does not meet the constraints in the to-be-verified excel file as failed data and prompt the reason for the detection failure; S400. Insert the excle data saved in the Es search engine into the database, export the failed data in excel and correct it, and execute steps S300 - S400 for the corrected failed data.

2. The data import verification method according to claim 1, characterized in that The basic constraints include: Basic constraints, which are used to constrain a single field in a single table, and are used to extract the basic constraints of the fields in the data table to be imported through table structure analysis, including non-null constraints, unique constraints, and primary key constraints; Content constraints, which are used to set the content format of the fields in the table, including time format, number format, and data range.

3. The data import verification method according to claim 1, wherein Based on the POI program, import the to-be-verified data into the data verification model through the following steps: Match the corresponding data verification model for the to-be-verified excel file through the correspondence between the excel template file and the data verification model; Extract the sheet pages of the to-be-verified excel file to save each table of the to-be-verified excel file separately in a sheet page; Extract the excel data in pages to ensure that the excel data can be imported into the corresponding data verification model quickly and efficiently.

4. The data import verification method according to any one of claims 1 to 3, characterized in that The format of the data verification model includes model name, model identifier, model creation time, model description, basic constraints, and association constraints; The excel template file is matched by the file name and the model identifier of the corresponding data verification model.

5. Data import verification system, including: Constraint configuration module, which is used to configure basic constraints and association constraints. The basic constraints are used to constrain fields in a single table, and the association constraints cooperate with the Es search engine to determine the index relationship of the associated tables in the Es search engine based on the table structure relationship between the associated tables, and import the table data into the Es search engine in sequence according to the index relationship; Model construction module, which is used to construct a data verification model based on basic constraints and association constraints, establish a physical index corresponding to the data verification model in the Es search engine, and synchronously generate an excel template file corresponding to the data verification model. The data verification model can export an excel file and can save and import the excel file for use. The excel template file is used to fill in data and import it into the corresponding data verification model; Data verification module, which is used to fill data into the excel template file to obtain a to-be-verified excel file, and import the to-be-verified excel file into the corresponding data verification model for basic constraint and association constraint verification. Save the excel data that meets the constraints in the to-be-verified excel file to the Es search engine, and save the excel data that does not meet the constraints in the to-be-verified excel file as failed data and prompt the reason for the detection failure; Data storage module, which is used to insert the excle data saved in the Es search engine into the database, export the failed data in excel and correct it, and import the corrected failed data into the data verification module.

6. The data import verification system according to claim 5, wherein The basic constraints include: Basic constraints, which are used to constrain a single field in a single table, and are used to extract the basic constraints of the fields in the data table to be imported through table structure analysis, including non-null constraint, unique constraint, and primary key constraint; Content constraints, which are used to set the content format of the fields in the table, including time format, number format, and data range.

7. The data import verification system according to claim 5, wherein The data verification module is used to import the to-be-verified excel file into the corresponding data verification model based on the POI program through the following steps: Match the corresponding data verification model for the to-be-verified excel file through the correspondence between the excel template file and the data verification model; Extract the sheet pages of the to-be-verified excel file to save each table of the to-be-verified excel file separately in a sheet page; Extract the excel data in pages to ensure that the excel data can be imported into the corresponding data verification model quickly and efficiently.

8. The data import verification system according to any one of claims 5-7, characterized in that The format of the data verification model includes model name, model identifier, model creation time, model description, basic constraints, and association constraints; The Excel template file is matched by the file name and the model identifier of the corresponding data verification model.

9. Apparatus, characterized in that, Including: At least one memory and at least one processor; The at least one memory is used to store machine-readable programs; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 4.

10. A computer-readable medium, characterized in that, Computer instructions are stored on the computer-readable medium, and when executed by a processor, cause the processor to execute the method according to any one of claims 1 to 4.

Images