Table file multi-thread import method and system and table file multi-thread export method and system
By using multi-threading technology and Java reflection principle in the banking system, the multi-threading technology and Java reflection principle are used to import and export table files from multiple threads, and the problems of inefficiency and high error rate are solved due to manual processing, and efficient and accurate report data processing is achieved.
Patent Information
- Application Number
- CN202311054076.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-21
- Publication Date
- 2025-02-21
AI Technical Summary
In the prior art, the import and export of banking system statement data relies on manual processing, resulting in inefficiency, high error rate and risk of review fatigue and negligence.
Using multi-threading technology and Java reflection principle, we create a thread pool for multi-threaded import and export of table files, including file verification, data insertion and export, and improve efficiency and reduce error rates through multi-threading.
By processing the import and export of table files through multi-threading, data processing efficiency is significantly improved, error rate is reduced, labor costs are reduced, and report operations are guaranteed.
Smart Images
Figure CN119493659A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a table file multi-thread import method and system, and an export method and system, belonging to the technical field of data sharing. Background Art
[0002] With the liberalization of financial markets, banks have developed rapidly, their business has continued to increase, and their systems have become more and more complex. As a result, isolated islands have been created between the various systems. In addition, bank system reports are diverse in type, with large amounts of data and strong real-time requirements. Currently, a large number of report processing between systems can only rely on manual processing by business personnel. Manual identification and entry is not only inefficient, time-consuming and labor-intensive, and has a slow processing speed, but sometimes errors may occur due to manual review fatigue or negligence. Summary of the invention
[0003] The purpose of the present invention is to provide a multi-threaded import method and system, and an export method and system for table files, so as to solve the problems of low data processing efficiency and high error rate that are easily generated when importing and exporting report data manually.
[0004] To achieve the above objectives, the technical solutions and corresponding solutions provided by the present invention have the following beneficial effects:
[0005] A multi-threaded import method for a table file of the present invention comprises the following steps:
[0006] 1) Create a thread pool according to the thread pool parameters defined in the configuration file, and use the created thread pool to read the file data to be imported in a multi-threaded manner based on the Java reflection principle, and verify it while reading;
[0007] 2) Based on the Java reflection principle and using the created thread pool, a multi-threaded method is used to insert the verified file data into the data table to be imported and output the import results.
[0008] The beneficial effect of the above scheme is: different from the traditional simple table file data import method, when using multi-threading to import file data, it is first verified while reading the file based on the Java reflection principle, and then based on the Java reflection principle and using multi-threading, the read file data is written into the corresponding data table, so as to improve the data import efficiency and reduce the data import error rate.
[0009] As a further improvement of the method: the verification includes file format verification and title verification, and when both the file format verification and the title verification are passed, it indicates that the verification is passed; the file format verification is used to filter out table files; the title verification is to compare the first row of title data obtained based on the Java reflection principle with the annotation obtained based on the Java annotation principle, and when the comparison result is consistent, it is determined that the title verification is passed.
[0010] The beneficial effect of the above scheme is: when importing files for verification, the file format is first verified to filter out table files, and then the title is verified to filter out table files with imported titles, thereby improving import accuracy and efficiency.
[0011] As a further improvement of the method: when inserting the verified file data into the data table to be imported, it is necessary to determine whether there is the same data in the current verified file data and the current data table: if there is the same data, no subsequent operations will be performed on the current verified file data; if there is no identical data, the field length and null value check will be performed on the current verified file data, and after the check is passed, it will be inserted into the data table to be imported.
[0012] The beneficial effects of the above scheme are: the data table may contain data that is exactly the same as the data read from the current file, so it is necessary to perform the same data judgment, and no further operation is performed for the data already in the data table, so as to reduce the insertion of duplicate data and improve system efficiency. At the same time, the field length and null value check are performed for the data that is judged to be non-duplicate, so as to ensure the accuracy of the data written to the table.
[0013] As a further improvement of the method: after inserting into the data table to be imported, deduplication processing is required.
[0014] The beneficial effect of the above scheme is: after inserting the read file data into the data table to be imported, the data table needs to be deduplicated to ensure the uniqueness of the table writing data and prevent duplicate key value errors from occurring when recording the table, causing process interruption.
[0015] As a further improvement of the method: when using a multi-threaded method to read the file data to be imported and insert the verified file data into the data table to be imported, if there is a waiting queue in the thread pool, continue to use multi-threading to read the file data to be imported and insert the verified file data into the data table to be imported until there is no waiting queue in the thread pool and then close the thread pool.
[0016] The beneficial effect of the above scheme is: using a multi-thread method based on a fixed thread pool to insert table file data, thereby improving the efficiency of data writing to the table.
[0017] A multi-threaded export method for a table file of the present invention comprises the following steps:
[0018] a. Create a thread pool according to the thread pool parameters defined in the configuration file, and obtain the exported data by performing multi-threaded queries on the data table;
[0019] b. Divide the exported data into pages and read each page of data through multiple threads;
[0020] c. Based on the Java reflection principle, the data read by multiple threads is written into the newly generated blank table file and the corresponding data report is finally generated.
[0021] The beneficial effects of the above scheme are: different from the traditional simple table file data import method, when using multi-threading to export file data, first call the logic layer query method based on Java reflection according to the query conditions to perform multi-threaded paging query on the data table and summarize all the result data, then read the query data based on Java reflection multi-threading and write it into a blank table file, output the file stream to the foreground for users to download, thereby improving data export efficiency and reducing data export error rate.
[0022] As a further improvement of the method: when writing the data read by multiple threads into the newly generated blank table file based on the Java reflection principle, if the writing fails, an error message indicating that the data writing to the table failed is obtained and the thread pool is closed, otherwise a judgment is made as to whether the reading of the data in the page has been completed.
[0023] The beneficial effect of the above scheme is that by judging the success or failure of multi-threaded data reading, the export results and their error causes can be displayed more intuitively, so that relevant personnel can analyze the import results in a timely manner.
[0024] As a further improvement of the method: whether the in-page data reading is completed is judged as follows: if the in-page data reading is completed and there is no unexecuted thread in the waiting queue, the thread pool is closed and the multi-threaded data export process is ended; otherwise, the in-page data reading or multi-threaded reading of each page of data continues.
[0025] The beneficial effect of the above scheme is: by judging the thread pool, it is determined whether there are any unexecuted threads in the thread pool, thereby avoiding data reading residue.
[0026] A table file multi-threaded import system of the present invention includes a memory and a processor, wherein the processor is used to execute computer program instructions stored in the memory to implement the table file multi-threaded import method described above.
[0027] A multi-threaded export system for table files of the present invention is characterized in that it includes a memory and a processor, wherein the processor is used to execute computer program instructions stored in the memory to implement the multi-threaded export method for table files described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is the overall implementation principle diagram of the present invention;
[0029] Figure 2 It is a file verification flow chart of the present invention;
[0030] Figure 3It is a file import flow chart of the present invention;
[0031] Figure 4 It is a file export flow chart of the present invention. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0033] The purpose of the present invention is to provide a table file multi-threaded import method and system, export method and system. Using multi-threading to import and export table file data not only reduces the error rate and labor cost of report processing, but also ensures the real-time nature of report operation, solving the actual needs of bank report development.
[0034] Table file multi-threaded import method embodiment:
[0035] like Figure 1 As shown, a multi-threaded import process of a table file includes report file verification and multi-threaded import of report data (ie, multi-threaded reading of reports, multi-threaded table recording, and statistics of success and failure data).
[0036] Specifically, report file verification includes: table file verification, report title verification, and field verification. Among them, table file verification verifies the file format and file size based on the file name and the first 4 bytes of the file; report title verification obtains the report title through the Entity annotation based on the custom annotation interface and Java reflection method and compares it with the first line of the file title read; field verification verifies the field value and its corresponding field length, whether it is empty, and other field verification conditions based on annotation and reflection principles.
[0037] Multi-threaded report reading: used for multi-threaded reading of report files to obtain file data. Specifically, a thread pool is created based on the parameter "number of file reading threads", and the currently read cell data is assigned to the corresponding attribute of the entity based on the Java reflection method for multi-threaded reading. At the same time, when reading data, the data format is checked according to the field length and whether it is empty and other check rules defined in the annotation, and after reading all the data, the data with the same key value is checked in the read data for duplicate key value check.
[0038] Multi-threaded table recording: used for multi-threaded file data insertion. Specifically, a thread pool is created based on the parameter "number of file writing threads". For each thread, the query method in the logic layer method is called based on Java reflection to check whether there is data with the same key value. If not, the table is recorded. If yes, it is checked whether all fields are the same. If they are different, the data is updated.
[0039] Result statistics: used to count the number of successful and failed rows and other information, and generate import failure files for download. Specifically, statistics on success and failure data (such as: number of successful / failed rows, number of duplicate rows, number of updated successful / failed rows, number of duplicated marked rows, etc.), if there is failed data (failed verification, failed table insertion, failed update, etc.), a failed data file is generated and marked with different colors according to the failure reason, and the import results and failed file names are placed in the view for users to view and download.
[0040] Combination Figure 2 , Figure 3 It can be seen that the steps for importing table files are:
[0041] 1) Create a thread pool according to the thread pool parameters (thread pool size) defined in the configuration file, and use multi-threading based on the Java reflection principle to perform multi-threaded paging and concurrent reading of the required imported file data (the number of rows read is based on the parameter "page size"), and verify it while reading.
[0042] 2) Based on the Java reflection principle and using the created thread pool, a multi-threaded method is used to insert the verified file data into the data table to be imported and output the import results.
[0043] Specifically, Figure 2 For the table file verification process, first check whether the file format is a table file, and then check whether the first row title is consistent with the field title obtained by annotation. If one of these two checks fails, the verification result fails, and an error message is directly output, and subsequent table recording operations will not be performed.
[0044] If the first two checks are passed, each row of data is read by multiple threads to check whether there is the same data in the data table. If the same value exists, the duplicate number is increased by 1 and the data is added to the success list. If the same value does not exist, the name and type of each field in the read file data are obtained based on the Java reflection principle, and the field length and null value are checked based on Java annotations. The file data that passes the check is added to the success list, and the file data that fails the check is added to the failure list, and the failure number is increased by 1. Then, the duplicate rows in the success list are judged and deduplicated to complete the check.
[0045] Figure 3 The figure shows the process of importing a table file. The program creates a thread pool, pages the verified data, and creates a thread pool of a fixed size N. In each thread, the current page data is read and recorded. If the recording fails, it is added to the failure list. If the current N concurrent threads are executed, the waiting queue thread is awakened. If there is no waiting queue, the thread is terminated and the thread pool is closed. Finally, statistical data such as the total number of successes, the total number of failures, and the total number of repetitions are output. If there is failed data, a failure data file is generated.
[0046] Table file multi-threaded export method embodiment:
[0047] like Figure 1 As shown, the table file export includes multi-threaded paging query and multi-threaded export of report files (i.e., multi-threaded file writing).
[0048] Specifically, multi-threaded paging query: used for multi-threaded paging query to obtain the data required for the export file. That is, according to the page query conditions, the logic layer query method is called based on Java reflection to perform multi-threaded paging query on the data table and summarize all the result data.
[0049] Multi-threaded file writing: used for multi-threaded file writing and report export. That is, first generate a blank ET table file, then read the query data based on Java reflection multi-threading and write it into the table file, and then output the file stream to the front end for users to download.
[0050] like Figure 4 As shown, the specific steps for multi-threaded export of table files are as follows:
[0051] Step 1: Paging query. The program obtains the query method through reflection and performs multi-threaded paging query to obtain the data required for exporting the file.
[0052] Step 2: Write the query results to the file in multiple threads and finally generate a report.
[0053] Specifically, firstly, multi-threaded query is performed according to the query conditions to obtain the required imported data, and then multi-threaded writing is performed to the file. If an error occurs when writing the file, the thread pool is directly closed and the error message is output to the front end. If no error occurs, the execution continues until all the data is written to the file, and finally the file stream is output.
[0054] Example of a multi-threaded import system for table files:
[0055] A table file multi-threaded import system includes a memory and a processor, and data is exchanged between the memory and the processor via an internal bus. The processor is used to execute computer program instructions stored in the memory to implement a table file multi-threaded import system of the present invention. The specific content of the method has been described in detail in the embodiment of the table file multi-threaded import method, and will not be repeated in this embodiment. Among them, the memory can be selected from semiconductor memory, magnetic surface memory, read-only memory, sequential memory, etc., and the processor can be selected from Intel processor, MD processor, IBM processor, MediaTek processor, etc.
[0056] Example of a multi-threaded export system for table files:
[0057] A table file multi-threaded export system includes a memory and a processor, and data is exchanged between the memory and the processor via an internal bus. The processor is used to execute computer program instructions stored in the memory to implement a table file multi-threaded export system of the present invention. The specific content of the method has been described in detail in the table file multi-threaded export method embodiment, and will not be repeated in this embodiment. Among them, the memory can be selected from semiconductor memory, magnetic surface memory, read-only memory, sequential memory, etc., and the processor can be selected from Intel processor, MD processor, IBM processor, MediaTek processor, etc.
[0058] The present invention uses multithreading to read and write table files that have passed verification based on annotation and reflection principles to realize table file import, and uses multithreaded paging query to obtain the required export data and write the data into the report to realize table file export. This method is simple and fast to import and export table data, reduces the error rate and labor cost of report processing, ensures the real-time nature of report operation, and solves the actual needs of bank report development.
[0059] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. After reading this application, technicians in the relevant field may make various modifications or changes to the present invention with reference to the above embodiments, which are all within the scope of protection claimed in the patent application of the present invention.
Claims
1. A multi-threaded import method for a table file, characterized in that: The steps include: 1) Create a thread pool according to the thread pool parameters defined in the configuration file, and use the created thread pool to read the file data to be imported in a multi-threaded manner based on the Java reflection principle, and verify it while reading; 2) Based on the Java reflection principle and using the created thread pool, a multi-threaded method is used to insert the verified file data into the data table to be imported and output the import results.
2. The multi-threaded import method of table files according to claim 1, characterized in that: The verification includes a file format verification and a title verification. When both the file format verification and the title verification are passed, the verification is passed. The file format verification is used to filter out table files. The title check is to compare the first row of title data obtained based on the Java reflection principle with the annotation obtained based on the Java annotation principle, and when the comparison result is consistent, it is determined that the title check has passed.
3. The multi-threaded import method of table files according to claim 1, characterized in that: When inserting the verified file data into the data table to be imported, it is necessary to determine whether there is the same data in the current verified file data and the current data table: if there is the same data, no subsequent operations will be performed on the current verified file data; if there is no identical data, the field length and null value check will be performed on the current verified file data, and after the check passes, it will be inserted into the data table to be imported.
4. The multi-threaded import method of table files according to claim 3, characterized in that: After inserting into the data table to be imported, deduplication processing is required.
5. The multi-threaded import method of table files according to claim 1, characterized in that: When using multi-threading to read the file data to be imported and inserting the verified file data into the data table to be imported, if there is a waiting queue in the thread pool, continue to use multi-threading to read the file data to be imported and insert the verified file data into the data table to be imported until there is no waiting queue in the thread pool and then close the thread pool.
6. A multi-threaded export method for a table file, characterized in that: The steps include: a. Create a thread pool according to the thread pool parameters defined in the configuration file, and obtain the exported data by performing multi-threaded queries on the data table; b. Divide the exported data into pages and read each page of data through multiple threads; c. Based on the Java reflection principle, the data read by multiple threads is written into the newly generated blank table file and the corresponding data report is finally generated.
7. The multi-threaded export method for table files according to claim 6, characterized in that: When writing data read by multiple threads into a newly generated blank table file based on the Java reflection principle, if the writing fails, an error message indicating that the data has failed to be written into the table is obtained and the thread pool is closed. Otherwise, a judgment is made as to whether the reading of the data in the page has been completed.
8. The multi-threaded export method for table files according to claim 7, characterized in that: Whether the in-page data reading is completed is judged as follows: if the in-page data reading is completed and there is no unexecuted thread in the waiting queue, the thread pool is closed and the multi-threaded data export process is ended; otherwise, the in-page data reading or multi-threaded reading of each page of data continues.
9. A multi-threaded import system for table files, characterized in that: It comprises a memory and a processor, wherein the processor is used to execute computer program instructions stored in the memory to implement the multi-threaded import method of a table file as described in any one of claims 1 to 5.
10. A multi-threaded export system for table files, characterized in that: It comprises a memory and a processor, wherein the processor is used to execute computer program instructions stored in the memory to implement the multi-threaded export method for table files as described in any one of claims 6 to 8.