Method, device and processor for importing or exporting table-type file data into or from a database

Abstract

The embodiment of the present application provides a method, device and processor for importing or exporting data from a table-type file into or from a database. The method includes: when importing into a database, performing the following operations: receiving a table-type file; performing format verification on the table-type file, and extracting first structural information of the table-type file, wherein the first structural information includes a first data type; and based on the first structural information, performing block reading and data stream processing on the table-type file to obtain each corresponding row of data or column of data; and for each row of data or column of data, mapping each data in the row of data or column of data from a first data type to a second data type through a pre-set data mapping rule, and importing the processing result into the database. The method is used to reduce memory usage and avoid the risk of memory overflow.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and processor for importing or exporting table-type file data into or from a database. Background Art

[0002] Tabular files are widely used in the field of data storage and management. In the field of programming, data is often recorded in tabular files to achieve batch upload and then stored in the database. In many applications, fast and efficient import and export of tabular file data is crucial to improving application performance and user experience.

[0003] However, existing import and export methods often have performance bottlenecks, especially when processing large-scale data, which may lead to problems such as high memory usage and slow processing speed. In particular, when the table-type file data is large, the memory usage is relatively large, and there may be a risk of memory overflow. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device and processor for importing or exporting table file data into or from a database, wherein the method is used to reduce memory usage and avoid the risk of memory overflow.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a method for importing or exporting data from a table file into or from a database, comprising: when importing into a database, performing the following operations: receiving a table file; performing format verification on the table file and extracting first structural information of the table file, wherein the first structural information includes a first data type; based on the first structural information, performing block reading and data stream processing on the table file to obtain each corresponding row data or column data; and for each row data or column data, mapping each data in the row data or column data from the first data type to the second data type through a pre-set data mapping rule, and importing the processing result into the database The library performs the following operations when exporting a database: determining the data to be exported; extracting the second structural information of the data to be exported, wherein the second structural information includes a second data type; based on the second structural information, performing block reading and data stream processing on the data to be exported to obtain each corresponding row data or column data; and for each row data or column data, mapping each data in the row data or column data from the second data type to the first data type through a pre-set data mapping rule, and exporting the processing result to a table file, wherein the first data type is the data type of the data in the table file, and the second data type is the data type of the data in the database.

[0006] In an embodiment of the present application, the first structural information also includes the number of tables and the number of rows and columns, and the second structural information also includes the number of rows and columns.

[0007] In an embodiment of the present application, the block reading and the data stream processing include: based on the number of tables and / or the number of rows and columns, dividing the tabular file or the data to be exported into data blocks less than or equal to a preset size; and creating a thread for each of the data blocks, loading the data blocks into the memory based on multi-threading technology, and applying data stream processing to each data block loaded into the memory to obtain each row of data or column of data of the tabular file or the data to be exported.

[0008] In an embodiment of the present application, based on the number of tables and / or the number of rows and columns, dividing the tabular file or the data to be exported into data blocks less than or equal to a preset size includes: based on the number of tables and / or the number of rows and columns, performing an equal division operation on the tabular file or the data to be exported to obtain two data blocks of the same size; and determining whether each of the equally divided data blocks is less than or equal to a preset size, if not, repeating the equal division operation on each of the equally divided data blocks until each of the equally divided data blocks is less than or equal to the preset size.

[0009] In an embodiment of the present application, the method further includes: when performing the equal division operation, numbering each data block obtained by the equal division; and based on the number, importing each data block into a preset position of the database or exporting it to a preset position of a table file.

[0010] In an embodiment of the present application, the method further includes: performing lock management on the read and write operations of the file when executing the import database or the export database.

[0011] In an embodiment of the present application, the method further includes: when performing format verification on the table-type file, if the format verification fails, stopping processing of the table-type file and issuing an alarm.

[0012] In the embodiment of the present application, the table-type file is an Excel file.

[0013] A second aspect of the present application provides a device for importing or exporting data from a table-type file into or from a database, wherein the device for importing data from a table-type file into a database comprises: a first processing module for receiving a table-type file; a second processing module for performing format verification on the table-type file and extracting first structural information of the table-type file, wherein the first structural information comprises a first data type; a third processing module for performing block reading and data stream processing on the table-type file based on the first structural information to obtain each corresponding row of data or column of data; and a fourth processing module for mapping each data in the row of data or column of data from a first data type to a second data type through a pre-set data mapping rule for each row of data or column of data, and importing the processing result into the database, wherein the table-type The device for exporting file data to a database includes: a fifth processing module, used to determine the data to be exported; a sixth processing module, used to extract second structural information of the data to be exported, wherein the second structural information includes a second data type; a seventh processing module, used to read the data to be exported in blocks and perform data stream processing based on the second structural information to obtain each corresponding row data or column data; and an eighth processing module, used to map each data in the row data or column data from the second data type to the first data type through a pre-set data mapping rule for each row data or column data, and export the processing result to a table file, wherein the first data type is the data type of the data in the table file, and the second data type is the data type of the data in the database.

[0014] In an embodiment of the present application, the first structural information also includes the number of tables and the number of rows and columns, and the second structural information also includes the number of rows and columns.

[0015] In an embodiment of the present application, the block reading and the data stream processing include: based on the number of tables and / or the number of rows and columns, dividing the tabular file or the data to be exported into data blocks less than or equal to a preset size; and creating a thread for each of the data blocks, loading the data blocks into the memory based on multi-threading technology, and applying data stream processing to each data block loaded into the memory to obtain each row of data or column of data of the tabular file or the data to be exported.

[0016] In an embodiment of the present application, based on the number of tables and / or the number of rows and columns, dividing the tabular file or the data to be exported into data blocks less than or equal to a preset size includes: based on the number of tables and / or the number of rows and columns, performing an equal division operation on the tabular file or the data to be exported to obtain two data blocks of the same size; and determining whether each of the equally divided data blocks is less than or equal to a preset size, if not, repeating the equal division operation on each of the equally divided data blocks until each of the equally divided data blocks is less than or equal to the preset size.

[0017] A third aspect of the present application provides a processor configured to execute the above-mentioned method of importing or exporting table-type file data into or from a database.

[0018] A fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configures the processor to execute the above-mentioned method of importing or exporting tabular file data into or from a database.

[0019] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the methods for importing or exporting tabular file data into or from a database.

[0020] Through the above technical solution, when importing or exporting table files into or from a database, the data to be processed is read in blocks and processed as a data stream. This processing method divides the data to be processed into blocks and then processes the data row by row or column by column in the memory, avoiding the construction of the entire document structure in the memory at one time, reducing memory usage, and helping to avoid the risk of memory overflow. At the same time, by processing data row by row or column by column, performance problems caused by loading a large amount of data at one time are avoided, and the response speed of the system is also improved.

[0021] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0023] Figure 1 A schematic diagram of a process for importing tabular file data into a database according to an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram of a process of exporting table-type file data to a database according to an embodiment of the present application is shown;

[0025] Figure 3 Another flowchart of a method for importing tabular file data into a database according to an embodiment of the present application is schematically shown;

[0026] Figure 4 A schematic diagram of a structure of a device for importing tabular file data into a database according to an embodiment of the present application is shown;

[0027] Figure 5 A schematic diagram of a structure of a device for exporting table-type file data to a database according to an embodiment of the present application is shown;

[0028] Figure 6 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown.

[0029] Description of Reference Numerals

[0030] 100- A device for importing tabular file data into a database;

[0031] 200- a device for exporting data from a table-type file to a database;

[0032] 101-first processing module;

[0033] 102-second processing module;

[0034] 103-third processing module;

[0035] 104- fourth processing module;

[0036] 201- fifth processing module;

[0037] 202-sixth processing module;

[0038] 203-seventh processing module;

[0039] 204-eighth processing module. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0042] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0043] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In addition, it should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, and they should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0044] Figure 1 The following is a schematic diagram showing a flow chart of a method for importing tabular file data into a database according to an embodiment of the present application. Figure 2 The flowchart of the method for exporting table-type file data to a database according to an embodiment of the present application is schematically shown. Figure 1 and Figure 2 As shown, in one embodiment of the present application, a method for importing or exporting table-type file data into or from a database is provided. In the method, when importing into a database, the following steps S101 to S104 are performed.

[0045] Step S101, receiving a table file.

[0046] Step S102 , performing format verification on the table-type file to ensure the validity and integrity of the file, and extracting first structural information of the table-type file, where the first structural information includes a first data type.

[0047] Step S103: Based on the first structural information, the table file is read in blocks and data streams are processed to obtain corresponding row data or column data.

[0048] Step S104 , for each row of data or column of data, map each data in the row of data or column of data from a first data type to a second data type according to a preset data mapping rule, and import the processing result into the database.

[0049] When exporting the database, the following steps S201 to S204 are performed.

[0050] Step S201, determining the data to be exported.

[0051] Step S202: extracting second structural information of the data to be exported, where the second structural information includes a second data type.

[0052] Step S203: Based on the second structure information, the data to be exported is read in blocks and processed as a data stream to obtain corresponding row data or column data.

[0053] Step S204, for each row of data or column of data, map each data in the row of data or column of data from the second data type to the first data type according to a preset data mapping rule, and export the processing result to a table file.

[0054] The first data type is the data type of data in a table file, and the second data type is the data type of data in a database.

[0055] In the past, when the ss.usermodel package was used to parse tabular files, such as Excel files, the efficiency was low and the memory usage was large. Similar to DOM in XML, this processing method would build the structure of the entire document in the memory, so memory overflow is prone to occur when processing large files. However, in most scenarios, such as when importing or exporting data into or from a database, it is not necessary to randomly access the nodes in the tabular file. In the method proposed in the present invention, the tabular file data that needs to be imported or exported is subjected to the block reading and data stream processing, and all the data to be processed is divided into multiple data blocks and loaded into the memory using block reading, thereby reducing the memory usage and helping to avoid memory overflow.

[0056] After the data is divided into blocks, the data stream processing mechanism is used to process the data row by row or column by column, which further reduces the memory usage. At the same time, by processing the data row by row or column by column, the performance problems caused by loading a large amount of data at one time are avoided, and it also helps to improve the response speed of the system. In addition, after a small amount of data is loaded into the memory, data mapping and transmission are performed. This cache mechanism speeds up data access and conversion, further improving the speed of data import or export.

[0057] According to a specific implementation manner, the first structural information further includes the number of tables and the number of rows and columns, and the second structural information further includes the number of rows and columns.

[0058] After obtaining the number of tables and the number of rows and columns, the total amount of data to be imported or exported can be counted, so that the data can be processed in blocks based on the total amount of data.

[0059] According to a specific implementation, the block reading and the data stream processing include: based on the number of tables and / or the number of rows and columns, dividing the tabular file or the data to be exported into data blocks less than or equal to a preset size; and creating a thread for each of the data blocks, loading the data blocks into memory based on multi-threading technology, and applying data stream processing to each data block loaded into the memory to obtain each row or column of data of the tabular file or the data to be exported.

[0060] The design of parallel processing is based on the idea of ​​divide-and-conquer algorithm. A large task is gradually divided into small tasks according to certain rules, and then each piece of data is processed step by step. This can greatly improve the efficiency of CPU usage, especially when the processor is multi-core. Based on multi-threading technology, multiple data blocks can be processed in parallel, thereby improving resource utilization and processing speed.

[0061] According to a specific implementation, based on the number of tables and / or the number of rows and columns, dividing the tabular file or the data to be exported into data blocks less than or equal to a preset size includes: based on the number of tables and / or the number of rows and columns, performing an equal division operation on the tabular file or the data to be exported to obtain two data blocks of the same size; and determining whether each of the equally divided data blocks is less than or equal to a preset size, and if not, repeating the equal division operation on each of the equally divided data blocks until each of the equally divided data blocks is less than or equal to the preset size.

[0062] Specifically, the created threads are managed by the ForkJoinPool thread pool. The threads call the compute method of the task, which can check whether the size of the data block is less than or equal to the preset size. If it is not small enough, the equal division operation will continue and the data will be divided into two new ForkJoinCalculators, which are also scheduled for execution by the ForkJoinPool. Therefore, this process can be repeated recursively, gradually splitting the data block into smaller data blocks until the size of the data block is less than or equal to the preset size. At this time, each data block is processed sequentially, and each data block can be traversed back to its root by the task binary tree implicitly created by the branching process.

[0063] It should be noted that if the data in the data block is an odd number and cannot be evenly divided, the method proposed in the present application will adaptively adjust. For example, when the data amount is N and N is an odd number, after performing the equal division operation, two data blocks with data amounts of (N+1) / 2 and (N-1) / 2 are obtained.

[0064] According to a specific implementation, the method further includes: when performing the equal division operation, numbering each data block obtained by the equal division; and based on the number, importing each data block into a preset position of the database or exporting it to a preset position of a table file.

[0065] It should be noted that the required numbering information is implicitly included in the task binary tree implicitly created in the branching process. If this method is adopted, no additional numbering processing is required. In other embodiments of the present invention, other methods can also be selected for numbering to avoid data confusion. The specific implementation method of numbering is a technology known to ordinary technicians in the relevant field, and it can be appropriately selected during implementation. The present invention will not discuss this in detail.

[0066] According to a specific implementation, the method further includes: performing lock management on the read and write operations of the file when executing the import or export of the database.

[0067] In multi-threaded processing, in order to avoid multiple threads from processing the same data at the same time, the method proposed in the present invention performs lock management on the read and write operations of the file. When one thread processes certain data, other threads cannot perform read and write operations at the same time, thereby ensuring the consistency and integrity of the data.

[0068] According to a specific implementation, the method further includes: when performing format verification on the table-type file, if the format verification fails, stopping processing of the table-type file and issuing an alarm.

[0069] When formatting a table file, if a data problem is found, the formatting verification will fail and the processing will not continue. This helps to avoid the impact of erroneous data on subsequent processing flows and also helps to avoid erroneous data being written into the database, thus reducing the subsequent processing costs caused by erroneous data. The alarm information can include the error type and error location so that relevant personnel can quickly locate and repair it.

[0070] Figure 1 The following is a flow chart of a method for importing table-type file data into a database in one embodiment. Figure 2 FIG. 1 is a flow chart of a method for exporting table-type file data to a database in one embodiment. It should be understood that although Figure 1 or Figure 2The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 or Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0071] Figure 3 Another flow chart of the method for importing tabular file data into a database according to an embodiment of the present application is schematically shown. When data is stored in the database, the method can be regarded as follows: Figure 3 Several steps are shown. First, the client uploads the table file to be stored in the database. After receiving it, the system pre-processes the table file, including format verification and extraction of structural information. Then the system performs memory optimization, including block reading and data stream processing. After the uploaded table file is split, data mapping is performed. The memory optimization and data mapping are both based on multi-threaded technology for parallel processing. After the parallel processing is completed, the data storage is completed.

[0072] Figure 4 The structure block diagram of the apparatus for importing tabular file data into a database according to an embodiment of the present application is schematically shown. Figure 5 The structure block diagram of the apparatus for exporting table-type file data to a database according to an embodiment of the present application is schematically shown. Figure 4 and Figure 5 As shown, an embodiment of the present application provides a device for importing or exporting tabular file data into or from a database, wherein the device 100 for importing tabular file data into a database includes the following modules.

[0073] The first processing module 101 is used to receive a table file.

[0074] The second processing module 102 is used to perform format verification on the table-type file and extract first structural information of the table-type file, where the first structural information includes a first data type.

[0075] The third processing module 103 is used to read the table file in blocks and perform data stream processing based on the first structural information to obtain corresponding data in each row or column.

[0076] The fourth processing module 104 is used to map each data in the row data or column data from the first data type to the second data type according to a preset data mapping rule for each row data or column data, and import the processing result into the database.

[0077] The apparatus 200 for exporting table-type file data to a database includes the following modules.

[0078] The fifth processing module 201 is used to determine the data to be exported.

[0079] The sixth processing module 202 is used to extract second structural information of the data to be exported, where the second structural information includes a second data type.

[0080] The seventh processing module 203 is used to read the data to be exported in blocks and perform data stream processing based on the second structure information to obtain corresponding data in each row or column.

[0081] The eighth processing module 204 is used to map each data in the row data or column data from the second data type to the first data type according to a preset data mapping rule for each row data or column data, and export the processing result to a table file.

[0082] The first data type is the data type of data in a table file, and the second data type is the data type of data in a database.

[0083] According to a specific implementation manner, the first structural information further includes the number of tables and the number of rows and columns, and the second structural information further includes the number of rows and columns.

[0084] According to a specific implementation, the block reading and the data stream processing include: based on the number of tables and / or the number of rows and columns, dividing the tabular file or the data to be exported into data blocks less than or equal to a preset size; and creating a thread for each of the data blocks, loading the data blocks into memory based on multi-threading technology, and applying data stream processing to each data block loaded into the memory to obtain each row or column of data of the tabular file or the data to be exported.

[0085] According to a specific implementation, based on the number of tables and / or the number of rows and columns, dividing the tabular file or the data to be exported into data blocks less than or equal to a preset size includes: based on the number of tables and / or the number of rows and columns, performing an equal division operation on the tabular file or the data to be exported to obtain two data blocks of the same size; and determining whether each of the equally divided data blocks is less than or equal to a preset size, and if not, repeating the equal division operation on each of the equally divided data blocks until each of the equally divided data blocks is less than or equal to the preset size.

[0086] According to a specific implementation, the fourth processing module 104 and the eighth processing module 204 are also used to number each data block obtained by equal division when performing the equal division operation; and based on the number, import each data block into a preset position of the database or export it to a preset position of a table file.

[0087] According to a specific implementation, the apparatus 100 for importing table-type file data into a database and the apparatus 200 for exporting table-type file data into a database are further configured to perform lock management on the read and write operations of the file when executing the import or export operation.

[0088] According to a specific implementation, the third processing module 103 is further configured to perform format verification on the table-type file, and if the format verification fails, stop processing the table-type file and generate an alarm.

[0089] According to a specific implementation manner, the table-type file is an Excel file.

[0090] The device proposed by the present invention can support the processing of tabular files in various business scenarios, can process tabular files with different numbers of tables and rows and columns, and can friendly support tabular files with large amounts of data. In addition, the device is compatible with old versions of POI, without modifying the original code, thus reducing the cost of use.

[0091] The device 100 for importing tabular file data into a database includes a processor and a memory. The first processing module 101, the second processing module 102, the third processing module 103 and the fourth processing module 104 are all stored in the memory as program units, and the processor executes the above program modules stored in the memory to implement corresponding functions.

[0092] The device 200 for exporting table-type file data to a database includes a processor and a memory. The fifth processing module 201, the sixth processing module 202, the seventh processing module 203 and the eighth processing module 204 are all stored in the memory as program units, and the processor executes the above program modules stored in the memory to implement corresponding functions.

[0093] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the method of importing or exporting table file data into or out of the database can be implemented by adjusting kernel parameters.

[0094] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0095] An embodiment of the present application provides a storage medium on which a program is stored. When the program is executed by a processor, the method for importing or exporting the above-mentioned table file data into or out of the database is implemented.

[0096] An embodiment of the present application provides a processor, which is used to run a program, wherein the program executes the above-mentioned method of importing or exporting table file data into or from a database when running.

[0097] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, a method for importing or exporting data from a table file into or out of a database is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0098] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] In one embodiment, the apparatus 100 for importing table-type file data into a database and the apparatus 200 for exporting table-type file data from a database provided by the present application can be implemented in the form of a computer program. The computer program can be used in a computer program such as Figure 6 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the apparatus 100 for importing table-type file data into a database and the apparatus 200 for exporting table-type file data from a database, for example, Figure 4 The first processing module 101, the second processing module 102, the third processing module 103 and the fourth processing module 104 are shown, and Figure 5 The fifth processing module 201, the sixth processing module 202, the seventh processing module 203 and the eighth processing module 204 are shown. The computer program composed of each program module enables the processor to execute the steps of the method of importing or exporting table file data into or out of the database in each embodiment of the present application described in this specification.

[0100] Figure 6 The computer device shown can be Figure 4 The first processing module 101 of the apparatus 100 for importing table-type file data into a database executes step S101, the second processing module 102 executes step S201, the third processing module 103 executes step S103, the fourth processing module 104 executes step S104, and the Figure 5 In the apparatus 200 for exporting table-type file data to a database, the fifth processing module 201 executes step S201 , the sixth processing module 202 executes step S202 , the seventh processing module 203 executes step S203 , and the eighth processing module 204 executes step S104 .

[0101] An embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of importing or exporting table file data into or out of a database according to each embodiment of the present application is implemented.

[0102] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of initializing the method of importing or exporting table file data into or out of a database according to various embodiments of the present application.

[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0108] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0109] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0111] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for importing tabular file data into a database, characterized in that: The method comprises: Receive tabular documents; Performing format verification on the table-type file and extracting first structural information of the table-type file, wherein the first structural information includes a first data type; Based on the first structure information, the table file is read in blocks and data stream processed to obtain corresponding row data or column data; and For each row of data or column of data, each data in the row of data or column of data is mapped from a first data type to a second data type according to a preset data mapping rule, and the processing result is imported into the database, The first data type is the data type of data in a table file, and the second data type is the data type of data in a database.

2. A method for exporting table-type file data to a database, characterized in that: The method comprises: Determine the data to be exported; Extracting second structural information of the data to be exported, where the second structural information includes a second data type; Based on the second structure information, the data to be exported is read in blocks and processed as a data stream to obtain corresponding row data or column data; and For each row of data or column of data, each data in the row of data or column of data is mapped from the second data type to the first data type according to a preset data mapping rule, and the processing result is exported to a table file, The first data type is the data type of data in a table file, and the second data type is the data type of data in a database.

3. The method according to claim 1 or 2, characterized in that: The structural information also includes the number of tables and / or the number of rows and columns.

4. The method according to claim 3, characterized in that The block reading and the data stream processing include: Based on the number of tables and / or the number of rows and columns, the tabular file or the data to be exported is divided into data blocks of a size less than or equal to a preset size; and A thread is created for each of the data blocks, and based on multi-threading technology, the data blocks are loaded into the memory, and data stream processing is performed on each data block loaded into the memory to obtain each row or column of data of the tabular file or the data to be exported.

5. The method according to claim 4, characterized in that Based on the number of tables and / or the number of rows and columns, dividing the table file or the data to be exported into data blocks of a size less than or equal to a preset size includes: Based on the number of tables and / or the number of rows and columns, the table-type file or the data to be exported is evenly divided to obtain two data blocks of the same size; and It is determined whether each data block obtained by equal division is smaller than or equal to a preset size. If not, the equal division operation is repeatedly performed on each data block obtained by equal division until each data block obtained by equal division is smaller than or equal to the preset size.

6. The method according to claim 5, characterized in that The method further comprises: When performing the equal division operation, numbering each data block obtained by the equal division; and Based on the number, each data block is imported into a preset position of a database or exported to a preset position of a table file.

7. The method according to claim 1 or 2, characterized in that: The method further comprises: When the database import or database export is executed, lock management is performed on the read and write operations of the file.

8. The method according to claim 1, characterized in that The method further comprises: When format verification is performed on the table-type file, if the format verification fails, the processing of the table-type file is stopped and an alarm is issued.

9. The method according to claim 1 or 2, characterized in that: The table-type file is an Excel file.

10. A device for importing tabular file data into a database, characterized in that: The device comprises: A first processing module, used for receiving a table-type file; A second processing module is used to perform format verification on the table-type file and extract first structural information of the table-type file, where the first structural information includes a first data type; A third processing module is used to read the table file in blocks and perform data stream processing based on the first structure information to obtain corresponding data of each row or column; and a fourth processing module, for mapping each data in the row data or column data from a first data type to a second data type according to a preset data mapping rule for each row data or column data, and importing the processing result into the database, The first data type is the data type of data in a table file, and the second data type is the data type of data in a database.

11. A device for exporting table-type file data to a database, characterized in that: The device comprises: A fifth processing module, used for determining the data to be exported; A sixth processing module, configured to extract second structural information of the data to be exported, wherein the second structural information includes a second data type; a seventh processing module, configured to read the data to be exported in blocks and perform data stream processing based on the second structure information to obtain corresponding row data or column data; and an eighth processing module, for mapping each data in the row data or column data from the second data type to the first data type according to a preset data mapping rule, and exporting the processing result to a table file, The first data type is the data type of data in a table file, and the second data type is the data type of data in a database.

12. The device according to claim 10 or 11, characterized in that The structural information also includes the number of tables and / or the number of rows and columns.

13. The device according to claim 12, characterized in that The block reading and the data stream processing include: Based on the number of tables and / or the number of rows and columns, the tabular file or the data to be exported is divided into data blocks of a size less than or equal to a preset size; and A thread is created for each of the data blocks, and based on multi-threading technology, the data blocks are loaded into the memory, and data stream processing is performed on each data block loaded into the memory to obtain each row or column of data of the tabular file or the data to be exported.

14. The device according to claim 13, characterized in that Based on the number of tables and / or the number of rows and columns, dividing the table file or the data to be exported into data blocks of a size less than or equal to a preset size includes: Based on the number of tables and / or the number of rows and columns, the table-type file or the data to be exported is evenly divided to obtain two data blocks of the same size; and It is determined whether each data block obtained by equal division is smaller than or equal to a preset size. If not, the equal division operation is repeatedly performed on each data block obtained by equal division until each data block obtained by equal division is smaller than or equal to the preset size.

15. A processor, characterized in that: The method is configured to execute the method for importing or exporting table-type file data into or from a database according to any one of claims 1 to 9.

16. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to execute the method for importing or exporting table file data into or out of a database according to any one of claims 1 to 9.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method for importing or exporting table file data into or out of a database according to any one of claims 1 to 9.

Images