Data export system, data export method, electronic equipment and storage medium

Abstract

The invention discloses a data exporting system, a data exporting method, electronic equipment and a storage medium. In the system, a paging server is used for paging to-be-exported data, obtaining exported data corresponding to each page and recording the total page number; the query server is used for creating a data query task for each page; storing the data query tasks in a preset task set in sequence; creating parallel threads according to self configuration; obtaining and executing the data query tasks from the task set in parallel in sequence by utilizing a parallel thread to obtain target data; sequentially pushing the target data as a to-be-written task to a preset message queue; the import server is used for creating parallel threads according to the total page number; obtaining the to-be-written tasks from the message queue in parallel in sequence by using a parallel thread; performing format processing on the data in each to-be-written task to obtain target written data; and writing the target write-in data into the target file.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a data export system, a data export method, an electronic device and a storage medium. Background Art

[0002] Data export refers to the process of exporting data stored in a computer system or database from one system or database to another computer system, database or file in a readable format or file. Data export can be used for a variety of purposes, such as data backup, data sharing, data analysis, and porting data to different applications or platforms. For example, exporting business data reports from a database to an Excel file to facilitate secondary sorting and analysis of the business data reports. However, the inventors have found that existing data export methods usually directly reference tool sets provided by third parties. For example, the Excel export toolkit in NPOI, users only need to provide the data to be exported to a third-party tool set to complete the operation of exporting the data to be exported to an Excel file. However, since third-party tool sets basically have the following problems: 1. When multi-threading is used to perform data export in order to improve data export efficiency, the execution order of multiple threads may be different, and the existing third-party tool set does not have a mechanism to control the order in which each thread writes data. Therefore, there is a problem that data consistency cannot be guaranteed, that is, it cannot be guaranteed that the data exported to the Excel file is consistent with the data to be exported; 2. When the amount of data to be exported is large, the existing third-party tool set will be very slow in querying data, which will slow down the data format of the data to be exported in the system, resulting in a slow speed, low efficiency and long time to export the data to Excel files; 3. During the export process, the server resources are highly occupied, which not only affects the user experience of other functions in the system, but also poses a risk of causing system crashes.

[0003] Due to the above problems with the existing third-party tool sets, the existing data export methods cannot guarantee data consistency, are very inefficient, take a long time, affect the user experience of other functions in the system, and have the risk of causing system crashes. Summary of the invention

[0004] The embodiments of the present invention provide a data export system, a data export method, an electronic device and a storage medium, which are used to solve the problems existing in the prior art, such as the inability to ensure data consistency, low efficiency, long time consumption, impact on the user experience of other functions in the system and the risk of causing system crash.

[0005] The specific technical solutions are as follows: In a first aspect, an embodiment of the present invention provides a data export system, including: a paging server, a query server and an import server; The paging server is used to obtain a data export request; determine the data to be exported according to the export request; paging the data to be exported according to the preset data volume of each page, obtain the export data corresponding to each page, and record the total number of pages; The query server is used to obtain the paging results of the paging server; according to the paging results, a data query task is created for each paging, and the page number of the paging is used as the identifier of the data query task corresponding to the paging; according to the identifier of each data query task, the data query task is stored in a preset task set in sequence; a parallel thread is created according to its own configuration; using the parallel thread, the data query tasks are obtained from the task set in sequence and in parallel, and the data query tasks are executed to obtain target data; according to the identifier of each data query task, the target data corresponding to each data query task is pushed to the preset message queue in sequence as a task to be written; wherein the message queue is a queue in the producer-consumer mode; The import server is used to obtain the total number of pages from the paging server; create parallel threads according to the total number of pages; use the parallel threads to obtain the tasks to be written from the message queue in parallel in sequence; format the data in each task to be written according to a preset writing format to obtain target writing data; and write the target writing data into a target file.

[0006] Optionally, the import server is further used to: Before writing the target write data into the target file, determine whether the previous target write data has been successfully written into the target file; if so, write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable; if not, wait until the previous target write data is successfully written into the target file, and add 1 to the value of the record variable, then write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable.

[0007] In a second aspect, an embodiment of the present invention provides a data export method, which is applied to a paging server in the data export system described in the first aspect, wherein the data export system further includes a query server and an import server, and the method includes: Get data export request; Determining data to be exported according to the export request; According to the preset data volume of each page, the data to be exported is paged, the export data corresponding to each page is obtained, and the total number of pages is recorded; so that the query server can obtain the paging results of the paging server, create a data query task for each page, and use the page number of the page as the identifier of the data query task corresponding to the page, according to the identifier of each data query task, the data query task is stored in a preset task set in sequence, a parallel thread is created according to its own configuration, and the parallel thread is used to obtain the data query tasks from the task set in sequence and in parallel, and execute the data query tasks , obtain the target data, and according to the identifier of each data query task, push the target data corresponding to each data query task as a task to be written to the preset message queue in sequence, wherein the message queue is a queue in the producer consumer mode; so that the import server can obtain the total number of pages from the paging server, create parallel threads according to the total number of pages, and use the parallel threads to obtain the tasks to be written from the message queue in sequence and in parallel, perform format processing on the data in each task to be written according to the preset writing format, obtain the target writing data, and write the target writing data into the target file.

[0008] In a third aspect, an embodiment of the present invention provides a data export method, which is applied to a query server in the data export system described in the first aspect, wherein the data export system further includes a paging server and an import server, and the method includes: Obtaining paging results from the paging server; According to the paging result, a data query task for each paging is created, and the page number of the paging is used as the identifier of the data query task corresponding to the paging; According to the identifier of each data query task, the data query task is stored in a preset task set in sequence; Create parallel threads according to their own configuration; Using the parallel threads, sequentially and in parallel, obtain data query tasks from the task set, and execute the data query tasks to obtain target data; According to the identifier of each data query task, the target data corresponding to each data query task is taken as a task to be written and pushed to the preset message queue in sequence; so that the import server can use parallel threads to obtain the tasks to be written from the message queue in sequence and in parallel, and format the data in each task to be written according to the preset writing format to obtain the target writing data, and write the target writing data into the target file, wherein the message queue is a queue in producer consumer mode; the parallel threads of the import server are created according to the total number of pages obtained from the paging server.

[0009] Optionally, the step of creating a data query task for each page according to the paging result, and using the page number of the page as an identifier of the data query task corresponding to the page, includes: According to the exported data corresponding to each page, determine the data query statement corresponding to each page; The data query statement corresponding to each paging is used as a data query task, and the page number of the paging is used as the identifier of the data query task corresponding to the paging.

[0010] In a fourth aspect, an embodiment of the present invention provides a data export method, which is applied to an import server in the data export system described in the first aspect, wherein the data export system further includes a paging server and a query server, and the method includes: Obtaining the total number of pages from the paging server; Creating parallel threads according to the total number of pages; Using the parallel threads, the tasks to be written are obtained from the message queue in parallel and in sequence; According to the preset writing format, the data in each task to be written is formatted to obtain the target writing data; The target write data is written into the target file.

[0011] Optionally, before the step of writing the target write data into the target file, the method further includes: Determine whether the last target written data has been successfully written into the target file; If yes, write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable; If not, wait until the last target write data is successfully written into the target file, and increase the value of the record variable by 1, then write the target write data into the target file, and after the target write data is successfully written into the target file, increase the value of the preset record variable by 1.

[0012] In a fifth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement any of the above-mentioned data export method steps when executing the program stored in the memory.

[0013] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the above-mentioned steps of the method for exporting data.

[0014] In a seventh aspect, an embodiment of the present invention provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned method steps for exporting data.

[0015] In summary, in the embodiment of the present invention, the paging server paginates the data to be exported; the query server uses multiple parallel threads to process the data query tasks corresponding to the multiple paging in parallel, thereby doubling the data query speed. For example, if 10 threads concurrently execute the data query task in parallel, the data query speed will be at least increased by 5-10 times. The import server also uses multiple parallel threads to write the export data to the target file in parallel, which also doubles the data import speed. For example, if 10 threads concurrently write the export data to the target file in parallel, the data import speed will be at least increased by 5-10 times. Therefore, the system provided by the embodiment of the present invention greatly improves the efficiency of mass data export. In addition, if the user has higher requirements for the data export speed, it is only necessary to expand the server resource configuration and increase the upper limit of the number of parallel threads. Therefore, the system provided by the embodiment of the present invention is also highly flexible and can better meet the personalized needs of users.

[0016] In addition, in the query server, the binding of the page number and the data query task identifier ensures the order consistency of the tasks to be written and the paging pushed to the message queue, and the setting of the variables recorded in the import server ensures the consistency of the order of the data written to the target file and the exported data. Therefore, the system provided by the embodiment of the present invention solves the problem that the prior art cannot ensure the order consistency of the data written to the target file and the exported data.

[0017] In addition, in an embodiment of the present invention, the message queue is a queue in the producer-consumer mode. Therefore, after a consumer thread completes the operation of pushing the task to be written to the message queue, the consumer thread can be immediately destroyed and resources can be recovered, so that the configured resources of the server can be seamlessly connected and used alternately, thereby achieving the beneficial effect of not wasting resources and improving server performance within the scope of limited resources.

[0018] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0019] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 A flow chart of a method of applying the data export method provided by an embodiment of the present invention to a paging server in a data export system; Figure 2 A flow chart of a method in which the data export method provided by an embodiment of the present invention is applied to a query server in a data export system; Figure 3 A flow chart of a method of applying the data export method provided in an embodiment of the present invention to an import server in a data export system; Figure 4 A schematic diagram of the internal structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0022] The embodiments of the present invention provide a data export system, a data export method, an electronic device, and a storage medium, which ensure the consistency of data and greatly improve the efficiency of data export during the data export process. The data export system, the data export method, the electronic device, and the storage medium provided by the embodiments of the present invention are described in detail below.

[0023] A data export system provided by an embodiment of the present invention includes: a paging server, a query server and an import server; The paging server is used to obtain a data export request; determine the data to be exported according to the export request; paginate the data to be exported according to the preset data volume of each page, obtain the export data corresponding to each page, and record the total number of pages; In actual applications, the data export request may include the data to be exported, and the data to be exported may include fields to be exported and screening conditions, etc.

[0024] Specifically, after obtaining the data export request, the paging server can first determine the total amount of data to be exported according to the data export request; then calculate the total number of pages of the data to be exported after paging according to the total amount of data to be exported and the preset data amount of each page; paging the data to be exported, obtain the exported data corresponding to each page, and record the total number of pages.

[0025] In a specific embodiment, the amount of data in each page can be set to 10,000 records, and the data to be exported can be paged according to the total amount of data to be exported, with 10,000 records per page. For example, the 1st to 10,000th data records in the data to be exported are allocated to the first page with page number 1; the 10,001st to 20,000th data records in the data to be exported are allocated to the second page with page number 2, and so on.

[0026] The query server is used to obtain the paging results of the paging server; according to the paging results, a data query task is created for each paging, and the page number of the paging is used as the identifier of the data query task corresponding to the paging; according to the identifier of each data query task, the data query task is stored in a preset task set in sequence; a parallel thread is created according to its own configuration; using the parallel thread, the data query task is obtained from the task set in parallel in sequence, and the data query task is executed to obtain the target data; according to the identifier of each data query task, the target data corresponding to each data query task is pushed to the preset message queue in sequence as a task to be written; the message queue is a queue in the producer-consumer mode; Specifically, the query server is used to determine the data query statement corresponding to each page according to the exported data corresponding to each page; the data query statement corresponding to each page is taken as a data query task, and is stored in a preset task set in the order of page numbers. For example, according to the exported data corresponding to each page, the SQL statement corresponding to each page is determined; the SQL statement corresponding to each page is taken as an SQL statement task, and is stored in the preset task set in the order of page numbers.

[0027] Specifically, the query server is used to use parallel threads to obtain SQL statement tasks from the task set in parallel and execute the SQL statement tasks. While monitoring whether each parallel thread has completed the execution of the SQL statement task, the identifier of the SQL statement task corresponding to each parallel thread is compared. According to the identifier of the SQL statement task corresponding to each thread, the execution result corresponding to each thread, that is, the target data queried by each thread, is pushed to the preset message queue in sequence as a task to be written. In a specific embodiment, the tasks to be written can be pushed to the message queue middleware RabbitMQ in sequence. RabbitMQ is a message queue middleware; the message producer stores the produced messages in RabbitMQ, and the message consumer takes the messages from RabbitMQ.

[0028] In a specific embodiment, the query server creates parallel threads according to the maximum number of parallel threads supported by its own server configuration; for example, the query server calculates that the maximum number of parallel threads supported by itself is 2000 according to its own server configuration, and then creates 2000 parallel threads for obtaining data query tasks from the task set in parallel according to the order of data query tasks.

[0029] An import server is used to obtain the total number of pages from a paging server; a parallel thread is created according to the total number of pages; the tasks to be written are obtained from a message queue in parallel and in sequence using the parallel thread; the data in each task to be written is formatted according to a preset writing format to obtain target writing data; and the target writing data is written into a target file.

[0030] Specifically, the import server can create multiple parallel threads according to the total number of pages corresponding to the data to be exported, and obtain the tasks to be written in the message queue middleware RabbitMQ in parallel; according to the preset write format, the data in each task to be written is formatted to obtain the target write data; the target write data is written into the target file. For example, if the target file is an Excel file, then the preset write format is the Excel format; the developer will establish a corresponding relationship with a column in the Excel table for each field of the exported data in advance to obtain the write format. The import server formats the data in each task to be written according to the preset write format, converts the data format in the data source into the Excel format, and then writes the data into the Excel file, thereby achieving the purpose of exporting the data into the Excel file.

[0031] In a specific embodiment, the import server can also be used to: Before writing the target write data into the target file, determine whether the previous target write data has been successfully written into the target file; if so, write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable; if not, wait until the previous target write data is successfully written into the target file, and add 1 to the value of the record variable, then write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable.

[0032] In order to ensure that the order of the exported data is consistent with the data written into the target file, the embodiment of the present invention further sets a record variable, wherein the record variable is a thread-safe object, and the initial value of the record variable is 0. Before writing the target write data into the target file, the import server will determine whether the previous target write data has been successfully written into the target file; if so, the target write data will be written into the target file, and after the target write data is successfully written into the target file, the value of the preset record variable will be increased by 1; if not, wait until the previous target write data is successfully written into the target file, and after the value of the record variable is increased by 1, the target write data will be written into the target file, and after the target write data is successfully written into the target file, the value of the preset record variable will be increased by 1.

[0033] Since in the embodiment of the present invention, the target write data is written into the target file in sequence according to the identification order, that is, according to the page number order, when a target write data is successfully written into the target file, the value of the record variable will be increased by 1. Therefore, in actual application, before writing the target write data into the target file, the import server can judge whether the previous target write data has been successfully written into the target file according to the value of the record variable. For example, the data to be written into the target file this time is the target write data identified as 4, then, before writing the target write data identified as 4 into the target file, the import server will first judge whether the value of the record variable has become 3, if yes, it means that the previous target write data, that is, the target write data identified as 3, has been successfully written into the target file, at this time, the thread can write the target write data identified as 4 into the target file; if not, it is necessary to wait for the target write data identified as 3 to be successfully written into the target file, and the value of the record variable is increased by 1 and becomes 3, and then the thread can write the target write data identified as 4 into the target file.

[0034] In actual applications, the paging server, query server and import server in the data export system provided by the embodiment of the present invention have a communication connection relationship established between them, wherein the query server plays the role of a message producer, and the import server plays the role of a message consumer. Of course, in a specific embodiment, only two servers can be designed: a query server and an import server; let the query server take on the tasks of the paging server, so that the data export system provided by the embodiment of the present invention only includes a query server and an import server.

[0035] In summary, in the embodiment of the present invention, the paging server paginates the data to be exported; the query server uses multiple parallel threads to process the data query tasks corresponding to the multiple paging in parallel, thereby doubling the data query speed. For example, if 10 threads concurrently execute the data query task in parallel, the data query speed will be increased by at least 5-10 times. The import server also uses multiple parallel threads to write the exported data to the target file in parallel, which also doubles the data import speed. For example, if 10 threads concurrently write the exported data to the target file in parallel, the data import speed will be increased by at least 5-10 times. Therefore, the system provided by the embodiment of the present invention greatly improves the efficiency of mass data export. The system provided by the embodiment of the present invention has passed the experiment, and it has been verified that the data export speed of the system provided by the embodiment of the present invention has been shortened from the original 30-60 minutes to 3-5 minutes, and the data written in the target file is completely consistent with the exported data. Therefore, it is proved that the system provided by the embodiment of the present invention greatly improves the data export efficiency and ensures the consistency of the data.

[0036] In addition, if the user has higher requirements for data export speed, it is only necessary to expand the server resource configuration and increase the upper limit of parallel threads. Therefore, the system provided by the embodiment of the present invention is also highly flexible and can better meet the personalized needs of users.

[0037] In addition, in the query server, the binding of the page number and the data query task identifier ensures the order consistency of the tasks to be written and the paging pushed to the message queue, and the setting of the variables recorded in the import server ensures the consistency of the order of the data written to the target file and the exported data. Therefore, the system provided by the embodiment of the present invention solves the problem that the prior art cannot ensure the order consistency of the data written to the target file and the exported data.

[0038] In addition, in an embodiment of the present invention, the message queue is a queue in the producer-consumer mode. Therefore, after a consumer thread completes the operation of pushing the task to be written to the message queue, the consumer thread can be immediately destroyed and resources can be recovered, so that the configured resources of the server can be seamlessly connected and used alternately, thereby achieving the beneficial effect of not wasting resources and improving server performance within the scope of limited resources.

[0039] Figure 1 A flow chart of a method for applying the data export method provided in an embodiment of the present invention to the paging server in the above data export system is shown in FIG. Figure 1 As shown, the method includes: S101: Obtaining a data export request; In actual applications, the data export request may include the data to be exported, and the data to be exported may include fields to be exported and screening conditions, etc.

[0040] S102: Determine the data to be exported according to the export request; S103: paging the data to be exported according to the preset data volume of each page, obtaining the export data corresponding to each page, and recording the total number of pages; so that the query server can obtain the paging result of the paging server, create a data query task for each page according to the paging result, and use the page number of the page as the identifier of the data query task corresponding to the page, store the data query tasks in a preset task set in sequence according to the identifier of each data query task, create parallel threads according to its own configuration, use the parallel threads to obtain the data query tasks from the task set in sequence and in parallel, and execute the data query tasks to obtain the target data, and according to the identifier of each data query task, push the target data corresponding to each data query task as a task to be written to the preset message queue in sequence, wherein the message queue is a queue in the producer-consumer mode; so that the import server can obtain the total number of pages from the paging server, create parallel threads according to the total number of pages, use the parallel threads to obtain the tasks to be written from the message queue in sequence and in parallel, format the data in each task to be written according to the preset writing format, obtain the target writing data, and write the target writing data into the target file.

[0041] Specifically, after obtaining the data export request, the paging server can first determine the total amount of data to be exported according to the data export request; then calculate the total number of pages of the data to be exported after paging according to the total amount of data to be exported and the preset data amount of each page; paging the data to be exported, obtain the exported data corresponding to each page, and record the total number of pages.

[0042] In a specific embodiment, the amount of data in each page can be set to 10,000 records, and the data to be exported can be paged according to the total amount of data to be exported, with 10,000 records per page. For example, the 1st to 10,000th data records in the data to be exported are allocated to the first page with page number 1; the 10,001st to 20,000th data records in the data to be exported are allocated to the second page with page number 2, and so on.

[0043] Figure 2 A flow chart of a method for applying the data export method provided in an embodiment of the present invention to a query server in the above data export system is shown as follows: Figure 2 As shown, the method includes: S201: Obtaining paging results from a paging server; S202: creating a data query task for each page according to the paging result, and using the page number of the page as an identifier of the data query task corresponding to the page; S203: according to the identifier of each data query task, sequentially store the data query tasks into a preset task set; Specifically, the query server can determine the data query statement corresponding to each page according to the exported data corresponding to each page; the data query statement corresponding to each page is taken as a data query task, and is stored in a preset task set in the order of page numbers. For example, according to the exported data corresponding to each page, the SQL statement corresponding to each page is determined; the SQL statement corresponding to each page is taken as an SQL statement task, and is stored in the preset task set in the order of page numbers.

[0044] S204: Create a parallel thread according to its own configuration; In a specific embodiment, the query server creates parallel threads according to the maximum number of parallel threads supported by its own server configuration; for example, the query server calculates that the maximum number of parallel threads supported by itself is 2000 according to its own server configuration, and then creates 2000 parallel threads for obtaining data query tasks from the task set in parallel according to the order of data query tasks.

[0045] S205: using parallel threads to sequentially and concurrently obtain data query tasks from the task set, and executing the data query tasks to obtain target data; S206: According to the identifier of each data query task, the target data corresponding to each data query task is taken as a task to be written and pushed to the preset message queue in sequence; so that the import server can use parallel threads to obtain the tasks to be written from the message queue in parallel in sequence, and format the data in each task to be written according to the preset writing format to obtain the target writing data, and write the target writing data into the target file, wherein the message queue is a queue in the producer consumer mode; the parallel threads of the import server are created according to the total number of pages obtained from the paging server.

[0046] Specifically, the steps of obtaining the paging result of the paging server, creating a data query task for each paging, using the page number of the paging as the identifier of the data query task corresponding to the paging, and storing the data query tasks in a preset task set in sequence according to the identifier of each data query task may include: According to the exported data corresponding to each page, determine the data query statement corresponding to each page; The data query statement corresponding to each paging is taken as a data query task, and the page number of the paging is taken as the identifier of the data query task corresponding to the paging. According to the identifier of each data query task, the data query tasks are sequentially stored in a preset task set.

[0047] In a specific embodiment, the query server can use parallel threads to obtain SQL statement tasks from the task set in parallel and execute the SQL statement tasks. While monitoring whether each parallel thread has completed the SQL statement task, the identifier of the SQL statement task corresponding to each parallel thread is compared. According to the identifier of the SQL statement task corresponding to each thread, the execution result corresponding to each thread, that is, the target data queried by each thread, is pushed to the preset message queue in sequence as a task to be written. In a specific embodiment, the tasks to be written can be pushed to the message queue middleware RabbitMQ in sequence.

[0048] Figure 3 A flow chart of a method for applying the data export method provided in an embodiment of the present invention to an import server in the above data export system, as shown in FIG. Figure 3 As shown, the method includes: S301: Obtain the total number of pages from the paging server; S302: Create parallel threads according to the total number of pages; S303: Using parallel threads, sequentially and in parallel, obtain tasks to be written from the message queue; S304: According to a preset writing format, format processing is performed on the data in each task to be written to obtain target writing data; S305: Write the target write data into the target file.

[0049] Specifically, the import server can create multiple parallel threads according to the total number of pages corresponding to the data to be exported, and obtain the tasks to be written in the message queue middleware RabbitMQ in parallel; according to the preset write format, the data in each task to be written is formatted to obtain the target write data; the target write data is written into the target file. For example, if the target file is an Excel file, then the preset write format is the Excel format; the developer will establish a corresponding relationship with a column in the Excel table for each field of the exported data in advance to obtain the write format. The import server formats the data in each task to be written according to the preset write format, converts the data format in the data source into the Excel format, and then writes the data into the Excel file, thereby achieving the purpose of exporting the data into the Excel file.

[0050] Specifically, before the step of writing the target data into the target file, the following steps may also be included: Determine whether the data written to the previous target has been successfully written into the target file; If yes, write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable; If not, wait until the last target write data is successfully written into the target file, and increase the value of the record variable by 1, then write the target write data into the target file, and after the target write data is successfully written into the target file, increase the value of the preset record variable by 1.

[0051] In a specific embodiment, in order to ensure that the order of the exported data is consistent with the data written into the target file, a record variable is also set, wherein the record variable is a thread-safe object, and the initial value of the record variable is 0. Before writing the target write data into the target file, the import server will determine whether the previous target write data has been successfully written into the target file; if so, the target write data is written into the target file, and after the target write data is successfully written into the target file, the value of the preset record variable is increased by 1; if not, wait until the previous target write data is successfully written into the target file, and after the value of the record variable is increased by 1, the target write data is written into the target file, and after the target write data is successfully written into the target file, the value of the preset record variable is increased by 1.

[0052] Since in the embodiment of the present invention, the target write data is written into the target file in sequence according to the identification order, that is, according to the page number order, when a target write data is successfully written into the target file, the value of the record variable will be increased by 1. Therefore, in actual application, before writing the target write data into the target file, the import server can judge whether the previous target write data has been successfully written into the target file according to the value of the record variable. For example, the data to be written into the target file this time is the target write data identified as 4, then, before writing the target write data identified as 4 into the target file, the import server will first judge whether the value of the record variable has become 3, if yes, it means that the previous target write data, that is, the target write data identified as 3, has been successfully written into the target file, at this time, the thread can write the target write data identified as 4 into the target file; if not, it is necessary to wait for the target write data identified as 3 to be successfully written into the target file, and the value of the record variable is increased by 1 and becomes 3, and then the thread can write the target write data identified as 4 into the target file.

[0053] Depend on Figure 1-3 As can be seen from the embodiment shown, in the method provided by the embodiment of the present invention, the paging server can paging the data to be exported; the query server can use multiple parallel threads to process the data query tasks corresponding to multiple paging in parallel, thereby doubling the query speed of the data. For example, if 10 threads concurrently execute the data query task in parallel, then the query speed of the data will be increased by at least 5-10 times. The import server can also use multiple parallel threads to write the export data to the target file in parallel, which also doubles the data import speed. For example, if 10 threads concurrently write the export data to the target file in parallel, then the data import speed will be increased by at least 5-10 times. Therefore, the method provided by the embodiment of the present invention greatly improves the efficiency of exporting massive data. In addition, if the user has higher requirements for the data export speed, it is only necessary to expand the server resource configuration and increase the upper limit of the number of parallel threads. Therefore, the method provided by the embodiment of the present invention is also highly flexible and can better meet the personalized needs of users.

[0054] In addition, in the method provided in the embodiment of the present invention, the page number in the query server is bound to the data query task identifier, thereby ensuring the order consistency of the tasks to be written and the paging pushed to the message queue, and the setting of the record variables in the import server ensures the consistency of the order of the data written to the target file and the exported data. Therefore, the method provided in the embodiment of the present invention solves the problem that the prior art cannot ensure the order consistency of the data written to the target file and the exported data.

[0055] In addition, in an embodiment of the present invention, the message queue is a queue in the producer-consumer mode. Therefore, after a consumer thread completes the operation of pushing the task to be written to the message queue, the consumer thread can be immediately destroyed and resources can be recovered, so that the configured resources of the server can be seamlessly connected and used alternately, thereby achieving the beneficial effect of not wasting resources and improving server performance within the scope of limited resources.

[0056] The embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. Memory 403, used for storing computer programs; The processor 401 is used to implement any data export method steps described in the above embodiments when executing the program stored in the memory 403.

[0057] In the electronic device provided by the embodiment of the present invention, the paging server can paginate the data to be exported; the query server can use multiple parallel threads to process the data query tasks corresponding to the multiple paging in parallel, thereby doubling the data query speed. For example, if 10 threads concurrently execute the data query task in parallel, the data query speed will be increased by at least 5-10 times. The import server can also use multiple parallel threads to write the export data to the target file in parallel, which also doubles the data import speed. For example, if 10 threads concurrently write the export data to the target file in parallel, the data import speed will be increased by at least 5-10 times. Therefore, the electronic device provided by the embodiment of the present invention greatly improves the efficiency of exporting massive data. In addition, if the user has higher requirements for the data export speed, it is only necessary to expand the server resource configuration and increase the upper limit of the number of parallel threads. Therefore, the electronic device provided by the embodiment of the present invention is also highly flexible and can better meet the personalized needs of users.

[0058] In addition, in the electronic device provided by the embodiment of the present invention, the page number in the query server is bound to the data query task identifier, thereby ensuring the order consistency of the tasks to be written and the paging pushed to the message queue, and the setting of the record variables in the import server ensures the consistency of the order of the data written to the target file and the exported data. Therefore, the electronic device provided by the embodiment of the present invention solves the problem that the prior art cannot ensure the order consistency of the data written to the target file and the exported data.

[0059] In addition, in an embodiment of the present invention, the message queue is a queue in the producer-consumer mode. Therefore, after a consumer thread completes the operation of pushing the task to be written to the message queue, the consumer thread can be immediately destroyed and resources can be recovered, so that the configured resources of the server can be seamlessly connected and used alternately, thereby achieving the beneficial effect of not wasting resources and improving server performance within the scope of limited resources.

[0060] In another embodiment provided by the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the data export method described in any one of the above embodiments.

[0061] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the data export method described in any one of the above embodiments.

[0062] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0063] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data export system, characterized in that: include: Paging server, query server and import server; The paging server is used to obtain a data export request; Determining data to be exported according to the export request; Paging the data to be exported according to the preset data volume of each page, obtaining the exported data corresponding to each page, and recording the total number of pages; The query server is used to obtain the paging results of the paging server; according to the paging results, create a data query task for each paging, and use the page number of the paging as the identifier of the data query task corresponding to the paging; according to the identifier of each data query task, store the data query tasks in a preset task set in sequence; Create parallel threads according to their own configuration; use the parallel threads to obtain data query tasks from the task set in parallel and execute the data query tasks to obtain target data; according to the identifier of each data query task, push the target data corresponding to each data query task as a task to be written to a preset message queue in sequence; wherein the message queue is a queue in producer-consumer mode; The import server is used to obtain the total number of pages from the paging server; create parallel threads according to the total number of pages; use the parallel threads to obtain the tasks to be written from the message queue in parallel in sequence; perform format processing on the data in each task to be written according to a preset writing format to obtain target writing data; The target write data is written into the target file.

2. The system according to claim 1, characterized in that The import server is further used for: Before writing the target write data into the target file, determine whether the previous target write data has been successfully written into the target file; if so, write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable; if not, wait until the previous target write data is successfully written into the target file, and add 1 to the value of the record variable, then write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable.

3. A data export method, characterized in that: The paging server used in the data export system of claim 1, wherein the data export system further comprises a query server and an import server, and the method comprises: Get data export request; Determining data to be exported according to the export request; According to the preset data volume of each page, the data to be exported is paged, the export data corresponding to each page is obtained, and the total number of pages is recorded; so that the query server can obtain the paging results of the paging server, create a data query task for each page, and use the page number of the page as the identifier of the data query task corresponding to the page, according to the identifier of each data query task, the data query task is stored in a preset task set in sequence, a parallel thread is created according to its own configuration, and the parallel thread is used to obtain the data query tasks from the task set in sequence and in parallel, and execute the data query tasks , obtain the target data, and according to the identifier of each data query task, push the target data corresponding to each data query task as a task to be written to the preset message queue in sequence, wherein the message queue is a queue in the producer consumer mode; so that the import server can obtain the total number of pages from the paging server, create parallel threads according to the total number of pages, and use the parallel threads to obtain the tasks to be written from the message queue in sequence and in parallel, perform format processing on the data in each task to be written according to the preset writing format, obtain the target writing data, and write the target writing data into the target file.

4. A data export method, characterized in that: The query server used in the data export system of claim 1, wherein the data export system further comprises a paging server and an import server, and the method comprises: Obtaining paging results from the paging server; According to the paging result, a data query task for each paging is created, and the page number of the paging is used as the identifier of the data query task corresponding to the paging; According to the identifier of each data query task, the data query task is stored in a preset task set in sequence; Create parallel threads according to their own configuration; Using the parallel threads, sequentially and in parallel, obtain data query tasks from the task set, and execute the data query tasks to obtain target data; According to the identifier of each data query task, the target data corresponding to each data query task is taken as a task to be written and pushed to the preset message queue in sequence; so that the import server can use parallel threads to obtain the tasks to be written from the message queue in sequence and in parallel, and format the data in each task to be written according to the preset writing format to obtain the target writing data, and write the target writing data into the target file, wherein the message queue is a queue in producer consumer mode; the parallel threads of the import server are created according to the total number of pages obtained from the paging server.

5. The method according to claim 4, characterized in that The step of creating a data query task for each page according to the paging result, and using the page number of the page as an identifier of the data query task corresponding to the page, includes: According to the exported data corresponding to each page, determine the data query statement corresponding to each page; The data query statement corresponding to each paging is used as a data query task, and the page number of the paging is used as the identifier of the data query task corresponding to the paging.

6. A data export method, characterized in that: The import server used in the data export system of claim 1, wherein the data export system further comprises a paging server and a query server, and the method comprises: Obtaining the total number of pages from the paging server; Creating parallel threads according to the total number of pages; Using the parallel threads, the tasks to be written are obtained from the message queue in parallel and in sequence; According to the preset writing format, the data in each task to be written is formatted to obtain the target writing data; The target write data is written into the target file.

7. The method according to claim 6, characterized in that Before the step of writing the target write data into the target file, the method further includes: Determine whether the last target written data has been successfully written into the target file; If yes, write the target write data into the target file, and after the target write data is successfully written into the target file, add 1 to the value of the preset record variable; If not, wait until the last target write data is successfully written into the target file, and increase the value of the record variable by 1, then write the target write data into the target file, and after the target write data is successfully written into the target file, increase the value of the preset record variable by 1.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method steps described in any one of claims 3-7 when executing the program stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 3 to 7 are implemented.

Images