Data export method and device, computer readable storage medium and electronic device

Abstract

The present disclosure relates to the field of data processing technology, and provides a data export method and device, a computer-readable storage medium, and an electronic device. The method includes: in response to a data export request from a client, obtaining export parameter information, and generating an export file identifier for the data to be exported, wherein the export parameter information includes query conditions for the data to be exported; obtaining the export path of the data to be exported through an asynchronous thread, and creating an export file according to the export path and the export file identifier; the asynchronous thread queries the data to be exported in pages according to the query conditions for the data to be exported, and writes the queried data to be exported into the export file in the order of the paged query to generate a target export file. This scheme performs data export operations based on asynchronous threads and paged queries, which can avoid the problem of memory overflow when exporting a large amount of data at one time, and at the same time, can improve the efficiency of data export.

Description

Background Art

[0002] In a big data management platform, different data exports are often required to perform different business needs analyses.

[0003] In the prior art, the client usually assembles query parameters to initiate a data export request, and the backend system business layer assembles SQL (Structured Query Language) according to the query parameters, queries all the data at once and returns it; the backend control layer and the model layer use the I / O (input / output) stream of the response (the object representing the response) to write the queried data, and the client downloads the data in the response, thereby realizing data export and download.

[0004] However, this method will increase the pressure on the database and cause memory overflow when the amount of exported data is large, such as millions of data, and there are problems such as low export efficiency and long export time.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a data export method and device, a computer-readable storage medium and an electronic device, thereby improving the problem of low efficiency in exporting large amounts of data, at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a data export method is provided, comprising:

[0009] In response to a data export request from a client, obtaining export parameter information and generating an export file identifier of the data to be exported, wherein the export parameter information includes a query condition of the data to be exported;

[0010] Obtaining an export path of the data to be exported through an asynchronous thread, and creating an export file according to the export path and the export file identifier;

[0011] The asynchronous thread queries the data to be exported in pages according to the query conditions of the data to be exported, and writes the queried data to be exported into the export file in the order of the page query to generate a target export file.

[0012] In an exemplary embodiment of the present disclosure, based on the above solution, the asynchronous thread queries the data to be exported in pages according to the query conditions of the data to be exported, including:

[0013] Get the amount of data to be exported and the number of preset asynchronous threads;

[0014] According to the amount of data to be exported and the number of preset asynchronous threads, allocating a sub-amount of data to be exported to each asynchronous thread;

[0015] Each of the asynchronous threads queries the sub-data to be exported in pages in parallel according to the query conditions of the data to be exported and the amount of the sub-data to be exported;

[0016] The step of writing the queried data to be exported into the export file in the order of the paging query to generate a target export file includes:

[0017] Each of the asynchronous threads writes the queried sub-data to be exported into the sub-export files corresponding to each of the asynchronous threads in the order of the paging query;

[0018] The sub-export files are merged to generate a target export file.

[0019] In an exemplary embodiment of the present disclosure, based on the above solution, before writing the queried sub-data to be exported into the sub-export files corresponding to each of the asynchronous threads, the method further includes:

[0020] Generate a sub-export file identifier corresponding to each asynchronous thread according to the export file identifier and the thread identifier of each asynchronous thread;

[0021] The export path and the sub-export file identifier are obtained to create a sub-export file corresponding to each of the asynchronous threads.

[0022] In an exemplary embodiment of the present disclosure, based on the above solution, merging the sub-export files to generate a target export file includes:

[0023] Read each sub-export file according to the export path and each sub-export file identifier;

[0024] Each sub-export file read is written into the export file to generate a target export file.

[0025] In an exemplary embodiment of the present disclosure, based on the above solution, it is characterized in that the export parameter information also includes at least one of an export user identifier and an export time;

[0026] The step of generating an export file identifier for the data to be exported includes:

[0027] An export file identifier for the data to be exported is generated according to the export parameter information.

[0028] In an exemplary embodiment of the present disclosure, based on the above solution, after generating an export file identifier of the data to be exported, the method further includes:

[0029] Generate a target export task according to the export file identifier and the export parameter information;

[0030] The target export task is written into the export task information table, and the state of the target export task is configured as an exporting state.

[0031] In an exemplary embodiment of the present disclosure, based on the above solution, after generating the target export file, the method further includes:

[0032] Uploading the target export file to the storage platform;

[0033] Receiving the uniform resource locator of the target export file sent by the storage platform;

[0034] The uniform resource locator is written into the export task information table to associate the uniform resource locator with the export file identifier.

[0035] In an exemplary embodiment of the present disclosure, based on the above solution, the method further includes:

[0036] When the target export file is uploaded successfully, the status of the target export task corresponding to the export file identifier is updated from the exporting status to the success status;

[0037] When the target export file fails to be uploaded, the state of the target export task corresponding to the export file identifier is updated from the exporting state to the failed state, and the failure identifier of the failed state is recorded, wherein the failure identifier includes an export exception or an upload exception.

[0038] In an exemplary embodiment of the present disclosure, based on the above solution, the method further includes:

[0039] Returning the export file identifier to the client, where the export file identifier is used to instruct the client to determine the uniform resource locator of the target export file;

[0040] In response to a triggering operation of the client on the uniform resource locator, the target export file is downloaded at the client according to the uniform resource locator.

[0041] In an exemplary embodiment of the present disclosure, based on the above solution, after obtaining the derived parameter information, the method further includes:

[0042] Querying the export task information table to determine whether there is an export task in the export task information table with the same query condition as the query condition in the export parameter information;

[0043] If it exists, sending a prompt message to the client, wherein the prompt message is used to instruct the client to determine the uniform resource locator corresponding to the target export file of the export task;

[0044] In response to a triggering operation of the client on the uniform resource locator, the target export file is downloaded at the client according to the uniform resource locator.

[0045] According to a second aspect of the present disclosure, there is provided a data export device, comprising:

[0046] An export request response module is configured to respond to a data export request from a client, obtain export parameter information, and generate an export file identifier of the data to be exported, wherein the export parameter information includes a query condition of the data to be exported;

[0047] An export file creation module is configured to obtain an export path of the data to be exported through an asynchronous thread, and create an export file according to the export path and the export file identifier;

[0048] The target export file generation module is configured such that the asynchronous thread queries the data to be exported in pages according to the query conditions of the data to be exported, and writes the queried data to be exported into the export file in the order of the page query to generate the target export file.

[0049] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the data export method as described in the first aspect of the above embodiment is implemented.

[0050] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the data export method as described in the first aspect of the above embodiment.

[0051] It can be seen from the above technical solutions that the data export method, data export device, and computer-readable storage medium and electronic device for implementing the data export method in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

[0052] In the technical solutions provided by some embodiments of the present disclosure, first, in response to the data export request of the client, the export parameter information is obtained, and an export file identifier of the data to be exported is generated; then, the export path of the data to be exported is obtained through an asynchronous thread, and an export file is created according to the export path and the export file identifier; finally, the asynchronous thread queries the data to be exported in pages according to the query conditions, and writes the queried data to be exported into the export file in the order of the page query to generate a target export file. Compared with the related art, on the one hand, the present disclosure performs data export operations based on asynchronous threads, which can improve the efficiency of data export, thereby avoiding the problem of data being unable to be exported due to the interruption of the front-end and back-end network connections when synchronously exporting large data; on the other hand, the present disclosure is based on page query, which can reduce the impact on database performance when exporting large amounts of data, and solve the memory overflow problem that exists when exporting large amounts of data.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0055] Figure 1 A schematic flow chart showing a data export method of the prior art in an exemplary embodiment of the present disclosure;

[0056] Figure 2 A schematic diagram showing a flow chart of a data export method in an exemplary embodiment of the present disclosure;

[0057] Figure 3 A schematic flow chart showing a method for exporting data using multiple asynchronous threads in an exemplary embodiment of the present disclosure;

[0058] Figure 4 A schematic flow chart showing another data export method in an exemplary embodiment of the present disclosure;

[0059] Figure 5 A schematic diagram showing an interactive process of a data export method in an exemplary embodiment of the present disclosure;

[0060] Figure 6 A schematic diagram showing the structure of a data export device in an exemplary embodiment of the present disclosure is shown;

[0061] Figure 7A schematic diagram showing the structure of a computer storage medium in an exemplary embodiment of the present disclosure; and

[0062] Figure 8 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0063] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0064] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.

[0065] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0066] In related technologies, such as Figure 1As shown, when exporting data, in step S110, the web (World Wide Web) application of the client 11 assembles query parameters to initiate a data export request; in step S120, the backend control layer 12 forwards the data export request to the backend service layer 13; in step S130, the backend service layer 13 assembles SQL according to the query parameters; in step S140, the backend service layer 13 initiates a query request to the database 14 according to the assembled SQL; in step S150, the database 14 returns the queried data to the backend control layer 12; in step S160, the backend control layer 12 uses the response I / O stream to write the queried data; in step S170, the web application of the client 11 can download the data in the response. Thus, data export and download are realized.

[0067] However, when the amount of data to be exported is large, such as more than one million, searching for all the data at once will increase the database pressure, causing memory overflow or even application shutdown. At the same time, the data export method in the related technology has low export efficiency and long data export time, which will increase the possibility of failing to export data due to interruption of the front-end and back-end network connections, thereby reducing the stability of data export.

[0068] In the embodiments of the present disclosure, firstly, a data export method is provided to overcome the defects existing in the above-mentioned related technologies at least to a certain extent.

[0069] Figure 2 A flow chart showing a data export method in an exemplary embodiment of the present disclosure is shown, referring to Figure 2 , the method comprising:

[0070] Step S210, in response to a data export request from a client, obtaining export parameter information, and generating an export file identifier of the data to be exported, wherein the export parameter information includes a query condition of the data to be exported;

[0071] Step S220, obtaining the export path of the data to be exported through the asynchronous thread, and creating an export file according to the export path and the export file identifier;

[0072] Step S230, the asynchronous thread queries the data to be exported in pages according to the query conditions of the data to be exported, and writes the queried data to be exported into the export file in the order of the page query to generate a target export file.

[0073] exist Figure 2In the technical solution provided by the illustrated embodiment, first, in response to the data export request of the client, the export parameter information is obtained, and an export file identifier of the data to be exported is generated; then, the export path of the data to be exported is obtained through an asynchronous thread, and an export file is created according to the export path and the export file identifier; finally, the asynchronous thread queries the data to be exported by paging according to the query conditions, and writes the queried data to be exported into the export file in the order of the paging query to generate a target export file. Compared with the related art, on the one hand, the present disclosure performs data export operations based on asynchronous threads, which can improve the efficiency of data export, thereby avoiding the problem of data being unable to be exported due to the interruption of the front-end and back-end network connections when synchronously exporting large data; on the other hand, the present disclosure is based on paging queries, which can reduce the impact on database performance when exporting large amounts of data, and at the same time solve the memory overflow problem that exists when exporting large amounts of data.

[0074] The following Figure 2 The specific implementation methods of each step in the embodiment shown are described in detail:

[0075] In step S210, in response to a data export request from a client, export parameter information is acquired, and an export file identifier of the data to be exported is generated, wherein the export parameter information includes a query condition of the data to be exported.

[0076] Among them, the query condition may include conditional parameters for determining the data to be exported in the database. Taking the case where the data to be exported is data in the database of an e-commerce platform as an example, the query condition may be the inventory in the first quarter of 2020.

[0077] In an exemplary embodiment, the export parameter information may also include at least one of an export user identifier and an export time. For example, the export parameter information may include information such as query conditions, exporter, and export time of the data to be exported, which is not specifically limited in this exemplary embodiment.

[0078] After the export parameter information is acquired, illustratively, generating an export file identifier for the data to be exported includes: generating an export file identifier for the data to be exported according to the export parameter information.

[0079] Specifically, the export file identifier can be generated according to the export time in the export parameter information. For example, if a data export request is issued at 11:12:13 on September 10, 2020, the generated file identifier can be 202009111213.

[0080] After generating the export file identifier of the data to be exported, it also includes: generating a target export task according to the export file identifier and the export parameter information, writing the target export task into the export task information table, and configuring the state of the target export task to the exporting state.

[0081] For example, after the client initiates a data export request, the server receiving thread can generate the file name to be exported according to the export time in the export request, and then generate a target export task record together with the exporter, export time and other information, write the record into the export task information table, and configure the task status corresponding to the file name to the exporting status to remind the client that the export task is in progress. The export task information table can be stored in a database.

[0082] After the export file identifier is generated, in step S220, the export path of the data to be exported is obtained through an asynchronous thread, and an export file is created according to the export path and the export file identifier.

[0083] Exemplarily, before obtaining the export path of the data to be exported through the asynchronous thread, the synchronous thread may first start the asynchronous thread. After starting the asynchronous thread, the synchronous thread may send the export file identifier and the export parameter information to the asynchronous thread. After receiving the export file identifier, the asynchronous thread obtains the export path of the data to be exported, and then creates the export file according to the export path and the export file identifier. For example, the asynchronous thread may create an export file object according to the export path and the export file identifier.

[0084] Among them, the export path can be the root directory path of the backend server where the data to be exported is located, which can be set as the default export path. Of course, the export path can also be customized according to user needs, and this exemplary embodiment does not impose any special restrictions on this.

[0085] It should be noted that the export file at this time is just an empty file and no data to be exported has been written into it.

[0086] After the export file is created, in step S230, the asynchronous thread queries the data to be exported in pages according to the query conditions of the data to be exported, and writes the queried data to be exported into the export file in the order of the page query to generate a target export file.

[0087] When the amount of data is large, in order to solve the memory overflow problem caused by querying all data at once, in this exemplary embodiment, the asynchronous thread can perform paging query according to the query conditions. Then, according to the query order, the queried files to be exported are appended to the export file in sequence using the I / O stream to generate the target export file. The amount of data for each paging query can be customized according to user needs.

[0088] For example, you can preset the number of data records for each paging query to be 1000. Then the first paging queries 1000 data records at a time, and writes the data to be exported obtained by the paging query into the export file. The second paging queries 1001-2000 data records at a time, and writes the data to be exported obtained by the paging query into the export file, and so on, until all the data to be exported are sequentially queried and written into the export file in sequence, and the export is completed.

[0089] It should be noted that the query data volume of the last page is greater than or equal to 1 and less than or equal to the preset query data volume of each page. Taking the above-mentioned data record preset as 1000 for each page query as an example, the data volume of the last page query is greater than or equal to 1 and less than or equal to 1000.

[0090] Furthermore, in an exemplary embodiment, the number of asynchronous threads may be multiple. Specifically, multiple subthreads may be started in the asynchronous thread in step S220 to obtain multiple asynchronous threads. In this way, a larger amount of data may be exported, for example, more than 10 million data may be exported. At the same time, the efficiency of data export may be improved to avoid the problem of data being unable to be exported due to the interruption of the front-end and back-end network connections during a long data export process.

[0091] Exemplarily, the method of using multiple asynchronous threads to export data can be as follows: Figure 3 Reference Figure 3 The method may include steps S310 to S350:

[0092] In step S310, the amount of data to be exported and the number of preset asynchronous threads are obtained.

[0093] In an exemplary embodiment, the amount of data to be exported may be the total amount of data that needs to be queried in the database of the backend server. The number of preset asynchronous threads, that is, the number of sub-threads opened in the asynchronous thread in the above step S220, may be customized according to user needs.

[0094] Next, in step S320, a sub-data amount to be exported is allocated to each asynchronous thread according to the data amount to be exported and the number of preset asynchronous threads.

[0095] Exemplarily, the amount of data to be exported can be evenly allocated to each asynchronous thread according to the amount of data to be exported and the number of preset asynchronous threads. The amount of data to be exported can be the amount of data to be processed by the asynchronous thread. When it cannot be evenly divided, the excess amount of data can be allocated to any one of the threads, for example, the last thread.

[0096] For example, the amount of data to be exported is 10 million, and the number of preset asynchronous threads is 100, then the amount of sub-data to be exported of each asynchronous thread is 100,000.

[0097] Continue to refer Figure 3 In step S330, each asynchronous thread queries the sub-data to be exported in parallel according to the query conditions of the data to be exported and the amount of the sub-data to be exported, in pages.

[0098] For example, a thread pool can be used to allow each asynchronous thread to query the sub-data to be exported in the database in a paging query manner in parallel according to the query conditions and the amount of sub-data to be exported. In this way, the problem of memory overflow in the export of large amounts of data can be further solved, and the export of large amounts of data can be realized.

[0099] The specific implementation of the paging query of each asynchronous thread is the same as the specific implementation of the paging query of the single asynchronous thread mentioned above, and will not be repeated here.

[0100] Of course, each asynchronous thread may also directly perform data query according to actual needs without performing paging query, and this exemplary embodiment does not make any special limitation on this.

[0101] After each asynchronous thread finds the sub-data to be exported, in step S340, each asynchronous thread writes the found sub-data to be exported into the sub-export file corresponding to each asynchronous thread in the order of paging query.

[0102] Exemplarily, before writing the queried sub-data to be exported into the sub-export file corresponding to each asynchronous thread, the sub-export file may be created first. Exemplarily, the specific implementation of creating the sub-export file may be to generate the sub-export file identifier corresponding to each asynchronous thread according to the export file identifier and the thread identifier of each asynchronous thread; and to obtain the export path and the sub-export file identifier to create the sub-export file corresponding to each asynchronous thread.

[0103] The thread identifier of each asynchronous thread may include a thread number, for example, the first asynchronous thread, the second asynchronous thread, the third asynchronous thread, and so on.

[0104] For example, the export file identifier can be 202009111213 as mentioned above, then the sub-export file identifier corresponding to the first asynchronous thread can be 202009111213_001, the sub-export file identifier corresponding to the second asynchronous thread can be 202009111213_002, and so on. The sub-export file identifier corresponding to the 100th asynchronous thread can be 202009111213_100.

[0105] After generating the sub-export file identifier corresponding to each asynchronous thread, the export path and the sub-export file can be obtained to create the sub-export file corresponding to each asynchronous thread. The export path is the same as the export path in the above step S220.

[0106] After the sub-export file is created, each asynchronous thread can write the sub-data to be exported of the paging query into its corresponding sub-export file in sequence according to the export path and the sub-export file identifier.

[0107] After all asynchronous threads have completed the export operation, in step S350, the sub-export files are merged to generate a target export file.

[0108] Exemplarily, a specific implementation of step S350 may be to read each sub-export file according to the export path and each sub-export file identifier; and write each read sub-export file into the export file to generate a target export file.

[0109] Specifically, each sub-export file can be read according to the export path and the sub-export file identification order, for example, each sub-export file can be read in the order of 202009111213_001, 202009111213_002, 202009111213_003, ..., 202009111213_100, and each sub-export file can be written into the export file 202009111213 according to the reading order to generate a target export file. The target export file is a file in which the data to be exported is written.

[0110] Through the above-mentioned steps S310-step S350, through multiple asynchronous threads, the data range to be exported by each asynchronous thread is allocated according to the total amount of exported data, and each asynchronous thread exports data in parallel to its corresponding sub-export file, and finally the sub-export files are merged into one export file, thereby realizing the export of large data volumes of tens of millions.

[0111] Furthermore, after the target export file is generated, the target export file may be uploaded to a storage platform.

[0112] Exemplarily, uploading the target export file to the storage platform may include: uploading the target export file to the storage platform; receiving the uniform resource locator of the target export file sent by the storage platform; writing the uniform resource locator into the export task information table to associate the uniform resource locator with the export file identifier. The above-mentioned storage platform may be a third-party distributed storage platform.

[0113] For example, after the export is successful, the target export file can be uploaded to a third-party storage platform, and the URL (Uniform Resource Locator) sent by the third-party storage platform can be written into the export record corresponding to the target export task in the export task information table to associate the uniform resource locator and the export file identifier.

[0114] This method of uploading the target export file to a third-party storage platform can share the target export file, thereby achieving one-time export and multiple reuse. Users of other clients can directly find the uniform resource locator corresponding to the export file identifier in the export task information table after determining the export file identifier, and thus directly download the required target export file.

[0115] Exemplarily, when the target export file is uploaded successfully, the status of the target export task corresponding to the export file identifier is updated from the exporting status to the successful status; when the target export file is uploaded unsuccessfully, the status of the target export task corresponding to the export file identifier is updated from the exporting status to the failed status, and the failure identifier of the failure status is recorded, wherein the failure identifier includes an export exception or an upload exception.

[0116] By updating the status of the export task, when the upload is successful, the user can be reminded that the current export task has been completed and the exported data can be downloaded.

[0117] For example, when the target export file is successfully uploaded, a prompt message indicating that the export is complete can be sent to the client. For example, after the upload is successful, "Export task 202009111213 has been exported and can be downloaded" can be sent to the client, so that the client user can download the target export file in time according to the prompt message. At the same time, when the target export file fails to be uploaded, a prompt message indicating that the export failed can also be sent to the client, so that the client user can find the abnormality in time and reprocess the export task, thereby improving work efficiency.

[0118] In addition, when updating the status, the time of the status update can also be recorded in the export task information table.

[0119] Furthermore, after writing the uniform resource locator of the target export file in the export task information table, the method also includes: returning an export file identifier to the client, wherein the export file identifier is used to instruct the client to determine the uniform resource locator of the target export file; and in response to the client's triggering operation on the uniform resource locator, downloading the target export file on the client according to the uniform resource locator.

[0120] For example, after the export file identifier is generated, the export file identifier can be returned to the client, so that the user of the client can find the corresponding uniform resource locator according to the export file identifier and click the uniform resource locator to download the target export file.

[0121] For example, Figure 4 Another data export method in an exemplary embodiment of the present disclosure is shown, referring to Figure 4 , the method may include steps S410 to S450.

[0122] In step S410, the query condition in the export parameter information is obtained;

[0123] In step S420, the export task information table is queried to determine whether there is an export task with the same query condition as the query condition in the export task information table; if so, go to step S430-step S440; if not, go to step S450.

[0124] In step S430, a prompt message is sent to the client, where the prompt message is used to instruct the client to determine the uniform resource locator corresponding to the target export file of the export task;

[0125] In step S440, in response to a trigger operation of the client on the uniform resource locator, the target export file is downloaded at the client according to the uniform resource locator.

[0126] For example, a prompt message "The query condition of the export task 202009111213 is the same as your query condition, and you can directly download the export data corresponding to the export task" can be sent to the client. Then, the user of the client can find the export task in the export task information table displayed on the client according to the export task identifier 202009111213 in the prompt message, and then trigger the uniform resource locator in the export task, such as clicking the uniform resource locator, so that the target export file corresponding to the export task can be directly downloaded on the client.

[0127] In this way, after uploading the target export file to a third-party storage platform and associating the target export file identifier with the uniform resource locator of the target export file, through the query conditions, it is possible to automatically judge whether the target export task exists in the export task information table without the client user knowing, so as to remind the client user to download directly, thereby improving work efficiency.

[0128] In step S450, an asynchronous thread paging export operation is performed.

[0129] Exemplarily, the specific implementation of the asynchronous thread paging export operation in step S450 is the same as the above-mentioned steps S210 to S230, and will not be described in detail here.

[0130] In order to explain the above data export method more clearly and in detail. For example, Figure 5 A schematic diagram showing the interaction between the client, the backend control layer, the backend service layer, the database, and the file storage platform in an exemplary embodiment of the present disclosure is shown. The backend control layer and the backend service layer are both deployed in the backend server. Figure 5 , which may include steps S501 to S516.

[0131] In step S501 , the client 51 sends a data export request for an export task to the backend control layer 52 .

[0132] The data export request includes export parameter information, which is the same as the export parameter information in the above step S210.

[0133] In step S502 , the backend control layer 52 receives a data export request sent by the client 51 .

[0134] In step S503 , the backend control layer 52 forwards the received data export request to the backend service layer 53 .

[0135] In step S504, the backend service layer 53 receives the data export request forwarded by the backend control layer 52, and generates an export file identifier according to the export parameter information in the data export request.

[0136] In step S505, the backend service layer 53 writes the export parameter information and the export file identifier into the export task information table.

[0137] In step S506 , the backend service layer 53 returns the generated export file identifier to the client 51 .

[0138] In step S507, the backend service layer 53 starts an asynchronous thread and obtains the export path through the asynchronous thread, so that the asynchronous thread creates an export file according to the export path and the export file identifier.

[0139] In step S508, the asynchronous thread determines the paging parameters and obtains the query conditions in the export parameter information;

[0140] In step S509, the asynchronous thread initiates a data query request to the database 54 according to the query conditions in the paging parameters and the export parameter information;

[0141] In step S510, the database 54 returns the paging query data to the backend service layer 53;

[0142] In step S511, the asynchronous thread sequentially writes the data queried by paging into the created export file to generate a target export file.

[0143] In step S512 , after the data export is completed, the backend service layer 53 uploads the generated target export file to the file storage platform 55 .

[0144] In step S513 , the file storage platform 55 returns the upload result to the backend service layer 53 .

[0145] In an exemplary embodiment, the upload result includes upload failure or upload success. When the upload is successful, a message indicating the upload success is returned to the backend service layer 53, and the storage address of the target export file is sent; when the upload fails, a message indicating the upload failure is directly returned to the backend service layer 53. The storage address of the target export file may include the uniform resource locator of the target export file.

[0146] In step S514, the backend service layer 53 updates the status of the export task in the export task information table according to the received upload result. If the upload is successful, the address of the received target export file is simultaneously written into the export task information table.

[0147] In step S515 , the client 51 sends a download request for the target export file to the file storage platform 55 according to the storage address of the target export file corresponding to the export file identifier.

[0148] Exemplarily, the above-mentioned export task information table is stored in a database, wherein the database storing the export task information table and the database of the data to be exported may be the same or different, that is, the export task information table may be stored in database 54 or in other databases, and this exemplary embodiment does not impose any limitation on this.

[0149] At the same time, any export task information of any user will be recorded in the export task information table. Furthermore, an export task information table identification control can be created in the graphical user interface of the client 51. In response to the triggering operation of the export task information table identification control, the export task information table can be displayed in the graphical user interface of the client 51.

[0150] After the export task information table is displayed in the graphical user interface of the client 51, the user in the client 51 can query the export task information table to find the export task corresponding to the export file identifier. When the status of the export task is successful upload, the user can click on its corresponding uniform resource locator to send a download request for the target export file to the file storage platform 55.

[0151] Of course, the user of the client does not need to query the export task information table himself to determine whether the target export file can be downloaded. Instead, after the target export file is successfully uploaded, a prompt message indicating that the data export is completed is sent to the client to remind the user of the client to download the target export file. This exemplary embodiment does not make any special limitations on this.

[0152] In step S516, the file storage platform 55 sends the target export file to the client 51, so that the client completes the download of the target export file.

[0153] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided by the present invention are performed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk.

[0154] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0155] Figure 6 FIG. 1 is a schematic diagram showing the structure of a data export device in an exemplary embodiment of the present disclosure. Figure 6 The method may include: an export request response module 610, an export file creation module 620, and a target export file generation module 630. Wherein:

[0156] The export request response module 610 is configured to respond to the data export request of the client, obtain export parameter information, and generate an export file identifier of the data to be exported, wherein the export parameter information includes a query condition of the data to be exported;

[0157] The export file creation module 620 is configured to obtain the export path of the data to be exported through an asynchronous thread, and create an export file according to the export path and the export file identifier;

[0158] The target export file generation module 630 is configured as an asynchronous thread to query the data to be exported in pages according to the query conditions of the data to be exported, and write the queried data to be exported into the export file in the order of the page query to generate the target export file.

[0159] In an exemplary embodiment of the present disclosure, based on the above-mentioned embodiment, the export request response module 610 is further specifically configured as follows:

[0160] Generate a target export task according to the export file identifier and export parameter information;

[0161] Write the target export task into the export task information table, and configure the status of the target export task to the exporting status.

[0162] In an exemplary embodiment of the present disclosure, based on the above-mentioned embodiment, the above-mentioned export file creation module 620 is further specifically configured as follows:

[0163] Generate a sub-export file identifier corresponding to each asynchronous thread according to the export file identifier and the thread identifier of each asynchronous thread;

[0164] The export path and the sub-export file identifier are obtained to create a sub-export file corresponding to each of the asynchronous threads.

[0165] In an exemplary embodiment of the present disclosure, based on the above-mentioned embodiment, the above-mentioned target export file generation module 630 is further specifically configured as follows:

[0166] Get the amount of data to be exported and the number of preset asynchronous threads;

[0167] According to the amount of data to be exported and the number of preset asynchronous threads, allocating a sub-amount of data to be exported to each asynchronous thread;

[0168] Each of the asynchronous threads queries the sub-data to be exported in pages in parallel according to the query conditions of the data to be exported and the amount of the sub-data to be exported;

[0169] Each of the asynchronous threads writes the queried sub-data to be exported into the sub-export files corresponding to each of the asynchronous threads in the order of the paging query;

[0170] The sub-export files are merged to generate a target export file.

[0171] In an exemplary embodiment of the present disclosure, based on the above-mentioned embodiment, the data export device 600 further includes a target export file storage module, which is configured to:

[0172] Uploading the target export file to the storage platform;

[0173] Receiving the uniform resource locator of the target export file sent by the storage platform;

[0174] The uniform resource locator is written into the export task information table to associate the uniform resource locator with the export file identifier.

[0175] In an exemplary embodiment of the present disclosure, based on the above-mentioned embodiment, the above-mentioned target export file storage module is further specifically configured as follows:

[0176] When the target export file is uploaded successfully, the status of the target export task corresponding to the export file identifier is updated from the exporting status to the success status;

[0177] When the target export file fails to be uploaded, the state of the target export task corresponding to the export file identifier is updated from the exporting state to the failed state, and the failure identifier of the failed state is recorded, wherein the failure identifier includes an export exception or an upload exception.

[0178] In an exemplary embodiment of the present disclosure, based on the above-mentioned embodiment, the data export device 600 further includes a target export file download module, which is configured to:

[0179] Returning the export file identifier to the client, where the export file identifier is used to instruct the client to determine the uniform resource locator of the target export file;

[0180] In response to a triggering operation of the client on the uniform resource locator, the target export file is downloaded at the client according to the uniform resource locator.

[0181] In an exemplary embodiment of the present disclosure, based on the above-mentioned embodiment, the above-mentioned target export file download module is further specifically configured as follows:

[0182] Querying the export task information table to determine whether there is an export task in the export task information table with the same query condition as the query condition in the export parameter information;

[0183] If it exists, sending a prompt message to the client, wherein the prompt message is used to instruct the client to determine the uniform resource locator corresponding to the target export file of the export task;

[0184] In response to a triggering operation of the client on the uniform resource locator, the target export file is downloaded at the client according to the uniform resource locator.

[0185] The specific details of each unit in the above-mentioned data export device have been described in detail in the corresponding data export method, so they will not be repeated here.

[0186] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0187] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0188] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0189] In the exemplary embodiments of the present disclosure, a computer storage medium capable of implementing the above method is also provided. A program product capable of implementing the above method of this specification is stored thereon. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0190] refer to Figure 7 As shown, a program product 700 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0191] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0192] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0193] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0194] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0195] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0196] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0197] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0198] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0199] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 As shown in: Step S210, in response to the data export request of the client, the export parameter information is obtained, and an export file identifier of the data to be exported is generated, wherein the export parameter information includes the query conditions of the data to be exported; Step S220, the export path of the data to be exported is obtained through the asynchronous thread, and an export file is created according to the export path and the export file identifier; Step S230, the asynchronous thread queries the data to be exported in pages according to the query conditions of the data to be exported, and writes the queried data to be exported into the export file in the order of the paged query to generate a target export file.

[0200] For example, the processing unit 810 may also perform the following steps: Figure 2-5 The steps shown in .

[0201] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0202] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0203] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0204] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0205] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0206] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0207] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A data export method, characterized in that: include: In response to a data export request from a client, obtaining export parameter information and generating an export file identifier of the data to be exported, wherein the export parameter information includes a query condition of the data to be exported; Obtaining an export path of the data to be exported through an asynchronous thread, and creating an export file according to the export path and the export file identifier; The asynchronous thread queries the data to be exported by paging according to the query conditions of the data to be exported, and writes the queried data to be exported into the export file in the order of the paging query to generate a target export file; The asynchronous thread queries the data to be exported in pages according to the query conditions of the data to be exported, including: obtaining the amount of data to be exported and the number of preset asynchronous threads; allocating sub-amounts of data to be exported to each asynchronous thread according to the amount of data to be exported and the number of preset asynchronous threads; each asynchronous thread queries the sub-amounts of data to be exported in pages in parallel according to the query conditions of the data to be exported and the sub-amounts of data to be exported; In response to a triggering operation on an export task information table identifier control in the client, an export task information table is sent to the client to display the export task information table in a graphical user interface of the client, wherein the export task information table is used to store export file identifiers corresponding to data export requests of different clients and uniform resource locators of target export files corresponding to the export file identifiers; In response to a triggering operation on any uniform resource locator in the export task information table, a target export file corresponding to the export file identifier associated with the uniform resource locator is sent to the client, so that the client downloads the target export file.

2. The data export method according to claim 1, characterized in that: The step of writing the queried data to be exported into the export file in the order of the paging query to generate a target export file includes: Each of the asynchronous threads writes the queried sub-data to be exported into the sub-export files corresponding to each of the asynchronous threads in the order of the paging query; The sub-export files are merged to generate a target export file.

3. The data export method according to claim 2, characterized in that: Before writing the queried sub-data to be exported into the sub-export files corresponding to each of the asynchronous threads, the method further includes: Generate a sub-export file identifier corresponding to each asynchronous thread according to the export file identifier and the thread identifier of each asynchronous thread; The export path and the sub-export file identifier are obtained to create a sub-export file corresponding to each of the asynchronous threads.

4. The data export method according to claim 3, characterized in that: The merging of the sub-export files to generate a target export file includes: Read each sub-export file according to the export path and each sub-export file identifier; Each sub-export file read is written into the export file to generate a target export file.

5. The data export method according to claim 1, characterized in that: The export parameter information also includes at least one of an export user identifier and an export time; The step of generating an export file identifier for the data to be exported includes: An export file identifier for the data to be exported is generated according to the export parameter information.

6. The data export method according to claim 1, characterized in that: After generating the export file identifier of the data to be exported, the method further includes: Generate a target export task according to the export file identifier and the export parameter information; The target export task is written into the export task information table, and the state of the target export task is configured as an exporting state.

7. The data export method according to any one of claims 1 to 6, characterized in that: After generating the target export file, the method further includes: Uploading the target export file to the storage platform; Receiving the uniform resource locator of the target export file sent by the storage platform; The uniform resource locator is written into the export task information table to associate the uniform resource locator with the export file identifier.

8. The data export method according to claim 7, characterized in that: The method further comprises: When the target export file is uploaded successfully, the status of the target export task corresponding to the export file identifier is updated from the exporting status to the success status; When the target export file fails to be uploaded, the state of the target export task corresponding to the export file identifier is updated from the exporting state to the failed state, and the failure identifier of the failed state is recorded, wherein the failure identifier includes an export exception or an upload exception.

9. The data export method according to claim 7, characterized in that: The method further comprises: Returning the export file identifier to the client, where the export file identifier is used to instruct the client to determine the uniform resource locator of the target export file; In response to a triggering operation of the client on the uniform resource locator, the target export file is downloaded at the client according to the uniform resource locator.

10. The data export method according to claim 1, characterized in that: After obtaining the export parameter information, the method further includes: Querying the export task information table to determine whether there is an export task in the export task information table with the same query condition as the query condition in the export parameter information; If it exists, sending a prompt message to the client, wherein the prompt message is used to instruct the client to determine the uniform resource locator corresponding to the target export file of the export task; In response to a triggering operation of the client on the uniform resource locator, the target export file is downloaded at the client according to the uniform resource locator.

11. A data export device, characterized in that: include: An export request response module is configured to respond to a data export request from a client, obtain export parameter information, and generate an export file identifier of the data to be exported, wherein the export parameter information includes a query condition of the data to be exported; An export file creation module is configured to obtain an export path of the data to be exported through an asynchronous thread, and create an export file according to the export path and the export file identifier; The target export file generation module is configured such that the asynchronous thread queries the data to be exported by pages according to the query conditions of the data to be exported, and writes the queried data to be exported into the export file in the order of the page query to generate a target export file; The asynchronous thread queries the data to be exported in pages according to the query conditions of the data to be exported, including: obtaining the amount of data to be exported and the number of preset asynchronous threads; allocating sub-amounts of data to be exported to each asynchronous thread according to the amount of data to be exported and the number of preset asynchronous threads; each asynchronous thread queries the sub-amounts of data to be exported in pages in parallel according to the query conditions of the data to be exported and the sub-amounts of data to be exported; In response to a triggering operation on an export task information table identifier control in the client, an export task information table is sent to the client to display the export task information table in a graphical user interface of the client, wherein the export task information table is used to store export file identifiers corresponding to data export requests of different clients and uniform resource locators of target export files corresponding to the export file identifiers; In response to a triggering operation on any uniform resource locator in the export task information table, a target export file corresponding to the export file identifier associated with the uniform resource locator is sent to the client, so that the client downloads the target export file.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data export method according to any one of claims 1 to 10 is implemented.

13. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the data export method according to any one of claims 1 to 10.

Images