Data exporting method, device and system, storage medium and computer program product

By intercepting database query operations and judging that streaming queries or paging queries are used, the problem of memory overflow during data export is solved, the efficient and stable data export process is realized, and the operation process is simplified.

CN118916391APending Publication Date: 2024-11-08KE COM (BEIJING) TECHNOLOGY CO LTD

Patent Information

Application Number
CN202410955009.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing data export technologies can easily lead to memory overflow when processing big data, which in turn leads to program crashes or performance degradation, especially when the amount of data exceeds the memory limit.

Method used

By intercepting the entry method of database query operations, we can determine whether to implement stream query operations. We will use stream query or paging query based on the amount of data and concurrent tasks, transmit data in chunks to reduce memory usage, and convert the query into streaming Query statements to implement data export.

Benefits of technology

It effectively avoids memory overflow problems, improves the stability and performance of the data export process, simplifies user operations, and reduces the cumbersome steps of manually converting ordinary queries.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916391A_ABST
    Figure CN118916391A_ABST
Patent Text Reader

Abstract

The invention provides a data exporting method, device and system, a storage medium and a computer program product. The data exporting method provided by the invention comprises the following steps: acquiring a data exporting request sent by a user terminal; executing a first query operation according to the data export request; intercepting an entry method of the first query operation; obtaining a streaming query statement according to the parameters of the entry method and a preset construction rule; setting streaming query related parameters corresponding to the streaming query statement according to the data export request; executing streaming query operation according to the streaming query statement and the streaming query related parameters to obtain a streaming query result; and sending exported data corresponding to the streaming query result to the user terminal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a data export method, device, system, storage medium and computer program product. Background Art

[0002] With the rapid development of technologies such as big data and artificial intelligence, a huge amount of data has been generated on the Internet, and data has become one of the core resources in today's information age. In order to facilitate the flow of data, it is often necessary to export data from the database; for example, export part or all of the database as an Excel file.

[0003] Existing data export technologies usually require loading database data into memory for processing to achieve data export.

[0004] However, the capacity of memory is limited. If the amount of data loaded at one time is too large and exceeds the memory limit, it will cause memory overflow, which will lead to program crash or performance degradation. Summary of the invention

[0005] The present disclosure provides a data export method, device, system, storage medium and computer program product.

[0006] According to one aspect of the present disclosure, a data export method is provided, comprising:

[0007] Obtaining a data export request sent by a user terminal;

[0008] Execute a first query operation according to the data export request;

[0009] Intercepting the entry method of the first query operation;

[0010] Obtaining a streaming query statement according to the parameters of the entry method and preset construction rules;

[0011] Setting streaming query related parameters corresponding to the streaming query statement according to the data export request;

[0012] Execute a streaming query operation according to the streaming query statement and streaming query related parameters to obtain a streaming query result;

[0013] The exported data corresponding to the streaming query result is sent to the user terminal.

[0014] According to the data export method of at least one embodiment of the present disclosure, the entry method of intercepting the first query operation includes:

[0015] Determining whether to implement a streaming query operation according to the data export request;

[0016] If a streaming query operation is implemented, intercepting the entry method of the first query operation;

[0017] The streaming query operation and the first query operation use different query strategies.

[0018] According to at least one embodiment of the data export method of the present disclosure, the step of determining whether to implement a streaming query operation according to the data export request includes:

[0019] Acquire the amount of exported data and the number of exported pages according to the data export request;

[0020] Get the preset maximum number of concurrent tasks;

[0021] Determining whether the amount of exported data is greater than a preset first threshold;

[0022] If the amount of exported data is greater than the preset first threshold, comparing the number of exported pages with the maximum amount of concurrent tasks;

[0023] If the number of exported pages is greater than the maximum concurrent task amount, a streaming query operation is performed.

[0024] According to at least one embodiment of the data export method of the present disclosure, the determining whether to implement a streaming query operation according to the data export request further includes:

[0025] If the amount of exported data is not greater than the preset first threshold, obtaining a first query result corresponding to the first query operation;

[0026] The exported data corresponding to the first query result is sent to the user terminal.

[0027] According to at least one embodiment of the data export method of the present disclosure, the determining whether to implement the second query operation according to the data export request further includes:

[0028] If the number of exported pages is less than the maximum concurrent task amount, intercepting a first query statement obtained by the first query operation;

[0029] Converting the first query statement into a paging query statement according to the data export request;

[0030] Execute a paging query operation according to the paging query statement to obtain a paging query result;

[0031] The exported data corresponding to the paging query result is sent to the user terminal.

[0032] According to at least one embodiment of the data export method of the present disclosure, the step of performing a streaming query operation according to the streaming query statement and streaming query related parameters to obtain a streaming query result includes:

[0033] Acquire streaming data according to the streaming query statement and streaming query related parameters;

[0034] Determine whether the acquired streaming data reaches the export data volume corresponding to the data export request;

[0035] If not reached, executing the step of obtaining streaming data according to the streaming query statement and streaming query related parameters on subsequent data of the current streaming data;

[0036] If reached, the acquired streaming data is aggregated to obtain the streaming query result.

[0037] According to another aspect of the present disclosure, there is provided a data export device, comprising:

[0038] a memory storing a computer program; and

[0039] A processor, wherein the processor executes the computer program stored in the memory, so that the processor performs the data export method of any embodiment of the present disclosure.

[0040] According to another aspect of the present disclosure, there is provided a data export system, comprising:

[0041] At least one user terminal and a data export device according to any one embodiment of the present disclosure; the at least one user terminal is communicatively connected with the data export device to execute a data export process.

[0042] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the execution instructions are used to implement the data export method of any embodiment of the present disclosure.

[0043] According to another aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, it is used to implement at least the data export method of any one embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0045] Figure 1It is a schematic diagram of an application scenario of the data export method according to an embodiment of the present disclosure.

[0046] Figure 2 It is an interactive schematic flowchart of a data export method according to an embodiment of the present disclosure.

[0047] Figure 3 It is a flowchart of the data export method provided in Example 1 of the present disclosure.

[0048] Figure 4 yes Figure 3 A schematic diagram of a common query in the data export method shown.

[0049] Figure 5 yes Figure 3 The diagram shows the paging query in the data export method.

[0050] Figure 6 yes Figure 3 Schematic diagram of streaming query in the data export method shown.

[0051] Figure 7 yes Figure 3 A flowchart of the interception entry method in the data export method shown.

[0052] Figure 8 yes Figure 7 The process of the query mode judgment method in the interception entry method shown Figure 1 .

[0053] Fig. 9 yes Figure 7 The process of the query mode judgment method in the interception entry method shown Figure 2 .

[0054] Fig.10 yes Figure 7 The process of the query mode judgment method in the interception entry method shown Figure 3 .

[0055] Fig.11 yes Figure 3 A flowchart of a method for obtaining query results in a data export method is shown.

[0056] Fig.12 yes Figure 3 A flowchart of a method for obtaining exported data in the data export method shown.

[0057] Fig.13 It is a structural diagram of the data export device provided in Example 2 of the present disclosure. DETAILED DESCRIPTION

[0058] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0059] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0060] Data has become one of the core resources in today's information age. In order to facilitate resource circulation, it is usually necessary to load the database data into the memory for processing and export the data. However, the capacity of the memory is limited. If the amount of data to be exported is too large and exceeds the memory limit, it will cause memory overflow, which will lead to program crashes or performance degradation.

[0061] Figure 1 The application scenario of the data export method of the embodiment of the present disclosure is shown. In the application scenario, at least one user terminal 10 and a data export device 20 may be included. At least one user terminal 10 communicates with the data export device 20 through a network. When at least one user terminal 10 needs to export data, the data export process is implemented through the data export device 20. The data export device may be an electronic device, such as a server.

[0062] Figure 2 An interactive schematic flow chart of the data export method provided by an embodiment of the present disclosure is shown.

[0063] refer to Figure 2 The data export method of this embodiment includes:

[0064] Step S210: When the user needs to export data, the user inputs an export instruction on the user terminal.

[0065] Step S220: the user terminal sends a data export request corresponding to the export instruction to the data export device.

[0066] Step S230: the data export device performs a first query operation according to the data export request.

[0067] In some embodiments of the present disclosure, the operation mode corresponding to the first query operation can be preset, and the first query operation can be a normal query (also known as a basic query or a standard query) operation. The normal query operation refers to executing a preset (conventional) database query operation, usually by presetting a normal SQL statement to obtain the data to be exported. This type of query can retrieve data from the database according to the data export request and return the result to the user. Normal queries usually do not involve special processing or complex logic, but simply retrieve data from the database and return it.

[0068] Step S240: The data export device determines whether to implement a first query operation, a streaming query operation, or a paging query operation according to the data export request.

[0069] In some embodiments of the present disclosure, it is determined in step S240 that the first query operation is to be implemented, and step S250 is executed; if a paging query operation is to be implemented, step S260 is executed; if a streaming query operation is to be implemented, step S290 is executed. The query strategies used by the first query operation, the streaming query operation, and the paging query operation are different.

[0070] Step S250: the data export device obtains a first query result corresponding to the first query operation.

[0071] Step S260: the data export device intercepts the first query statement obtained by the first query operation.

[0072] Step S270: the data export device converts the first query statement into a paging query statement according to the data export request.

[0073] Step S280: the data export device executes a paging query operation according to the paging query statement to obtain a paging query result.

[0074] Step S290: the data export device intercepts the entry method of the first query operation.

[0075] In some embodiments of the present disclosure, the entry method of the first query operation refers to the starting point of triggering or starting the first query operation. The entry method may be a function, an API call, or an HTTP request. Step S290 refers to blocking the first query operation at the starting point of triggering or starting the first query operation, and the subsequent process of the first query operation is no longer executed.

[0076] Step S291, the data export device obtains the streaming query statement according to the parameters of the entry method.

[0077] In some embodiments of the present disclosure, the parameters of the entry method refer to a series of variables and values ​​passed to specify query conditions, control query behavior, or format query results when performing a data query operation. The parameters correspond to the data export request and determine how the database will perform the query operation and the results returned; the parameters of the entry method are usually used to provide the information required to perform the export operation, which may include query conditions, sorting methods, return fields, and other information.

[0078] In some embodiments of the present disclosure, the streaming query statement obtained in step S291 may be obtained by using an interface or framework that supports streaming processing.

[0079] Step S292: The data export device sets streaming query related parameters corresponding to the streaming query statement according to the data export request.

[0080] Step S293, the data export device performs a streaming query operation according to the streaming query statement and streaming query related parameters to obtain a streaming query result.

[0081] In some implementations of the present disclosure, step S293 may control the amount of data obtained from the database each time according to the streaming query statement and streaming query related parameters, thereby achieving the effect of streaming query.

[0082] Step S294: the data export device processes the streaming query result, the first query result or the paging query result to obtain an intermediate result.

[0083] Step S295: the data export device generates export data based on the intermediate results.

[0084] Step S296: the data export device sends the exported data to the user terminal.

[0085] In some embodiments of the present disclosure, the exported data in step S296 may be stored via an export file. Step S296 may directly send the export file itself or send a download link corresponding to the export file.

[0086] Combine the following Figures 3 to 12 The data export method disclosed in the present invention is described in detail.

[0087] Embodiment 1:

[0088] Figure 3 The present invention is a flowchart of a data export method according to an embodiment of the present invention.

[0089] refer to Figure 3 The data export method M300 of this embodiment includes:

[0090] Step S310: obtaining a data export request sent by a user terminal.

[0091] Step S320: Execute a first query operation according to the data export request.

[0092] Step S330, intercepting the entry method of the first query operation.

[0093] Step S340, obtaining a streaming query statement according to the parameters of the entry method and preset construction rules.

[0094] Step S350: setting streaming query related parameters corresponding to the streaming query statement according to the data export request.

[0095] Step S360: Execute a streaming query operation according to the streaming query statement and streaming query related parameters to obtain a streaming query result.

[0096] Step S370: Send the exported data corresponding to the streaming query result to the user terminal.

[0097] The data export method provided by the present disclosure realizes data export by intercepting the entry method of the first query operation and performing a streaming query operation according to the parameters of the data export request and the entry method. Since the streaming query can transmit the query results in blocks, only part of the data is loaded into the memory for processing each time, thereby reducing the memory usage. This method solves the problem in the prior art that the capacity of the memory is limited. If the amount of data loaded at one time is too large and exceeds the memory limit, it will cause memory overflow, which will cause the program to crash or performance degradation. In addition, by converting the first query into a streaming query, the method also solves the problem that users need to manually convert ordinary queries, which is cumbersome.

[0098] In some embodiments of the present disclosure, the data export request obtained through step S310 may include the export data source, the range of the exported data, the export format, the export path, etc.; the streaming query related parameters set through step S350 may include: fetch size fetchSize and structure set type resultSetType.

[0099] In some embodiments of the present disclosure, Figure 4 As shown, for ordinary queries, the data reading (referring to querying the data to be exported from the database), conversion (converting the format of the exported data) and writing (referring to persisting the converted data through the output stream to generate the exported data) processes can be executed by the main thread of the data export device. Figure 5 As shown in the figure, for paging query, the "read" and "transfer" operations can be performed by the child threads in the thread pool created by the main thread. After the "read" and "transfer" operations are completed, the "write" operation is performed by the main thread. Figure 6As shown, for streaming queries, the "read" operation can be performed through the main thread, the "transfer" operation can be performed through the child thread of the thread pool, and then the "write" operation can be performed through the main thread.

[0100] In some embodiments of the present disclosure, after the first query operation obtains the parameters of the entry method, the parameters of the entry method can be stored in a "shared variable within the thread" (such as ThreadLocal, etc.). After intercepting the entry method of the first query operation, the subsequent process of the first query operation may no longer be executed, and an empty result may be directly returned to end the first query operation. Streaming query related parameters may include: fetch size fetchSize and structure set type resultSetType; for example, setting fetchSize = Integer.MIN_VALUE, resultSetType = FORWARD_ONLY, etc. Streaming query related parameters can be stored in the MappedStatement variable.

[0101] Furthermore, if Figure 7 As shown, step S330 may include:

[0102] Step S331, determining whether to implement a streaming query operation based on the data export request.

[0103] Step S332: if a streaming query operation is implemented, intercept the entry method of the first query operation.

[0104] In some implementations of the present disclosure, the query strategy used by the streaming query operation is different from that used by the first query operation.

[0105] In some embodiments of the present disclosure, Figure 8 As shown, the specific process of determining whether to implement a streaming query operation through step S331 may include: step S3311, obtaining the amount of exported data and the number of exported pages according to a data export request; step S3312, obtaining a preset maximum concurrent task amount; step S3313, determining whether the amount of exported data is greater than a preset first threshold; step S3314, if the amount of exported data is greater than the preset first threshold, comparing the relationship between the number of exported pages and the maximum concurrent task amount; step S3315, if the number of exported pages is greater than the maximum concurrent task amount, implementing a streaming query operation.

[0106] In some embodiments of the present disclosure, when it is determined in step S3313 that the amount of exported data is not greater than the preset first threshold, Fig. 9 As shown, step S331 may also include:

[0107] Step S3316, obtaining a first query result corresponding to the first query operation.

[0108] Step S3317: Send the exported data corresponding to the first query result to the user terminal.

[0109] In some embodiments of the present disclosure, when it is determined in step S3314 that the number of exported pages is less than the maximum concurrent task amount, such as Fig.10 As shown, step S331 may also include:

[0110] Step S3318: intercepting the first query statement obtained by the first query operation.

[0111] Step S3319: convert the first query statement into a paging query statement according to the data export request.

[0112] Step S3320, perform a paging query operation according to the paging query statement to obtain a paging query result.

[0113] Step S3321, sending the exported data corresponding to the paging query result to the user terminal.

[0114] In some embodiments of the present disclosure, when the number of exported pages is equal to the maximum concurrent task amount, step S3315 may be executed or step S3318 may be executed.

[0115] In some embodiments of the present disclosure, Fig.11 As shown, step S360 may include: step S3601, acquiring streaming data according to the streaming query statement and streaming query related parameters; step S3602, determining whether the acquired streaming data reaches the export data volume corresponding to the data export request; when it is determined by step S3602 that it has not reached it, executing step S3601 on the subsequent data of the current streaming data; when it is determined by step S3602 that it has reached it, executing step S3603; step S3603, summarizing the acquired streaming data to obtain the streaming query result.

[0116] In some implementations of the present disclosure, the above judgment process may be performed by an iterator to prevent data overflow.

[0117] In some embodiments of the present disclosure, Fig.12 As shown, sending the streaming query result, the first query result or the paging query result to the user terminal through step S370, step S323 or step S327 may include: step S3701, processing the streaming query result, the first query result or the paging query result to obtain an intermediate result; step S3702, generating export data according to the intermediate result; step S3703, sending the export data to the user terminal. The processing of step S3701 includes one or more of data cleaning, format conversion, data aggregation, data completion and data verification.

[0118] In some embodiments of the present disclosure, a specific export method is adaptively selected according to a data export request, and the export method can be flexibly selected according to the usage scenario, so that the user does not need to pay attention to the specific details of the data query, which is convenient for the user.

[0119] In some embodiments of the present disclosure, processing the streaming query result, the first query result, or the paginated query result may include: performing one or more of data cleaning, format conversion, data aggregation, data completion, and data verification on the streaming query result, the first query result, or the paginated query result. When generating export data based on the intermediate results, the export data may be generated based on the intermediate results and a specified export format, and the specified export format may be Excel, JSON, XML, TXT, HTML, etc.

[0120] In some embodiments of the present disclosure, taking the number of exported pages as M, the amount of data per page as O, the maximum number of concurrent tasks (i.e., the maximum number of threads allowed in the thread pool) as N, and the total time taken to scan O pieces of data as T as an example, the total time taken for a single-threaded streaming query is Time serial =M×O×T; the total time consumed by multi-threaded paging query parallel =max(T i ), i∈[1,N], where To make the total time consumption of single-threaded streaming query greater than the total time consumption of multi-threaded paging query, N>M; therefore, the query method to be used can be determined by the relationship between the number of exported pages and the maximum number of concurrent tasks.

[0121] Combine the following Fig.13 The data export device of the present disclosure is described in detail.

[0122] Embodiment 2:

[0123] Fig.13 A schematic diagram of a data export device according to an embodiment of the present disclosure.

[0124] refer to Fig.13 The data export device 1300 of this embodiment includes:

[0125] A memory 1310 storing a computer program;

[0126] Processor 1320, the processor executes the computer program stored in the memory, so that the processor executes the data export method provided in any embodiment of the present disclosure.

[0127] The present disclosure further provides a data export system, comprising: at least one user terminal and the above-mentioned data export device; at least one user terminal is communicatively connected with the data export device to execute a data export process.

[0128] The present disclosure also provides a readable storage medium, in which execution instructions are stored, and when the execution instructions are executed by a processor, they are used to implement the above method. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit programs for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0129] The present disclosure also provides a computer program product, and the method of the present disclosure can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part. A computer-readable storage medium can be used as an implementation of a computer program product, that is, the present disclosure also provides a computer-readable storage medium, which includes a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0130] The computer program or instructions may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0131] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0135] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0136] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0137] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A data export method, It is characterized in that include: Obtaining a data export request sent by a user terminal; Execute a first query operation according to the data export request; Intercepting the entry method of the first query operation; Obtaining a streaming query statement according to the parameters of the entry method and preset construction rules; Setting streaming query related parameters corresponding to the streaming query statement according to the data export request; Execute a streaming query operation according to the streaming query statement and streaming query related parameters to obtain a streaming query result; The exported data corresponding to the streaming query result is sent to the user terminal.

2. The data export method according to claim 1, It is characterized in that The entry method for intercepting the first query operation includes: Determining whether to implement a streaming query operation according to the data export request; If a streaming query operation is implemented, intercepting the entry method of the first query operation; The streaming query operation and the first query operation use different query strategies.

3. The data export method according to claim 2, It is characterized in that The determining whether to implement a streaming query operation according to the data export request includes: Acquire the amount of exported data and the number of exported pages according to the data export request; Get the preset maximum number of concurrent tasks; Determining whether the amount of exported data is greater than a preset first threshold; If the amount of exported data is greater than the preset first threshold, comparing the number of exported pages with the maximum amount of concurrent tasks; If the number of exported pages is greater than the maximum concurrent task amount, a streaming query operation is performed.

4. The data export method according to claim 3, It is characterized in that The determining whether to implement a streaming query operation according to the data export request further includes: If the amount of exported data is not greater than the preset first threshold, obtaining a first query result corresponding to the first query operation; The exported data corresponding to the first query result is sent to the user terminal.

5. The data export method according to claim 3, It is characterized in that The determining whether to implement a second query operation according to the data export request further includes: If the number of exported pages is less than the maximum concurrent task amount, intercepting a first query statement obtained by the first query operation; Converting the first query statement into a paging query statement according to the data export request; Execute a paging query operation according to the paging query statement to obtain a paging query result; The exported data corresponding to the paging query result is sent to the user terminal.

6. The data export method according to any one of claims 1 to 5, It is characterized in that The performing a streaming query operation according to the streaming query statement and streaming query related parameters to obtain a streaming query result includes: Acquire streaming data according to the streaming query statement and streaming query related parameters; Determine whether the acquired streaming data reaches the export data volume corresponding to the data export request; If not reached, executing the step of acquiring streaming data according to the streaming query statement and streaming query related parameters on subsequent data of the current streaming data; If reached, the acquired streaming data is aggregated to obtain the streaming query result.

7. A data export device, It is characterized in that include: a memory storing a computer program; as well as A processor, wherein the processor executes the computer program stored in the memory so that the processor executes the data export method according to any one of claims 1 to 6.

8. A data export system, It is characterized in that include: at least one user terminal and the data export device as claimed in claim 7; The at least one user terminal is communicatively connected with the data export device to execute a data export process.

9. A readable storage medium, It is characterized in that The readable storage medium stores a computer program, which, when executed by a processor, is used to implement the data export method according to any one of claims 1 to 6.

10. A computer program product, It is characterized in that The computer program product comprises a computer program, which is used to implement at least the data export method according to any one of claims 1 to 6 when executed by a processor.

Citation Information

Patent Citations

  • Mass data quick query method based on polymorphic composition

    CN110489445A

  • Database multi-table association query speed optimization method based on Elasticsearch

    CN114385680A

  • Data processing method and device, electronic equipment and computer storage medium

    CN114969098A

  • Data report generation method and device, computer equipment and storage medium

    CN115357628A

  • Data paging query method and device, electronic equipment and storage medium

    CN115576977A

Cited By

  • Educational administration system-oriented large-scale data efficient exporting method based on streaming processing engine

    CN121166793A