Low-memory-based asynchronous stream data export implementation method, device and equipment

Abstract

The application discloses a low-memory-based asynchronous stream data export implementation method, device and equipment, relates to the technical field of data processing, and comprises the following steps: constructing API services for realizing data export application management and data template configuration and query, and constructing Task services for realizing data export task processing; acquiring a data template based on the API services, and submitting screening fields and table header fields corresponding to to-be-exported data; acquiring a database connection through JDBC based on the Task services and in combination with the submitted screening fields and table header fields, and generating a ResultSet result set; and performing line-by-line reading and streaming writing on data of the ResultSet result set through a ResultSet cursor mechanism, so that the to-be-exported data is exported. The application can significantly improve the scalability and running efficiency of a data export service.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method, apparatus, and device for implementing asynchronous stream data export based on low memory. Background Technology

[0002] Once a product is launched and enters the operational phase, it often needs to provide data analysis and report export functions for multiple user groups. Existing data export typically uses a synchronous query and export mode: after a user completes a data query on the interface and clicks the export button, the system immediately performs data reading and file generation. When dealing with large amounts of data, users must wait for a long time, unable to use other functions, resulting in a poor user experience.

[0003] Meanwhile, as the product becomes more widespread across different platforms (such as web and mobile apps), the export function must be available on all platforms. However, due to differences in filtering conditions, access control, and header fields across platforms, it is often necessary to develop separate export interfaces and query logic for each platform. When fields are updated or export requirements change, the service code for each platform needs to be modified and redeployed, resulting in redundant development and a significant increase in maintenance costs.

[0004] On the other hand, traditional paginated query export methods require multiple database accesses and caching of the entire page of data in memory before writing it to a file. This increases the database load due to the multiple database accesses. When the data volume is large, this method not only increases the database query pressure but also leads to excessive memory usage and frequent garbage collection (GC) due to the large amount of data accumulating in memory. This affects system stability and forces the server to rely on higher-configuration hardware resources, increasing the overall cost.

[0005] It is evident that the existing methods have shortcomings in terms of user export experience, cross-platform reusability, and efficiency of memory and database resources. Summary of the Invention

[0006] This application provides a method, apparatus, and device for implementing asynchronous streaming data export based on low memory, which can significantly improve the scalability and operating efficiency of data export services.

[0007] In a first aspect, embodiments of this application provide a method for exporting asynchronous stream data based on low memory, the method comprising:

[0008] Build an API service for managing data export applications and configuring and querying data templates, and a Task service for processing data export tasks;

[0009] Obtain the data template based on the API service, and submit the corresponding filter fields and header fields of the data to be exported;

[0010] Based on the Task service and combined with the submitted filter fields and header fields, a database connection is obtained via JDBC and a ResultSet result set is generated;

[0011] The ResultSet cursor mechanism is used to read and stream data from the ResultSet result set, thereby enabling the export of the data to be exported.

[0012] In conjunction with the first aspect, in one implementation method,

[0013] The data template consists of filter fields and header fields;

[0014] The filter fields include the filter field name, whether it must exist, the filter field type, and the filter field description;

[0015] The header fields include header field category, header field name, header field description, and access control role;

[0016] The API services include a data module list query API, a data template details query API, an export application API, an export application list query API, and a file download API.

[0017] In conjunction with the first aspect, in one implementation, the step of obtaining a data template based on the API service and submitting the corresponding filter fields and header fields for the data to be exported specifically includes:

[0018] Obtain the complete structure of the data template by querying the data template details via the API.

[0019] Based on the export request API, submit the corresponding filter fields and header fields of the data to be exported so that the final exported data contains only the requested content.

[0020] In conjunction with the first aspect, in one implementation, the step of obtaining a database connection via JDBC and generating a ResultSet based on the Task service and in conjunction with the submitted filter fields and header fields specifically includes:

[0021] Based on the Task service, SQL is created according to the submitted filter fields and header fields. The SELECT part of the SQL is the submitted header fields, and the WHERE part is the filter fields.

[0022] A database connection is obtained via JDBC, and a DbConnectionBean is created, which includes a Connection object and a ResultSet.

[0023] In conjunction with the first aspect, in one implementation, obtaining a database connection via JDBC and creating a DbConnectionBean specifically includes:

[0024] Connect to the database via JDBC and obtain a Connection object by calling the ConnectiongetConnection method in JDBC;

[0025] The Connection object is invoked, and the SQL is pre-compiled using the preparedStatement method to obtain a PreparedStatement object;

[0026] The PreparedStatement object's executeQuery method is invoked to perform a data query operation, resulting in a ResultSet, which in turn creates the DbConnectionBean.

[0027] In conjunction with the first aspect, in one implementation, the step of reading and streaming data from the ResultSet result set using a ResultSet cursor mechanism to export the data to be exported specifically includes:

[0028] The ResultSet.next function reads data from the ResultSet result set line by line and writes the read data to the target file based on a preset batch write strategy.

[0029] The final export file is generated based on the target file, and the download address is returned, thus realizing the export of the data to be exported.

[0030] In conjunction with the first aspect, in one implementation, the step of reading data line by line from the ResultSet result set using the ResultSet.next function and writing the read data to the target file based on a preset batch write strategy specifically includes:

[0031] Retrieve individual data entries sequentially from the ResultSet and add them to memory. After each data entry is retrieved, a TradeSerialTableTemplate object is created and field values ​​are set. The bean is added to the allData list, and the allData counter is incremented.

[0032] Determine if the current number of records written to memory has reached the maximum number of records per batch:

[0033] If so, write the data currently added to memory to the target file and clear the allData list. At the same time, when the amount of data written to the current target file reaches the maximum amount of data, close the current target file and increment the target file counter, generate the path to the new target file and create a new target file write stream to write data to the new target file.

[0034] If not, continue to retrieve individual data entries from the ResultSet and add them to memory sequentially.

[0035] In conjunction with the first aspect, in one implementation, when the total amount of data written to the target file reaches a warning threshold:

[0036] Send an alert, write the remaining data in the ResultSet to the final summary row, and close the target file write stream.

[0037] Secondly, embodiments of this application provide a low-memory asynchronous stream data export implementation device, the low-memory asynchronous stream data export implementation device comprising:

[0038] The template configuration module is used to build API services for managing data export applications and configuring and querying data templates, as well as Task services for processing data export tasks.

[0039] The export execution module is used to obtain a data template based on the API service, submit the corresponding filter fields and header fields of the data to be exported, and obtain a database connection through JDBC and generate a ResultSet based on the Task service and in combination with the submitted filter fields and header fields.

[0040] The data reading module is used to read and stream data from the ResultSet result set line by line using the ResultSet cursor mechanism, thereby exporting the data to be exported.

[0041] Thirdly, embodiments of this application provide a low-memory-based asynchronous stream data export implementation device. The low-memory-based asynchronous stream data export implementation device includes a processor, a memory, and a low-memory-based asynchronous stream data export implementation program stored in the memory and executable by the processor. When the low-memory-based asynchronous stream data export implementation program is executed by the processor, it implements the steps of the low-memory-based asynchronous stream data export implementation method described above.

[0042] The beneficial effects of the technical solutions provided in this application include:

[0043] (1) By configuring and abstracting the data template, the filtering conditions and header fields required for export are uniformly defined, realizing dynamic combination in multi-terminal scenarios, enabling different business terminals to flexibly combine and export content on the same data basis, thereby achieving the effect of one-time development and multi-terminal reuse, effectively reducing R&D and maintenance costs.

[0044] (2) By using JDBC+ResultSet cursor streaming to read data row by row, large data volume export can be completed with only one database connection. This avoids multiple database accesses and full-page memory caching caused by traditional pagination query methods. It can also flexibly control the data scale of a single processing, thereby reducing the number of database queries, reducing memory usage, avoiding GC pressure, and still running stably under limited hardware resources. This reduces the dependence on high-configuration servers and overall costs, and significantly improves the scalability and operating efficiency of the data export service. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the low-memory asynchronous stream data export implementation method of this application;

[0046] Figure 2 A flowchart illustrating the process of reading and writing exported data;

[0047] Figure 3 This is a schematic diagram illustrating the practical application of the asynchronous stream data export implementation method based on low memory in this application;

[0048] Figure 4 This is a schematic diagram of the functional modules of the device for implementing asynchronous stream data export based on low memory in this application;

[0049] Figure 5 This is a schematic diagram of the hardware structure of the device for implementing asynchronous streaming data export based on low memory, as described in this application. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0052] Firstly, this application provides a method for exporting asynchronous streaming data based on low memory. Specifically, it provides a data export scheme that supports templated configuration and low-memory streaming processing. By configuring and abstracting data templates, the filtering conditions and header fields required for export are uniformly defined, enabling dynamic combination in multi-terminal scenarios. This achieves the effect of one-time development and multi-terminal reuse, effectively reducing R&D and maintenance costs. Furthermore, by using JDBC+ResultSet cursor streaming, it enables row-by-row data processing. Only one database connection is needed to complete the export of large amounts of data, and the scale of data processed in a single transaction can be flexibly controlled, thereby reducing the number of database queries, reducing memory usage, avoiding GC pressure, and ensuring stable operation even under limited hardware resources. This further reduces dependence on high-configuration servers and overall costs.

[0053] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the implementation method of the asynchronous stream data export based on low memory in this application. Figure 1 As shown, the implementation methods for asynchronous stream data export based on low memory include:

[0054] S1: Build an API service for managing data export applications and configuring and querying data templates, and a Task service for processing data export tasks;

[0055] Specifically, the low-memory asynchronous streaming data export implementation method of this application consists of two parts in its overall architecture: an API (Application Programming Interface) service and a Task service (a mechanism for handling asynchronous tasks). The API service is responsible for receiving and managing data export requests, and provides data template configuration and query capabilities, realizing data templateization and multi-terminal reuse mechanisms. The Task service is responsible for the actual data export task processing, including database access, streaming data reading and export file generation, implemented based on a row-by-row streaming data export mechanism using JDBC+ResultSet.

[0056] S2: Obtain the data template based on the API service, and submit the corresponding filter fields and header fields of the data to be exported;

[0057] S3: Based on the Task service and combined with the submitted filter fields and header fields, it obtains a database connection through JDBC (Java Database Connectivity) and generates a ResultSet (the core data structure in Java JDBC used to store database query results).

[0058] S4: The ResultSet result set is read row by row and written in a streaming manner through the ResultSet cursor mechanism to export the data to be exported.

[0059] In this application, the data template structure includes filter fields and header fields. Filter fields include filter field name, whether it is required, filter field type, and filter field description. Header fields include header field category, header field name, header field description, and access control role. API services include data module list query API, data template details query API, export application API, export application list query API, and file download API.

[0060] Specifically, this application addresses the current deficiency of repeatedly developing data export interfaces for different business platforms. This deficiency stems from several factors: different filtering parameters, resulting in varying filter options displayed to users on different interfaces (e.g., web pages support far more filtering options than mobile apps); and different header fields, requiring different header fields for exported Excel files across different platforms. To resolve this deficiency, this application introduces a unified data template mechanism, enabling one-time development and multi-platform reuse.

[0061] The supported filter field parameters and header field parameters are abstracted into a templated structure and persistently stored in the database. Filter fields include the filter field name (fieldName), whether it must exist (mustBeEnum), the filter field type (fieldTypeEnum), and the filter field description (fieldDesc). Filter field types include text, numeric, arrays, etc., and filter field descriptions include merchant ID, etc. Header fields include the header field category (tradeTagEnum), header field name (fieldName), header field description (fieldDesc), and access control role (roleType). Header field categories include terminal information, merchant information, order payment information, etc. The header field name is used to display in the exported file, such as merchant ID, etc. The access control role is in array form, used to specify which roles can export this field.

[0062] Furthermore, for API services, i.e. API interface support, interface capabilities are provided around the data template mechanism to facilitate multi-terminal calls, including: data module list query API, data template details query API, export application API, export application list query API, and file download API.

[0063] Furthermore, in one embodiment, obtaining a data template based on the API service and submitting the corresponding filter fields and header fields for the data to be exported specifically includes:

[0064] S201: Obtain the complete structure of the data template through the data template details query API;

[0065] S202: Submit the filter fields and header fields of the data to be exported based on the export request API, so that the final exported data contains only the requested content.

[0066] Specifically, in practical applications, clients can obtain the complete parameter definition of a data template through the data template details query API, and dynamically pass in the required filter fields and header fields when submitting the export request API, so that the final exported data only contains the requested content.

[0067] Furthermore, in one embodiment, based on the Task service and in conjunction with the submitted filter fields and header fields, a database connection is obtained via JDBC and a ResultSet is generated, specifically including:

[0068] S301: Based on the Task service, create SQL (Structured Query Language) based on the submitted filter fields and table header fields. The SELECT part of the SQL is the submitted table header fields, and the WHERE part is the filter fields. SELECT is the core statement in SQL, used to retrieve data from the database table and store the results in the result set. WHERE is a key instruction in SQL, used to set data filtering conditions to limit the range of query results.

[0069] S302: Obtain a database connection via JDBC and create a DbConnectionBean (database connection), which includes a Connection object (a programming interface representing a unique session established with the data source) and a ResultSet.

[0070] Specifically, in the data export processing stage, this application uses a JDBC + ResultSet cursor mechanism to achieve row-by-row reading and streaming writing, effectively solving the shortcomings of the current paginated query method. First, SQL is dynamically created based on the submitted filter fields and table header fields. Then, a database connection is obtained via JDBC, and a DbConnectionBean is created. In this application, the database is the database used to store data for application development, commonly a MySQL-based database.

[0071] Furthermore, in one embodiment, obtaining a database connection and creating a DbConnectionBean via JDBC specifically includes:

[0072] S3021: Connect to the database via JDBC and call the ConnectiongetConnection method in JDBC to obtain a Connection object;

[0073] The ConnectiongetConnection method is a commonly used method in JDBC for establishing database connections, primarily used in SQL Server connection scenarios.

[0074] S3022: Call the Connection object and use the preparedStatement method to execute the SQL to pre-compile and obtain a PreparedStatement object;

[0075] preparedStatement is an interface in JDBC used to execute prepared SQL statements;

[0076] S3022: The `executeQuery` method of the `PreparedStatement` object is called to perform a data query operation, obtaining a `ResultSet` result set, thereby creating the `DbConnectionBean`. `executeQuery` is a JDBC method in Java for executing SQL queries.

[0077] Furthermore, in one embodiment, the ResultSet result set is read row by row and written in a streaming manner using a ResultSet cursor mechanism to export the data to be exported, specifically including:

[0078] S401: The ResultSet result set is read line by line using the ResultSet.next function, and the read data is written to the target file based on a preset batch write strategy; the ResultSet.next function is used to move the cursor of the ResultSet to the next line and return whether there is valid data in the current line; in one possible implementation, the target file is an Excel file;

[0079] S402: Generates the final export file based on the target file and returns the download address, thus enabling the export of the data to be exported.

[0080] Specifically, the ResultSet.next function is used to read data line by line, instead of fetching a large batch of data at once. Then, based on a preset batch write strategy, the read data is written to the target file. That is, the number of target files written in a single batch is flexibly adjusted according to the server configuration. High-configuration servers can increase the batch write and improve the export speed, while low-configuration servers can reduce the batch processing volume, reduce memory consumption, and avoid GC pressure. Afterwards, the final export file is generated based on the target file and the download address is returned, realizing the export of the data to be exported. The data reading and writing mechanism of this application significantly reduces the number of database accesses and memory consumption, enabling stable and efficient completion of large-scale data export tasks even in environments with limited hardware resources.

[0081] It should be further explained that, regarding the use of the ResultSet.next function to read data row by row in this application, a database connection is first obtained via JDBC and a DbConnectionBean is created, resulting in a ResultSet object. The ResultSet contains the long-lived database query connection, from which rows can be continuously retrieved. The difference between using the ResultSet.next function and traditional direct SQL queries is that if a single data query needs to return 10,000 rows, the traditional query method would return 10 rows at a time. To store 10,000 data entries, the program would need to use memory to store them all at once. However, by using the ResultSet.next function, each call to the next method in the ResultSet returns one data entry from the database. By continuously calling the next method to retrieve data and storing it in memory, if memory is limited, you can accumulate 100 data entries, write them to Excel, clear the 100 data entries in memory, and continue calling the next method to retrieve data. Repeat this process until all 10,000 data entries are written to Excel. In this way, the program only needs to store a peak of 1,000 data entries.

[0082] Furthermore, in one embodiment, the data in the ResultSet result set is read line by line using the ResultSet.next function, and the read data is written to the target file based on a preset batch write strategy. Specifically, this includes:

[0083] a1: Retrieves individual data entries sequentially from the ResultSet and adds them to memory. After each data entry is retrieved, a TradeSerialTableTemplate object is created and field values ​​are set. The bean is added to the allData list, and the allData counter is incremented.

[0084] The TradeSerialTableTemplate object represents one of the template objects for data export. It can be understood as the header fields that need to be displayed in the exported table header. This object is the header field for the exported transaction log, storing all the header fields, such as order number, merchant number, merchant name, order time, transaction amount, etc. Each field of the TradeSerialTableTemplate object is assigned a specific value, which is a bean obtained from the database query. For example, the order number field is set to 0001, and the amount is set to 100 yuan, etc. It can also be understood as a bean that is a TradeSerialTableTemplate object populated with the values ​​retrieved from the database, or it can be understood as a row of data exported from Excel.

[0085] a2: Determine if the current number of data entries added to memory has reached the maximum number of entries that can be written in a single batch.

[0086] If so, write the data currently added to memory to the target file and clear the allData list. At the same time, when the amount of data written to the current target file reaches the maximum amount of data, close the current target file and increment the target file counter, generate the path to the new target file and create a new target file write stream to write data to the new target file.

[0087] If not, continue to retrieve individual data entries from the ResultSet and add them to memory sequentially.

[0088] Specifically, in combination Figure 2 The above process will be explained. Figure 2 In this code, Controller represents the controller, ExcelWriter is used to implement read and write operations on Excel files, and Util represents a utility class or general function module. Similarly, the target file can be an Excel file. First, a single data record is retrieved from the ResultSet and added to memory. Then, a TradeSerialTableTemplate object is created and its field value (getSpecifiedField) is set. The bean is added to the allData list, and the allData counter is incremented. allData is used to check whether all elements in the list (or other iterable object) meet specific conditions. Then, it is determined whether the data currently added to memory has reached the maximum number of records to be written in a single batch (e.g., 1000 records). If so, the data currently added to memory is written to the target file, i.e., the Excel file, and the allData list and the data in memory are cleared. If not, the process continues to retrieve single data records from the ResultSet and add them to memory sequentially.

[0089] At the same time, the system checks the amount of data already written to a single Excel file to see if it has reached the maximum number of rows in the single Excel file (e.g., 500,000 rows). If not, no action is taken. If so, writing to the current Excel file is stopped, the Excel file counter is incremented, a new Excel file path is generated, a new Excel file write stream is created, and data is written to the new Excel file.

[0090] Furthermore, when the total amount of data written to the target file reaches the warning threshold: a warning is pushed out, the remaining data in the ResultSet is written to the final summary row, and the target file write stream is closed. That is, when the total amount of data written during the data export process reaches the warning threshold (millions of records), a warning is sent, then the remaining data is processed, written to the final summary row, and the Excel file write stream is closed.

[0091] The following combination Figure 3 This paper describes the practical application of the low-memory asynchronous stream data export implementation method of this application.

[0092] In practical applications, it includes an API system, a Task system, and a database. The API system is used to implement API services, and the Task system is used to implement Task services.

[0093] First, in the API system, users submit data export requests, query data template configurations and receive a response. Then, the Task system executes the data export task. Instead of querying the database initially, it obtains the necessary query conditions through API parameters. Next, it queries the test te5 database. If there are cross-month time periods, they need to be queried separately and written to. For example, for a monthly table, the requirement for generating a CSV file is to query 5000 records and write 5000 records. The current query conditions are: Merchant ID, Start Time, End Time, Payment Channel, Payment Method, and Payment Status. The current CSV header includes: Order Number, Terminal Transaction Number, Merchant Name, Merchant ID, Transaction Store Name, Payment Channel, Payment Method, Payment Type, Order Amount, Actual Payment Amount, Settlement Time, and Payment Status (Payment Successful, Refund Successful). Finally, the data export request form includes Merchant ID, Start Time, End Time, Payment Method, Payment Type, Status (Pending Generation, Generating, Completed), and Download Link.

[0094] Furthermore, it should be noted that, in addition to reading row by row using JDBC+ResultSet cursors to reduce database pressure, this application can also perform parallel queries and exports of large datasets based on big data distributed computing frameworks, such as Spark and Flink components.

[0095] This application primarily addresses the problems of difficult, costly, and poor user experience in data export. Currently, although the exported data is essentially the same across different platforms (merchant and user), differences in filtering conditions and header fields necessitate writing separate interfaces and query logic, leading to the repetition of the same logic and cumbersome maintenance. This application solves this problem by designing a data export template mechanism. Filtering conditions and header fields are abstracted into templates and stored in the database. Different platforms only need to configure these templates to dynamically select and combine the content to be exported according to their needs, achieving one-time development and multi-platform reuse.

[0096] On the other hand, this application also solves the export performance problem. Currently, paginated queries are commonly used, requiring multiple database queries, each time loading the entire page of data into memory before writing it to a file. This can easily lead to memory exhaustion or frequent garbage collection (GC) when dealing with large amounts of data. Therefore, this application uses a JDBC ResultSet cursor, combined with `next` to read row by row. Data is read and written to the file simultaneously without large memory caching. This allows the data export operation to be completed with only one database connection. Furthermore, the number of rows written each time can be flexibly set according to server performance, avoiding excessive memory consumption. This mechanism enables stable data export operations even on low-spec machines, without requiring additional hardware upgrades.

[0097] Furthermore, this application separates the API interface and export processing into two services. The API is only responsible for receiving requests and template queries, while the export task is handled by a dedicated Task service, avoiding the impact of disk read / write and large-scale data processing on normal business requests during export. In summary, this application achieves one-time development and multi-platform reuse through a template mechanism, reducing redundant development and maintenance costs; and through a streaming export mechanism, it uses JDBC cursors for row-by-row read / write, reducing the number of database queries and memory consumption, and avoiding GC / OOM.

[0098] Secondly, embodiments of this application also provide an asynchronous streaming data export implementation device based on low memory.

[0099] In one embodiment, reference is made to Figure 4 , Figure 4 This is a schematic diagram of the functional modules of the device for implementing asynchronous stream data export based on low memory, as described in this application. Figure 4 As shown, the low-memory asynchronous stream data export implementation device includes: a template configuration module, an export execution module, and a data reading module.

[0100] The template configuration module is used to build an API service for managing data export applications and configuring and querying data templates, as well as a Task service for processing data export tasks. The export execution module is used to obtain data templates based on the API service, submit the corresponding filter fields and header fields of the data to be exported, and obtain a database connection via JDBC and generate a ResultSet based on the Task service and the submitted filter fields and header fields. The data reading module is used to read and stream data from the ResultSet through the ResultSet cursor mechanism to export the data to be exported.

[0101] By receiving export requests, invoking the filter conditions and header fields defined in the data template, and then using a JDBC cursor to read data row by row and write it to the export file, a streaming export is completed under a single database connection. Furthermore, the data template supports multi-platform differentiated configuration and dynamic combination, and the export file supports multiple formats such as Excel, CSV, and JSON. The number of rows processed in batches can be set during cursor reading to control memory usage. This application also supports dynamic updates of the data template when export fields are expanded without requiring code modifications across multiple platforms.

[0102] Thirdly, embodiments of this application provide a low-memory asynchronous stream data export implementation device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0103] Reference Figure 5 , Figure 5 This is a schematic diagram of the hardware structure of the low-memory asynchronous stream data export implementation device involved in the embodiments of this application. In the embodiments of this application, the low-memory asynchronous stream data export implementation device may include a processor, a memory, a communication interface, and a communication bus.

[0104] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0105] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces. These interfaces enable low-memory asynchronous streaming data export for interconnecting internal devices within the device, and also enable low-memory asynchronous streaming data export for interconnecting the device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0106] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0107] The processor can be a general-purpose processor, which can call a low-memory-based asynchronous stream data export implementation program stored in memory and execute the low-memory-based asynchronous stream data export implementation method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the low-memory-based asynchronous stream data export implementation program is called can be referred to in the various embodiments of the low-memory-based asynchronous stream data export implementation method of this application, and will not be repeated here.

[0108] Those skilled in the art will understand that Figure 5 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0109] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0110] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0111] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0112] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0114] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A low memory based asynchronous stream data export implementation method, characterized in that, The low-memory-based asynchronous stream data export implementation method comprises: API services for implementing data export application management and data template configuration and query are constructed, and Task services for implementing data export task processing are constructed; The data template is obtained based on the API services, and the filtering fields and the table header fields corresponding to the data to be exported are submitted; Based on the Task services and in combination with the submitted filtering fields and table header fields, a database connection is obtained through JDBC and a ResultSet result set is generated; Through the ResultSet cursor mechanism, the data in the ResultSet result set is read row by row and written in a stream, and the data to be exported is exported; The data in the ResultSet result set is read row by row through the ResultSet.next function, and the read data is written into a target file based on a preset batch writing strategy; A final export file is generated based on the target file, and a download address is returned, and the data to be exported is exported; The data in the ResultSet result set is read row by row through the ResultSet.next function, and the read data is written into a target file based on a preset batch writing strategy; Single data is obtained from the ResultSet result set in sequence and added to the memory, and a TradeSerialTableTemplate object is created and the field value is set after each time single data is obtained, the bean is added to the allData list, and the allData counter is increased; It is judged whether the data currently added to the memory reaches the maximum writing number of a single batch: If yes, the data currently added to the memory is written into the target file, and the allData list is emptied, and when the written data in the current target file reaches the maximum data amount, the current target file is closed and the target file counter is increased, the path of the new target file is generated and the new target file writing stream is created, and the data writing of the new target file is performed; If not, single data is obtained from the ResultSet result set in sequence and added to the memory; When the total data amount written into the target file reaches a warning threshold: A warning is pushed, and the remaining data in the ResultSet result set is written into a final summary row, and the target file writing stream is closed.

2. The low-memory-based asynchronous stream data export implementation method according to claim 1, wherein: The structure of the data template comprises filtering fields and table header fields; The filtering fields comprise filtering field names, whether they must exist, filtering field types, and filtering field descriptions; The table header fields comprise table header field categories, table header field names, table header field descriptions, and permission control roles; ​ The API service includes a data module list query API, a data template detail query API, an export application API, an export application list query API, and a file download API.

3. The low memory based asynchronous stream data export implementation method of claim 2, wherein, The data template is obtained based on the API service, and the filtering field and the header field corresponding to the data to be exported are submitted, specifically including: The complete structure of the data template is obtained through the data template detail query API; The filtering field and the header field corresponding to the data to be exported are submitted based on the export application API, so that the final exported data only contains the requested content.

4. The low memory based asynchronous stream data export implementation method of claim 1, wherein, The database connection is obtained through JDBC based on the Task service and in combination with the submitted filtering field and header field, and a ResultSet result set is generated, specifically including: Based on the Task service, an SQL is created according to the submitted filtering field and header field, wherein the SELECT part of the SQL is the submitted header field, and the WHERE part is the filtering field; The database connection is obtained through JDBC and a DbConnectionBean is created, wherein the DbConnectionBean includes a Connection object and a ResultSet result set.

5. The low memory based asynchronous stream data export implementation method of claim 4, wherein, The database connection is obtained through JDBC and the DbConnectionBean is created, specifically including: The database is linked through JDBC, and the Connection object is obtained by calling the Connection getConnection method in JDBC; The SQL is executed through the preparedStatement method to obtain a PreparedStatement object; The executeQuery of the PreparedStatement object is called to perform a data query operation to obtain a ResultSet result set, thereby realizing the creation of the DbConnectionBean.

6. A low memory based asynchronous stream data export implementation apparatus, characterized by, The low-memory-based asynchronous stream data export implementation device includes: A template configuration module is configured to build API services for implementing data export application management and data template configuration and query, and Task services for implementing data export task processing; An export execution module is configured to obtain a data template based on the API service, submit a filtering field and a header field corresponding to data to be exported, and obtain a database connection through JDBC based on the Task service and in combination with the submitted filtering field and header field, and generate a ResultSet result set; A data reading module is configured to read and stream data row by row from the ResultSet result set through a ResultSet cursor mechanism to realize the export of the data to be exported; The data reading module is configured to read and stream data row by row from the ResultSet result set through a ResultSet cursor mechanism to realize the export of the data to be exported, specifically including: Data in the ResultSet result set is read row by row through the ResultSet.next function, and the read data is written into the target file based on a preset batch writing strategy; A final export file is generated based on the target file, and a download address is returned, thereby realizing export of the data to be exported; The step of reading data in the ResultSet result set row by row through the ResultSet.next function and writing the read data into the target file based on a preset batch writing strategy specifically includes the following steps. Single data is sequentially obtained from the ResultSet result set and added to the memory, and a TradeSerialTableTemplate object is created and field values are set after each time single data is obtained, the bean is added to the allData list, and the allData counter is increased. It is judged whether the data currently added to the memory reaches the maximum writing number of a single batch: If yes, the data currently added to the memory is written into the target file, and the allData list is emptied, at the same time, when the written data in the current target file reaches the maximum data amount, the current target file is closed and the target file counter is increased, the path of a new target file is generated and a new target file writing stream is created, and data writing of the new target file is performed; If no, single data is sequentially obtained from the ResultSet result set and added to the memory. When the total data amount written into the target file reaches a warning threshold: A warning is pushed, and the remaining data in the ResultSet result set is written into a final summary row, and the target file writing stream is closed.

7. A low memory based asynchronous stream data export implementation device, characterized by, The low-memory-based asynchronous stream data export implementation device includes a processor, a memory, and a low-memory-based asynchronous stream data export implementation program stored on the memory and executable by the processor, wherein when the low-memory-based asynchronous stream data export implementation program is executed by the processor, the steps of the low-memory-based asynchronous stream data export implementation method according to any one of claims 1 to 5 are implemented.

Images