A large file export optimization method and system based on real-time memory monitoring

Abstract

The application relates to the technical field of data processing, and discloses a large file export optimization method and system based on real-time memory monitoring. The method comprises the following steps: monitoring a memory state in real time and calculating a memory growth rate; taking the growth rate as an independent risk dimension, and giving a warning when the rate exceeds a threshold value and the memory usage rate does not reach a threshold value; performing adjustment in response to the warning, including dynamically updating the throughput control of the data shard size, and triggering the degradation mode to reduce the export field to the data precision control of the core field; and attaching an instruction page to the export file during degradation. Through trend prediction and dynamic control, the application reduces the risk of memory overflow and guarantees core data export.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for optimizing the export of large files based on real-time memory monitoring. Background Technology

[0002] In business systems, exporting large files is a frequent requirement, often involving hundreds of thousands or even millions of data entries. Currently, the mainstream solution, represented by EasyExcel, uses fixed-size data chunks for batch reading and writing, avoiding loading all data into memory at once. However, this solution has several limitations in practical applications: the chunk size is fixed and cannot be dynamically adjusted based on the real-time memory status of the computer; existing solutions oversimplify memory risk assessment, typically relying solely on static indicators such as memory usage for anomaly detection, triggering processing when memory usage reaches a dangerous threshold. This mechanism fails to detect dynamic changes in memory pressure, failing to identify hidden risks when memory usage is not yet exceeded but the growth rate is too rapid; furthermore, it lacks a mechanism to confirm the persistence of risks, as instantaneous fluctuations may lead to false triggers while persistent risks may be missed; additionally, existing solutions lack field-level degradation mechanisms, leaving only full export or complete failure as options when memory resources are extremely limited, making it impossible to trace the range of successfully exported data and the reasons for missing data. Maintenance personnel can only perform emergency processing by terminating the process, resulting in the loss of written data. Currently, no effective solution exists to address these issues. Summary of the Invention

[0003] In view of this, the present invention provides a method, system, device, computer equipment, computer-readable storage medium and computer program product for optimizing large file export based on real-time memory monitoring, in order to solve the technical problems of high memory overflow risk and easy loss of core data during the export of large files due to the inability of fixed sharding strategies to adapt to dynamic resource changes, lack of risk prediction capabilities, lack of degradation mechanisms and insufficient operational flexibility.

[0004] In a first aspect, the present invention provides a method for optimizing the export of large files based on real-time memory monitoring, applied to a computer device, wherein the method is executed by the computer device, and includes: The system monitors physical memory usage and exported process heap memory usage in real time at preset intervals, sets a first preset threshold for physical memory usage, a second preset threshold for memory growth rate, a first count threshold for memory stress count, and a second count threshold for growth acceleration count, and calculates the memory growth rate. In response to the physical memory usage rate reaching a first preset threshold, an accumulated memory stress count is performed; In response to the memory growth rate exceeding the second preset threshold and the physical memory utilization rate not reaching the first preset threshold, the cumulative growth acceleration count is set. When the memory stress count reaches the first count threshold or the growth acceleration count reaches the second count threshold, a risk warning signal is generated. In response to the aforementioned risk warning signal, the following adjustment actions will be performed: Throughput control dynamically calculates and updates data fragment size based on current memory usage and memory growth rate; Data precision adjustment, in response to the risk level reaching the downgrade trigger condition, triggers the downgrade mode, reducing the field export range from all fields to only include predefined core fields; Data write operations are performed based on the aforementioned degradation mode, and in response to the triggering of the degradation mode, an independent information page is appended to the generated target file. The information page records at least the degradation trigger time, a memory state snapshot, and a list of omitted non-core fields.

[0005] In one optional implementation, the calculation cycle of the memory growth rate is independent of the data shard write cycle, and the calculation frequency is higher than the shard adjustment frequency; the window length of the sliding time window is dynamically and adaptively adjusted based on the total amount of exported data or historical memory volatility; it also includes: in response to two or more consecutive shard adjustments failing to reduce the memory growth rate to the expected range, forcibly suspending the read and write operations of new data shards until the memory growth rate falls back to a safe threshold and then automatically resumes.

[0006] In one optional implementation, the triggering of the degradation mode is independently decided by the export decision engine based on the memory risk level, and the decision logic is decoupled from the shard size adjustment logic; the degradation mode does not change the data source query statement, and the field range reduction is achieved by any of the following methods: generating a lightweight data transfer object containing only core fields through dynamic proxy during the memory object construction stage; or dynamically excluding non-core fields through the excludeColumnFieldNames method in the native interface of EasyExcel (a Java-based Excel processing framework) during the serialization writing stage; during the execution of the degradation mode, the average size of a single data object in memory is lower than the average size of the normal mode.

[0007] In one optional implementation, the core fields are predefined and annotated in the data entity class using annotations, and the field classification configuration is scanned and loaded when the system starts; the trigger threshold for the degradation mode is independently configurable and supports dynamic modification at runtime.

[0008] In an optional implementation, the method further includes: providing a standardized manual intervention interface, which is open based on HTTP or RPC protocols to receive external operation and maintenance instructions; in response to receiving a pause instruction, recording the currently written data line number and the last written file offset position, releasing temporary resources but not closing the output stream, so that the export task enters a suspended state; in response to receiving a resume instruction, continuing to read subsequent fragments from the data source and appending them to the original file based on the recorded line number and offset position.

[0009] In an optional implementation, the method further includes: in response to receiving a dynamic parameter tuning instruction, hot-swapping the data fragment size parameter or degradation mode switch state in the export strategy without restarting the task or rebuilding the Writer, and subsequent fragment write operations take effect immediately; in response to a conflict between a manual intervention instruction and an automatic decision, the manual intervention instruction prevails, and the conflict event, the decision result, and the timestamp are written to an independent audit log; in response to a manual intervention instruction that restores the system memory load to normal and has no lasting effect, automatically switching the export strategy back to normal mode.

[0010] Secondly, the present invention provides a large file export optimization device based on real-time memory monitoring, applied to computer equipment, the device comprising: The memory status monitoring module is used to monitor physical memory usage and exported process heap memory usage in real time at preset periods. It sets a first preset threshold for physical memory usage, a second preset threshold for memory growth rate, a first threshold for memory stress count, and a second threshold for accelerated growth count, and calculates the memory growth rate. In response to the physical memory usage reaching the first preset threshold, it accumulates a memory stress count; in response to the memory growth rate exceeding the second preset threshold but the physical memory usage not reaching the first preset threshold, it accumulates an accelerated growth count; when the memory stress count reaches the first threshold or the accelerated growth count reaches the second threshold, it generates a risk warning signal. The decision engine module is connected to the memory status monitoring module and is used to receive the risk warning signal and generate throughput control instructions and data accuracy control instructions. The hierarchical export execution module, connected to the export decision engine module, is used to receive the adjustment instructions, dynamically update the data shard size based on the throughput control instructions, trigger the degradation mode when the risk level reaches the degradation trigger condition based on the data precision control instructions, reduce the field export range from all fields to only include predefined core fields, perform data writing operations, and append independent information pages to the generated target file in response to the triggering of the degradation mode. The information pages record at least the degradation trigger time, memory state snapshot, and a list of omitted non-core fields. The circuit breaker recovery and manual intervention module provides a standardized manual intervention interface, receives external operation and maintenance instructions, and responds to the instructions to perform operations such as pausing, resuming, dynamically adjusting parameters, and forcibly downgrading the export task. In case of conflict between manual intervention instructions and automatic decision-making, the manual intervention instructions shall prevail. In case of a system load recovery, the automatic switchback export strategy shall be implemented.

[0011] Thirdly, the present invention provides a computer device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the large file export optimization method based on real-time memory monitoring as described in the first aspect or any optional embodiment thereof.

[0012] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions, the computer instructions being used to cause a computer to execute the large file export optimization method based on real-time memory monitoring as described in the first aspect or any optional embodiment thereof.

[0013] Fifthly, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the large file export optimization method based on real-time memory monitoring as described in the first aspect or any optional embodiment thereof.

[0014] This invention achieves continuous dual-dimensional monitoring and cumulative confirmation of memory risks by setting a first preset threshold for physical memory utilization and a second preset threshold for memory growth rate, and introducing memory stress count and growth acceleration count, along with their corresponding first and second count thresholds. When physical memory utilization reaches the first preset threshold, a memory stress count is accumulated; when the memory growth rate exceeds the second preset threshold but physical memory utilization does not reach the first preset threshold, a growth acceleration count is accumulated. Risk is confirmed and an early warning signal is generated only when either count reaches its threshold, effectively avoiding false triggers caused by instantaneous fluctuations and continuously tracking the development trend of memory pressure. This ensures accurate identification of hidden risks when memory utilization has not yet exceeded the limit but the growth rate continues to be too rapid. Based on the confirmed early warning signal, the system performs throughput and data precision control, dynamically adjusting the data fragment size and reducing the field export range to core fields when the risk level reaches the degradation trigger condition. This prioritizes the export of core data when memory is tight. Simultaneously, by attaching a separate information page recording degradation information to the generated target file, the export results are made traceable. Therefore, this invention reduces the risk of memory overflow during large file export and improves the stability and reliability of export tasks in dynamic resource environments. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating the steps of a large file export optimization method based on real-time memory monitoring according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall process of a large file export optimization method based on real-time memory monitoring according to an embodiment of the present invention; Figure 3 This is a structural block diagram of a large file export optimization device based on real-time memory monitoring according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a memory status monitoring module of a large file export optimization device based on real-time memory monitoring according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the export decision engine module of a large file export optimization device based on real-time memory monitoring according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a hierarchical export execution module of a large file export optimization device based on real-time memory monitoring according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the circuit breaker recovery and manual intervention module of a large file export optimization device based on real-time memory monitoring according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0018] According to an embodiment of the present invention, a method for optimizing the export of large files based on real-time memory monitoring is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0019] This embodiment provides a method for optimizing large file export based on real-time memory monitoring, which can be used in the aforementioned computer devices, such as servers, personal computers, and industrial control computers. Figure 1 This is a flowchart of a large file export optimization method based on real-time memory monitoring according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Monitor the server's physical memory usage and the exported process heap memory usage in real time, set a first preset threshold for physical memory usage and a second preset threshold for memory growth rate, and calculate the memory growth rate based on multiple sampling points within the sliding time window.

[0020] Specifically, this step aims to establish a continuous awareness of system resources and the resource consumption of the export task itself. Through system-level interfaces provided by the operating system (e.g., / proc / meminfo on Linux or GlobalMemoryStatusEx on Windows), the overall physical memory usage and available physical memory capacity of the server are periodically collected. Simultaneously, data such as heap memory usage and heap memory utilization of the current export process are obtained through native interfaces provided by the Java Virtual Machine (e.g., JMX). To capture the dynamic trends of memory pressure, this step further collects heap memory usage samples at fixed time intervals (e.g., every 2 seconds) and stores the most recent N samples (e.g., the most recent 15 points, forming a 30-second sliding window) in a memory queue. Each time a new sample is collected, a linear fit (e.g., using the least squares method) is performed on all samples within the window, and the slope of the fitted line is determined as the current memory growth rate. This growth rate reflects the acceleration of memory consumption and can provide early warning of potential surges in memory pressure. Through the above real-time monitoring and calculation, a raw data foundation is provided for subsequent risk assessment.

[0021] Step S102: Monitor physical memory usage and exported process heap memory usage in real time at a preset period; set a first preset threshold for physical memory usage, a second preset threshold for memory growth rate, a first count threshold for memory stress count, and a second count threshold for growth acceleration count; calculate memory growth rate; accumulate memory stress count in response to the physical memory usage reaching the first preset threshold; accumulate growth acceleration count in response to the memory growth rate exceeding the second preset threshold but the physical memory usage not reaching the first preset threshold; generate a risk warning signal when the memory stress count reaches the first count threshold or the growth acceleration count reaches the second count threshold.

[0022] The core of this step lies in introducing a dual-counter cumulative confirmation mechanism, shifting from "single-point triggering" to "trend confirmation." The system pre-configures four independent parameters: a first preset threshold to determine if physical memory usage is too high (e.g., 85%), a second preset threshold to determine if the memory growth rate is too fast (e.g., 5MB / s), a first counting threshold to set the cumulative upper limit for memory stress counts (e.g., 3 consecutive occurrences), and a second counting threshold to set the cumulative upper limit for accelerated growth counts (e.g., 3 consecutive occurrences). Unlike conventional solutions that rely solely on a single instance of exceeding the limit to trigger an alarm, this step combines periodic monitoring with cumulative counting: within each monitoring period, if physical memory usage reaches the first preset threshold, the memory stress count increases; if the memory growth rate exceeds the second preset threshold but physical memory usage does not reach the first preset threshold, the accelerated growth count increases. Only when the cumulative value of either counter reaches its preset counting threshold does the system confirm the existence of a real risk and generate a risk warning signal. This mechanism can effectively avoid false triggers caused by instantaneous memory fluctuations (such as GC pauses and cache refreshes), and can continuously track the development trend of memory pressure to ensure that hidden risks are accurately identified when memory usage has not exceeded the standard but the growth rate continues to be too fast, providing a stable and reliable decision-making basis for dynamically adjusting the export strategy.

[0023] Step S103: In response to the risk warning signal, the following adjustment actions are performed: Throughput control and data precision control are executed according to their respective independent triggering conditions, and can take effect individually or in combination according to real-time resource conditions: Throughput control dynamically calculates and updates the data shard size based on the current memory usage rate and memory growth rate; Data precision control triggers a downgrade mode in response to the risk level reaching the downgrade triggering condition, reducing the field export range from all fields to only including predefined core fields.

[0024] Upon receiving a risk warning signal, the decision engine initiates two independent control mechanisms. The triggering conditions and decision-making logic of these two mechanisms do not interfere with each other, and they can take effect individually or in combination depending on the real-time resource situation. Throughput Control: This mechanism aims to balance export efficiency and memory load by adjusting the number of data rows read from the data source and written to Excel each time (i.e., the shard size). The decision engine queries a pre-defined shard size mapping table or calculates a new shard size using a dynamic formula based on the current memory usage and growth rate. For example, when memory usage is below 60% and the growth rate is slow, the shard size can be appropriately increased (e.g., from the default 1000 rows to 2000 rows) to improve export speed; when memory usage exceeds 80% or the growth rate is rapid, the shard size is proactively reduced (e.g., reduced to 300 rows) to lower the peak memory usage per process and prevent excessive memory accumulation.

[0025] Data Precision Control: This mechanism is a more proactive degradation measure. When the risk level further escalates (e.g., memory usage exceeds 90% or the growth rate continues to exceed the limit) and reaches the preset degradation trigger conditions, the system automatically enters degradation mode. In this mode, the data export scope is reduced from all fields to only the most important core fields (such as order number, amount, time, etc.), while non-core fields (such as delivery address, remarks, device model, etc.) are discarded. This strategy of sacrificing pawns to save the king can significantly reduce the memory overhead of a single data entry, thereby prioritizing the complete export of core data with limited memory resources. Because the trigger conditions and adjudication logic of throughput control and data precision control are decoupled, the system can be flexibly combined: for example, only reducing the number of shards during mild stress, and simultaneously activating degradation mode during severe stress, to achieve tiered defense.

[0026] Step S104: Perform a data write operation based on the downgrade mode, and in response to the triggering of the downgrade mode, attach an independent information page to the generated target file. The information page records at least the downgrade trigger time, memory state snapshot, and a list of omitted non-core fields.

[0027] This step is responsible for implementing the adjusted strategy and providing traceability of the results. If the degradation mode is triggered in step S103, the tiered export execution module will write data according to the reduced set of fields: during the in-memory object construction phase, a lightweight data transfer object containing only core fields is generated through dynamic proxy; or during the serialization writing phase, the excludeColumnFieldNames method of the EasyExcel native interface is called to dynamically exclude non-core fields. Regardless of the method used, the data source query statement remains unchanged, ensuring minimal intrusion into the business code.

[0028] Specifically, once the downgrade mode is actually executed, the system will automatically create a separate information page at the end of the generated target file (for example, a worksheet named "Downgrade Instructions" in an Excel file, or presented as an appendix in other format files). This information page records the following content in a clear and easy-to-read format: Downgrade trigger time (accurate to milliseconds); A snapshot of the memory state when a downgrade is triggered (including physical memory usage, exported process heap memory usage, memory growth rate, etc.). List of omitted non-core fields (such as "shipping address", "remarks", "device model").

[0029] This design ensures that users are fully aware of the degradation measures taken due to resource constraints upon receiving the exported file, and understand which fields have been omitted. This avoids misunderstandings caused by missing data and provides a basis for subsequent auditing and troubleshooting. If the degradation mode is not triggered, the complete fields are exported according to the normal process, without the need for an additional information page.

[0030] In summary, this embodiment provides an intelligent, stable, and reliable solution for exporting large amounts of data to Excel through a complete closed loop of "monitoring-prediction-control-traceability". Example

[0031] In a typical implementation, this large file export optimization method based on real-time memory monitoring is applied to a business system oriented towards data analysis and report generation. The system is deployed on a physical server or virtualized instance, running a Java Virtual Machine, and integrates EasyExcel as the underlying Excel processing tool. The core service layer of the system maintains multiple concurrent large file export tasks, which may be triggered by front-end user operations or automatically initiated by scheduled tasks.

[0032] Taking a typical scenario of exporting millions of data points as an example, the export task starts when a user requests to export a comprehensive report containing core order information, payment details, and logistics tracking. The system first initializes the memory status monitoring module. This module uses the underlying interface provided by the operating system—` / proc / meminfo` (memory information file) in Linux and `GlobalMemoryStatusEx` (global memory status function) in Windows—to periodically collect the overall usage and available capacity of the server's physical memory. Simultaneously, it obtains the current JVM heap memory usage, heap memory utilization, and garbage collection frequency and time through the JMX (Java Management Extensions) interface. The monitoring thread performs sampling twice per second and stores the heap memory usage of the most recent 30 sampling points (i.e., a 15-second time window) in a circular queue. Whenever a new sampling point is enqueued, the monitoring module performs least-squares linear fitting on all data points within the window, calculating a fitted straight line. The slope of this line is defined as the current memory growth rate, in MB / s. This metric quantifies the acceleration of memory consumption and can reflect potential stress accumulation earlier than memory usage alone.

[0033] After the export task begins, the tiered export execution module runs according to the default configuration parameters. By default, the shard size is set to 1000 records, meaning that 1000 complete records are retrieved from the database each time. Each record contains all fields such as core order information, payment details, and logistics tracking, and then this batch of data is written to an Excel file. In the initial stage of export, the server's physical memory usage remains at around 60%, the JVM heap memory usage is stable at 1.2GB, and the memory growth rate is approximately 2MB / s, all of which are lower than the preset first threshold (e.g., 80%) and second threshold (e.g., 5MB / s). At this time, the export decision engine determines that the system is in a low-risk state, does not trigger any adjustments, and the task runs at full speed.

[0034] When the export task reached approximately 300,000 data entries, another independent data analysis task started executing on the server, consuming approximately 2GB of physical memory. The monitoring module immediately detected that the physical memory usage rate climbed from 60% to 72%. Simultaneously, due to increased operating system memory pressure, the JVM garbage collection frequency of the export process increased from once per second to three times per second, and the time taken for a single garbage collection also increased from 20 milliseconds to 80 milliseconds. This resulted in untimely release of JVM heap memory, with heap memory usage gradually rising from 1.2GB to 1.6GB, and the memory growth rate jumping from 2MB / s to 8MB / s, exceeding the second preset threshold of 5MB / s.

[0035] Although the physical memory utilization rate of 72% had not yet reached the first preset threshold of 80%, the export decision engine, based on the rule of "growth rate as an independent risk assessment dimension," determined that the current state was "accelerating stress" and generated a risk warning signal. The decision engine then initiated the control logic, with the throughput control branch querying the preset shard size mapping table based on the current memory utilization rate of 72% and the growth rate of 8MB / s. This mapping table, set according to previous stress test data, stipulated that when memory utilization was between 70% and 80% and the growth rate exceeded 5MB / s, the shard size should be adjusted to 300 rows. Therefore, the tiered export execution module dynamically adjusted the shard size for subsequent data reading from 1000 rows to 300 rows, reducing the peak memory usage for a single batch of data from approximately 80MB to approximately 25MB, and the memory growth rate immediately dropped back to 4MB / s.

[0036] However, due to the continued memory consumption of external data analysis tasks, the server's physical memory utilization continued to climb to 78%, while the JVM heap memory utilization remained stable at 1.8GB, but garbage collection frequency remained high. The decision engine detected that the degradation trigger condition (memory utilization exceeding 75% for three consecutive monitoring cycles) had been met, so the data precision control branch independently triggered the degradation mode. The system pre-defined the importance classification of fields in the order entity class using the `@CoreField` and `@NonCoreField` annotations: core fields include order number, transaction amount, order time, and payment status; non-core fields include shipping address, user remarks, device model, and extended attributes. After the degradation mode was triggered, the tiered export execution module used dynamic proxy technology. After retrieving the complete result set from the database, it dynamically generated a lightweight DTO (Data Transfer Object) containing only core fields during the memory object construction phase, discarding the original non-core fields. This operation reduced the average size of a single data record in memory from approximately 2KB to approximately 0.6KB, a reduction of 70%. Although the shard size remained at 300 rows, the memory usage of a single batch of data further decreased from 25MB to 7.2MB, significantly alleviating the JVM heap memory pressure and gradually restoring the garbage collection frequency to normal.

[0037] During the degrade mode execution, the system continuously monitored memory status. When the external data analysis task ended, the server's physical memory utilization dropped from 78% to 65%, the memory growth rate stabilized below 1MB / s, and no warnings were triggered for 10 consecutive monitoring cycles. The decision engine determined that the system load had returned to normal, and since there were no continuous manual intervention commands, it automatically switched the export strategy back to normal mode: data precision adjustment exited degrade mode, and subsequent data resumed full field export; throughput adjustment also gradually restored the shard size from 300 records to 1000 records to improve the export efficiency of the remaining data.

[0038] Throughout the export process, the circuit breaker recovery and manual intervention modules remain on standby. When operations and maintenance personnel detect an abnormal increase in server load through the monitoring system, they can send commands, such as requesting to pause the export task, via the REST interface opened based on HTTP or RPC protocols. Upon receiving the command, the system immediately records the currently written data row number and the physical offset position of the file output stream, then releases temporary resources but keeps the output stream open, and the task enters a suspended state. When the load returns to normal, operations and maintenance personnel send a recovery command. Based on the recorded breakpoint information, the system positions the file output stream pointer to the previous position and continues to read subsequent data from the database, completing the remaining export by appending data. The final generated file is complete and includes an automatically appended description page during the degradation process. This page details the degradation trigger time, the memory state snapshot at that time, and a list of omitted non-core fields.

[0039] This implementation method continuously monitors memory status and predicts trends. It dynamically adjusts throughput through sharding when resource pressure first appears. As pressure continues to rise, it downgrades and compresses data precision at the field level to ensure no core business data is lost. Simultaneously, a manual intervention interface provides operations and maintenance personnel with flexible emergency control measures. The entire process is automatic, continuous, and traceable, achieving intelligent resource adaptation and proactive risk defense in scenarios involving large-scale data export. Example

[0040] In another typical implementation, this large file export optimization method based on real-time memory monitoring is applied to the scenario of batch exporting daily reports from a financial transaction system. This system is deployed on a node in a high-configuration physical server cluster, running a Java Virtual Machine with 32GB of heap memory, and integrates EasyExcel as the underlying Excel processing framework. Unlike the aforementioned implementations, the export tasks in this scenario are characterized by periodicity, batch processing, and a larger data volume. Every day at midnight, the full transaction details of the previous day need to be exported, often reaching tens of millions of records. Furthermore, the report templates are complex, containing multi-level headers, data summary formulas, and multiple related worksheets, making the data processing logic more cumbersome than that of conventional reports.

[0041] When the end-of-day batch processing task starts, the system first performs a resource pre-check. The memory status monitoring module begins collecting server physical memory usage and exported process heap memory usage three times per second, storing the most recent 60 sampling points in a circular buffer corresponding to a 20-second time window. The monitoring module obtains physical memory data through low-level interfaces provided by the operating system, specifically ` / proc / meminfo` in Linux and `GlobalMemoryStatusEx` in Windows. Simultaneously, it obtains JVM heap memory usage details and garbage collection status through the JMX interface. After each sampling, the monitoring module performs a weighted moving average calculation on the heap memory usage data within the window to eliminate the impact of instantaneous fluctuations, and calculates the current memory growth rate based on the difference between the average values ​​of adjacent windows. This growth rate serves as a second risk assessment dimension independent of memory usage, enabling the system to identify the accelerating accumulation trend of memory pressure before memory usage reaches a dangerous threshold.

[0042] Transaction details are stored in a distributed database, totaling approximately 12 million records. Each transaction record contains 22 fields, including transaction serial number, transaction time, transaction amount, transaction fee, merchant ID, terminal ID, the first six and last four digits of the card number, transaction status, settlement status, and remarks. Among these, the transaction serial number, transaction time, transaction amount, and transaction status are pre-marked as core fields using the `@CoreField` annotation, while the remaining fields are marked as non-core fields. The tiered export execution module starts with the default configuration, with an initial shard size of 2000 records. Due to the complexity of the report templates, EasyExcel needs to maintain the styles and formula references of multiple worksheets during the write process, resulting in a prolonged memory release cycle after a single batch of data is written.

[0043] When the export task reached approximately 2 million data entries, the system began to exhibit measurable changes in memory pressure. The monitoring module detected that the JVM heap memory usage gradually increased from 40% at startup to 65%, and the memory growth rate rose from an initial 3MB / s to 6MB / s. At this point, the physical memory usage was 55%, which had not reached the first preset threshold of 85%, but the memory growth rate had exceeded the second preset threshold of 5MB / s. According to the judgment rules of this invention, the memory growth rate is used as a second risk judgment dimension independent of the memory usage rate. The export decision engine determines that the current system is in a state of accelerated accumulation of memory pressure and generates a risk warning signal.

[0044] The throughput control branch queries the preset shard size mapping table based on the current memory utilization of 65% and the growth rate of 6MB / s. This mapping table is set according to previous stress test data on complex report templates. When the memory utilization is between 60% and 70% and the growth rate exceeds 5MB / s, the shard size is adjusted to 800 rows. The tiered export execution module dynamically adjusts the shard size for subsequent data reads from 2000 rows to 800 rows. After the adjustment, the memory residence time for single batch data writes is shortened, the garbage collector's collection efficiency is improved, and the memory growth rate is reduced from 6MB / s to 4.5MB / s.

[0045] When the export task reached 5 million data entries, EasyExcel needed to maintain a large number of style objects and formula caches during the writing process due to the high complexity of the report template formulas, causing a continuous increase in JVM old generation memory usage. The monitoring module detected that the memory growth rate increased from 4.5MB / s to 7MB / s, the young generation garbage collection frequency increased from 2 times per second to 5 times per second, and some objects began to be promoted to the old generation prematurely. The physical memory usage rate rose to 68% due to increased pressure from the operating system file cache, but it was still below the first preset threshold. The decision engine detected that the growth rate exceeded the limit for the second time and continued for several cycles without decreasing, determining that the current risk level had reached the downgrade trigger condition.

[0046] Data precision adjustment branches trigger a degradation mode. Upon receiving a degradation instruction, the tiered export execution module keeps the data source query statement unchanged and generates a lightweight data transfer object containing only core fields during the memory object construction phase using dynamic proxy technology. Sixteen non-core fields from the original result set, including transaction fees, merchant ID, terminal ID, card number portion, settlement status, and remarks, are discarded and no longer participate in subsequent serialization and writing processes. Actual testing showed that the average size of a single data entry in memory decreased from approximately 1.5KB to approximately 0.3KB, a reduction of 80%. With a shard size of 800 records, the memory usage of a single batch of data decreased from approximately 1.2MB to approximately 240KB, and the growth rate of JVM old generation memory slowed, decreasing from 7MB / s to below 2MB / s.

[0047] During the downgrade process, the system automatically appends a separate worksheet named "Export Description" to the end of the exported file. This worksheet records the downgrade trigger time as 1:47 AM, and includes a snapshot of the memory state at the time of triggering, showing physical memory usage at 68%, JVM heap memory usage at 71%, memory growth rate at 7.2 MB / s, and a list of 16 omitted non-core fields. This record ensures the traceability of the downgrade operation, allowing business personnel to clearly understand the changes in data range when reviewing reports the following day.

[0048] When the export task reached 8 million records, the monitoring module detected that JVM heap memory usage began to decline, with the memory growth rate remaining consistently below 1MB / s, and no alerts were triggered for 15 consecutive monitoring cycles. The decision engine determined that the system load had returned to normal, and with no further manual intervention commands, automatically switched the export strategy back to normal mode. Data precision adjustment exited degraded mode, and subsequent data was exported with all fields restored; throughput adjustment gradually restored the shard size from 800 records to 2000 records to improve the export efficiency of the remaining 4 million records.

[0049] Throughout the export process, the circuit breaker recovery and manual intervention modules remained on standby. Operations personnel sent query commands to the standardized manual intervention interface for this export task via the internal operations platform to obtain real-time status, including the amount of data processed, current shard size, whether in degraded mode, current memory usage, and growth rate. After confirming that the system was in degraded mode but running stably, the operations personnel decided not to intervene and allowed the system to complete the export automatically. At 3:20 AM, the export task was completed, generating a file of approximately 380MB, including an explanatory page automatically attached during the degrade period. Post-analysis showed that if the dynamic control mechanism of this invention had not been used, and the system had exported all 2000 shard fields as originally planned, a memory overflow was expected around 2:30 AM, causing the day-end report generation to fail.

[0050] This implementation uses memory growth rate as an independent risk assessment dimension, enabling the system to identify the accelerating accumulation of memory pressure even when memory utilization has not reached a threshold. Through an independent decoupling design of throughput control and data accuracy control, it achieves tiered resource pressure response, first alleviating pressure by slowing down the system, then sacrificing accuracy for survival. The mandatory attachment mechanism of the degradation description page ensures the traceability of data range changes. The entire process is automatically completed by the system in a closed-loop control from perception and decision-making to execution, guaranteeing the complete export of core business data and the stable operation of the system. Example

[0051] This embodiment also provides a large file export optimization device based on real-time memory monitoring. This device is used to implement the above-described method embodiments and preferred embodiments, and the content already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0052] This embodiment provides a large file export optimization device based on real-time memory monitoring, such as... Figures 3 to 7 As shown, it includes: The memory status monitoring module 201 is used to monitor the physical memory usage and the exported process heap memory usage in real time at a preset period. It sets a first preset threshold for physical memory usage, a second preset threshold for memory growth rate, a first count threshold for memory stress count, and a second count threshold for growth acceleration count, and calculates the memory growth rate. In response to the physical memory usage reaching the first preset threshold, it accumulates a memory stress count. In response to the memory growth rate exceeding the second preset threshold but the physical memory usage not reaching the first preset threshold, it accumulates a growth acceleration count. When the memory stress count reaches the first count threshold or the growth acceleration count reaches the second count threshold, it generates a risk warning signal.

[0053] The export decision engine module 202 is connected to the memory status monitoring module and is used to receive the risk warning signal and generate throughput control instructions and data accuracy control instructions.

[0054] The hierarchical export execution module 203, connected to the export decision engine module, is used to receive the adjustment instructions, dynamically update the data shard size based on the throughput control instructions, trigger the degradation mode when the risk level reaches the degradation trigger condition based on the data precision control instructions, reduce the field export range from all fields to only include predefined core fields, perform data writing operations, and append independent information pages to the generated target file in response to the triggering of the degradation mode. The information pages record at least the degradation trigger time, memory state snapshot, and a list of omitted non-core fields.

[0055] The circuit breaker recovery and manual intervention module 204 is used to provide a standardized manual intervention interface, receive external operation and maintenance instructions, and respond to the instructions to perform operations such as pausing, resuming, dynamically adjusting parameters and forcibly downgrading the export task. In response to the conflict between the manual intervention instruction and the automatic decision, the manual intervention instruction shall prevail. In response to the system load returning to normal, the automatic switchback export strategy shall be implemented.

[0056] In some optional implementations, the memory status monitoring module 201 includes: The system resource acquisition unit 2011 is used to periodically collect the server's physical memory usage and available physical memory capacity through an operating system-level interface, which includes / proc / meminfo in Linux or GlobalMemoryStatusEx in Windows. The process resource acquisition unit 2012 is used to periodically collect and export the process's heap memory usage, heap memory utilization rate, garbage collection frequency, and single garbage collection time through the Java Virtual Machine local interface. The trend calculation unit 2013 is used to calculate the memory growth rate based on multiple sampling points within a sliding time window. Specifically, it includes a sampling queue maintenance subunit 20131 and a linear fitting subunit 20132. The sampling queue maintenance subunit 20131 is used to collect heap memory usage sampling points at fixed time intervals and save the most recent N sampling points to form a sliding window. The linear fitting subunit 20132 is used to perform linear fitting on the sampling points within the sliding window for each new sampling point collected, and the slope of the fitted line is determined as the current memory growth rate. The linear fitting includes the least squares method or the exponentially weighted moving average method. The dual-judgment unit 2014 is used to accumulate memory stress count when the physical memory utilization rate reaches a first preset threshold, accumulate growth acceleration count when the memory growth rate exceeds a second preset threshold and the physical memory utilization rate does not reach the first preset threshold, and generate a risk warning signal when the memory stress count reaches the first counting threshold or the growth acceleration count reaches the second counting threshold.

[0057] In some optional implementations, the derived decision engine module 202 includes: The throughput control decision unit 2021 is used to query the preset shard size mapping table based on the current memory usage and memory growth rate, determine the data shard size to be used, and generate throughput control instructions. The data precision control decision unit 2022 is used to monitor the memory risk level and generate a degradation mode trigger instruction independently of the throughput control decision when the risk level reaches the degradation trigger condition. The fragment size mapping table is pre-configured based on memory usage range and memory growth rate range, and supports dynamic updates at runtime.

[0058] In some optional implementations, the hierarchical export execution module 203 includes: The sharding adjustment execution unit 2031 is used to receive throughput control instructions, dynamically update the sharding size parameters of data reading operations, and make subsequent data shards execute according to the new size. The field filtering execution unit 2032 is used to reduce the field range by keeping the data source query statement unchanged after receiving the degradation mode trigger instruction, using any of the following methods: During the memory object construction phase, a lightweight data transfer object containing only core fields is generated through dynamic proxies; During the serialization writing phase, non-core fields are dynamically excluded using the excludeColumnFieldNames method in the EasyExcel native interface. The description page generation unit 2033 is used to automatically create an independent information page (e.g., a worksheet named "Degradation Description" in an Excel file, or an appendix in other format files) at the end of the generated target file after the actual execution of the degradation mode. This page records the degradation trigger time, a snapshot of the memory state at the time of triggering, and a list of omitted non-core fields.

[0059] In some optional implementations, the circuit breaker recovery and manual intervention module 204 includes: Interface service unit 2041 is used to provide standardized APIs based on HTTP or RPC protocols and receive external operation and maintenance instructions; The task status management unit 2042 is used to maintain the running status of the export task. In response to the pause command, it records the currently written data line number and the last written file offset position, releases temporary resources but does not close the output stream, and puts the task into a suspended state. In response to the resume command, it continues to read subsequent fragments from the data source based on the recorded line number and offset position and appends them to the original file. The dynamic parameter tuning unit 2043 is used to respond to dynamic parameter tuning instructions to hot replace the data fragment size parameter or degradation mode switch state in the export strategy without restarting the task or rebuilding the Writer, so that subsequent fragment write operations take effect immediately. The conflict handling unit 2044 is used to prioritize the manual intervention command when there is a conflict between the manual intervention command and the automatic decision, and to write the conflict event, the decision result and the timestamp into an independent audit log. The automatic recovery unit 2045 monitors the system memory load and automatically switches the export strategy back to normal mode when the load returns to normal and there are no ongoing manual intervention commands.

[0060] The further functional descriptions of the above modules and units are the same as those in the corresponding method embodiments described above, and will not be repeated here.

[0061] The device provided in this embodiment, through the collaborative work of its various modules, achieves a complete closed loop of real-time monitoring of memory risks, dual-counter cumulative confirmation, dynamic control, and manual intervention, which can effectively improve system stability and data export success rate in large file export scenarios. Example

[0062] In this embodiment, the large file export optimization device based on real-time memory monitoring is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0063] This invention also provides a computer device having the above-described features. Figure 8 The large file export optimization device shown is based on real-time memory monitoring.

[0064] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0065] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0066] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0067] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0069] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0070] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0071] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0072] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for optimizing the export of large files based on real-time memory monitoring, applied to computer equipment, characterized in that: The method is performed by the computer device and includes: The system monitors physical memory usage and exported process heap memory usage in real time at preset intervals, sets a first preset threshold for physical memory usage, a second preset threshold for memory growth rate, a first count threshold for memory stress count, and a second count threshold for growth acceleration count, and calculates the memory growth rate. In response to the physical memory usage rate reaching a first preset threshold, an accumulated memory stress count is performed; In response to the memory growth rate exceeding the second preset threshold and the physical memory utilization rate not reaching the first preset threshold, the cumulative growth acceleration count is set. When the memory stress count reaches the first count threshold or the growth acceleration count reaches the second count threshold, a risk warning signal is generated. In response to the risk warning signal, a preset adjustment action is executed.

2. The method according to claim 1, characterized in that, The adjustment actions include: Throughput control dynamically calculates and updates data fragment size based on current memory usage and memory growth rate; Data precision adjustment, in response to the risk level reaching the downgrade trigger condition, triggers the downgrade mode, reducing the field export range from all fields to only include predefined core fields; Data write operations are performed based on the aforementioned degradation mode, and in response to the triggering of the degradation mode, an independent information page is appended to the generated target file. The information page records at least the degradation trigger time, a memory state snapshot, and a list of omitted non-core fields.

3. The method according to claim 2, characterized in that, The calculation cycle for the memory growth rate is independent of the data sharding write cycle, and the calculation frequency is higher than the sharding adjustment frequency; the window length of the sliding time window is dynamically and adaptively adjusted based on the total amount of exported data or historical memory volatility. It also includes: in response to two or more consecutive fragment adjustments failing to reduce the memory growth rate to the expected range, forcibly suspending read and write operations on new data fragments until the memory growth rate falls back to a safe threshold and then automatically resumes.

4. The method according to claim 2, characterized in that, The triggering degradation mode is independently decided by the export decision engine based on the memory risk level, and the decision logic is decoupled from the fragment size adjustment logic; The degradation mode does not change the data source query statement and reduces the field range using any of the following methods: During the memory object construction phase, a lightweight data transfer object containing only core fields is generated through dynamic proxy; or during the serialization writing phase, non-core fields are dynamically excluded through the excludeColumnFieldNames method in the native interface of EasyExcel (a Java-based Excel processing framework). During the degraded mode execution, the average size of a single data object in memory is lower than the average size in normal mode.

5. The method according to claim 4, characterized in that, The core fields are predefined and annotated in the data entity class using annotations. The system scans and loads the field classification configurations when it starts up. The trigger threshold for the degradation mode is independently configurable and can be dynamically modified at runtime.

6. The method according to any one of claims 2 to 5, characterized in that, Also includes: A standardized manual intervention interface is provided, which is open based on HTTP or RPC protocols and is used to receive external operation and maintenance instructions; In response to receiving a pause command, the system records the currently written data line number and the last written file offset, releases temporary resources but does not close the output stream, and puts the export task into a suspended state. In response to receiving a recovery command, the system continues to read subsequent fragments from the data source and appends them to the original file based on the recorded line number and offset position. In response to receiving a dynamic parameter tuning command, the data fragment size parameter or degradation mode switch status in the export strategy can be hot-replaced without restarting the task or rebuilding the Writer, and subsequent fragment write operations will take effect immediately. In response to conflicts between manual intervention instructions and automated decisions, the manual intervention instructions shall prevail, and the conflict event, the decision result, and the timestamp shall be written to a separate audit log. In response to a manual intervention command that indicates the system memory load has returned to normal and has no lasting effect, the export strategy is automatically switched back to normal mode.

7. A large file export optimization device based on real-time memory monitoring, applied to computer equipment, characterized in that, The device includes: The memory status monitoring module is used to monitor physical memory usage and exported process heap memory usage in real time at preset periods. It sets a first preset threshold for physical memory usage, a second preset threshold for memory growth rate, a first threshold for memory stress count, and a second threshold for accelerated growth count, and calculates the memory growth rate. In response to the physical memory usage reaching the first preset threshold, it accumulates a memory stress count; in response to the memory growth rate exceeding the second preset threshold but the physical memory usage not reaching the first preset threshold, it accumulates an accelerated growth count; when the memory stress count reaches the first threshold or the accelerated growth count reaches the second threshold, it generates a risk warning signal. The decision engine module is connected to the memory status monitoring module and is used to receive the risk warning signal and generate throughput control instructions and data accuracy control instructions. The hierarchical export execution module, connected to the export decision engine module, is used to receive the adjustment instructions, dynamically update the data shard size based on the throughput control instructions, trigger the degradation mode when the risk level reaches the degradation trigger condition based on the data precision control instructions, reduce the field export range from all fields to only include predefined core fields, perform data writing operations, and append independent information pages to the generated target file in response to the triggering of the degradation mode. The information pages record at least the degradation trigger time, memory state snapshot, and a list of omitted non-core fields. The circuit breaker recovery and manual intervention module provides a standardized manual intervention interface, receives external operation and maintenance instructions, and responds to the instructions to perform operations such as pausing, resuming, dynamically adjusting parameters, and forcibly downgrading the export task. In case of conflict between manual intervention instructions and automatic decision-making, the manual intervention instructions shall prevail. In case of a system load recovery, the automatic switchback export strategy shall be implemented.

8. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the large file export optimization method based on real-time memory monitoring as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the large file export optimization method based on real-time memory monitoring as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to execute the large file export optimization method based on real-time memory monitoring as described in any one of claims 1 to 6.

Images