Data processing method and device, electronic equipment, storage medium and product

Abstract

This application provides a data processing method, apparatus, electronic device, storage medium, and product. The method includes: when a data change event occurs in a source database, writing the data to be processed associated with the data change event into a message queue and determining the message queue backlog; determining the open duration of a data aggregation window based on the message queue backlog; and within the open duration of the data aggregation window, aggregating and processing the data to be processed, generating a target file and the file storage path of the target file in the source database, and storing the target file to the file storage path. This method ensures the timeliness of data synchronization while improving the overall efficiency and stability of data writing, thus making it suitable for data processing needs in high-concurrency data synchronization scenarios.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device, storage medium and product. Background Technology

[0002] In existing large-scale deployment scenarios of energy storage power stations, data synchronization generally adopts HTTP API polling to write data to analytical databases one by one. This forces massive amounts of real-time data to be written to analytical databases one by one. In order to alleviate the writing pressure, fixed time windows are often introduced for data aggregation. This approach cannot adapt to the drastic load fluctuations in large-scale energy storage power station scenarios, and has become the core bottleneck restricting writing efficiency and stability.

[0003] Because fixed window strategies lack the ability to detect real-time business load, they often execute write operations immediately upon the end of the window, which can easily lead to resource contention with the high-load periods of the business system. Furthermore, the fixed window length cannot be dynamically adjusted according to the real-time data volume. During off-peak periods, excessively long windows result in invalid writes and wasted resources, while during peak periods, excessively short windows cause data backlog and write failures.

[0004] Therefore, breaking the rigid limitations of fixed windows and establishing a data synchronization mechanism that can dynamically adjust with the workload and actively avoid peaks has become the key to solving the data consistency problem in high-concurrency scenarios. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, storage medium, and product to ensure the timeliness of data synchronization while improving the overall efficiency and stability of data writing, thereby meeting the data processing needs in high-concurrency data synchronization scenarios.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0007] In the event of a data change event in the source database, the data to be processed associated with the data change event is written to the message queue, and the message queue backlog is determined.

[0008] Based on the message queue backlog, determine the open duration of the data aggregation window;

[0009] During the open duration of the data aggregation window, the data to be processed is aggregated and processed to generate a target file and the file storage path of the target file in the source database, and the target file is stored in the file storage path.

[0010] In one possible implementation, the data change event is monitored in the following manner:

[0011] Deploy a listening component in the source database;

[0012] The monitoring component monitors log data in the source database to determine whether a data change event has occurred.

[0013] In one possible implementation, determining the open duration of the data aggregation window based on the message queue backlog includes:

[0014] Obtain the window open duration calculation rule; in the window open duration calculation rule, the calculated open duration and the message queue backlog have a non-linear mapping relationship;

[0015] The open duration of the data aggregation window is determined based on the message queue backlog and the window open duration calculation rules.

[0016] In one possible implementation, the aggregation process of the data to be processed to generate a target file and the file storage path of the target file in the source database includes:

[0017] The data to be processed is aggregated based on preset encoding rules to generate the target file;

[0018] Based on the preset file format, the database name of the source database, and the table name, timestamp, and business feature code associated with the data to be processed, the file storage path of the target file in the source database is generated; the timestamp is the end time of the data aggregation window.

[0019] In one possible implementation, after storing the target file to the file storage path, the method further includes:

[0020] After a preset delay period after the data aggregation window is closed, the dynamic window is launched;

[0021] During the open duration of the dynamic window, the target files stored in the source database are loaded into the target database in batches.

[0022] In one possible implementation, the step of batch loading the target files stored in the source database into the target database includes:

[0023] The target files stored in the source database are imported in batches into the target directory deployed in the target database.

[0024] If an import fails during the batch import process, an alarm is triggered, and the target database is controlled to re-execute the batch import process.

[0025] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0026] The writing module is used to write the data to be processed associated with the data change event into the message queue when a data change event occurs in the source database, and to determine the message queue backlog of the message queue.

[0027] The determination module is used to determine the open duration of the data aggregation window based on the message queue backlog.

[0028] The generation module is used to aggregate and process the data to be processed within the open duration of the data aggregation window, generate a target file and the file storage path of the target file in the source database, and store the target file in the file storage path.

[0029] In one possible implementation, the data processing device further includes: a monitoring module;

[0030] The monitoring module is configured as follows:

[0031] Deploy a listening component in the source database;

[0032] The monitoring component monitors log data in the source database to determine whether a data change event has occurred.

[0033] In one possible implementation, the determining module is further configured to:

[0034] Obtain the window open duration calculation rule; in the window open duration calculation rule, the calculated open duration and the message queue backlog have a non-linear mapping relationship;

[0035] The open duration of the data aggregation window is determined based on the message queue backlog and the window open duration calculation rules.

[0036] In one possible implementation, the generation module is further configured to:

[0037] The data to be processed is aggregated based on preset encoding rules to generate the target file;

[0038] Based on the preset file format, the database name of the source database, and the table name, timestamp, and business feature code associated with the data to be processed, the file storage path of the target file in the source database is generated; the timestamp is the end time of the data aggregation window.

[0039] In one possible implementation, the data processing device further includes: a startup module and a loading module;

[0040] The startup module is configured as follows:

[0041] After a preset delay period after the data aggregation window is closed, the dynamic window is launched;

[0042] The loading module is configured as follows:

[0043] During the open duration of the dynamic window, the target files stored in the source database are loaded into the target database in batches.

[0044] In one possible implementation, the loading module is further configured to:

[0045] The target files stored in the source database are imported in batches into the target directory deployed in the target database.

[0046] If an import fails during the batch import process, an alarm is triggered, and the target database is controlled to re-execute the batch import process.

[0047] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0048] The memory stores computer-executed instructions;

[0049] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0050] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0051] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0052] The data processing method, apparatus, electronic device, storage medium, and product provided in this application's embodiments use source database change events as the processing starting point, message queues as asynchronous buffer carriers, and message queue backlog as the basis for window duration adjustment. Within a controlled window, the aggregation of data to be processed and the creation of structured files for disk storage are completed, enabling the data processing chain to dynamically adjust its processing rhythm according to the actual load. This avoids resource waste caused by frequently generating small files in low-load scenarios and alleviates the consumption lag problem caused by a fixed window's inability to process backlogged data in a timely manner in high-load scenarios. Furthermore, by stably associating the target file path with the source database, the feasibility and consistency of subsequent file retrieval, data traceability, and archiving management are improved. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0054] Figure 1 Flowchart of the data processing method provided in this application Figure 1 ;

[0055] Figure 2 Flowchart of the data processing method provided in this application Figure 2 ;

[0056] Figure 3 A schematic diagram of the data processing apparatus provided in this application;

[0057] Figure 4 A schematic diagram of the structure of a data processing apparatus according to yet another embodiment of this application;

[0058] Figure 5 A schematic diagram of the structure of a data processing apparatus according to another embodiment of this application;

[0059] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0060] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0062] With the development of smart grid operation monitoring and energy storage dispatch technologies, high-frequency acquisition, centralized aggregation, and analysis of energy storage power station operation data have become key foundational capabilities of the power information system. In multi-power station interconnection scenarios, the dispatch platform needs to continuously receive multi-dimensional operation data to support dispatch decisions. The system consists of modules such as data acquisition, access services, message queues, source databases, and file storage.

[0063] With deployments at scales of tens to hundreds of stations, the system faces high concurrency and real-time data processing pressure. How to efficiently organize data flow based on database change events and adapt to system load control and aggregation strategies directly determines the stability and scalability of subsequent analysis and processing links, and has significant engineering application value.

[0064] In existing technologies, the most common practice for energy storage monitoring platforms is to use interface polling or event triggering to synchronize power plant operation data one by one or in small batches to the subsequent processing system. Although some solutions also introduce message queues as buffer components, their core processing logic still tends to send and write data as soon as possible after it arrives, lacking an aggregation control mechanism that matches the real-time accumulation status.

[0065] When data changes occur in the source database, existing solutions typically only deliver the data to a message queue and process it at fixed intervals or with a fixed number of messages. This works under low concurrency, but its shortcomings are obvious in high-concurrency scenarios with multiple data centers. The fixed processing rhythm cannot adaptively adjust to message accumulation. Under low load, frequent writes are likely to occur, resulting in high resource consumption. Under high load, the window is too short, causing queue backlog and consumption delays.

[0066] Meanwhile, the lack of reasonable rhythm control in data aggregation makes it difficult to balance real-time performance and resource utilization. Furthermore, the lack of unified planning for file generation and storage paths leads to difficulties in retrieval and verification, as well as complex management. These problems are exacerbated by network fluctuations or sudden increases in traffic, resulting in link jitter, timing disorder, and reduced timeliness, failing to meet the requirements of high concurrency, high reliability, and stable operation in smart grid energy storage scenarios.

[0067] In view of this, how to organize the subsequent processing flow of the data to be processed in a way that is more suitable for high-concurrency business loads after a data change event occurs in the source database, so as to reduce resource consumption and improve data processing efficiency and timeliness, has become an urgent technical problem to be solved.

[0068] To address the aforementioned issues, a data processing method is proposed. By introducing a "message queue backlog-driven data aggregation window open duration determination mechanism," the aggregation process of the data to be processed can dynamically adapt to the queue status and further form the target file and its file storage path in the source database. This enables a more orderly data processing flow for high-concurrency scenarios and provides a stable foundation for subsequent data synchronization, archiving, and analysis.

[0069] Figure 1 A flowchart illustrating a data processing method provided in this application embodiment. Figure 1 .like Figure 1 As shown in the embodiment of this application, a data processing method is provided, the method comprising:

[0070] S101. In the event of a data change event in the source database, write the pending data associated with the data change event into the message queue and determine the message queue backlog.

[0071] In this embodiment, the source database can be understood as a database entity that receives operational data from the power station and stores structured business records. It can be a relational database or a distributed database with change log output capabilities. Data change events refer to insert, update, and delete operations performed on the energy storage power station's operational data table in the source database, including but not limited to the addition and modification of business data such as real-time voltage, current, SOC, temperature, equipment status, and alarm information. The data to be processed can be, for example, a structured record corresponding one-to-one with a change event, containing key fields such as the power station's unique identifier, equipment number, data acquisition timestamp, data type, specific value, and change operation type, ensuring traceability and verifiability in subsequent aggregation, disk storage, and warehousing processes.

[0072] A message queue can be understood as an asynchronous buffer component located between the data producer and the data consumer. It is used to smooth out the peaks and valleys of data streams generated concurrently by multiple power plants, preventing data from directly flooding into processing nodes and causing impact. Specifically, middleware such as Kafka, RabbitMQ, RocketMQ, or others with queue backlog statistics capabilities can be used. The message queue backlog refers to the total number of messages that have been received in the message queue but have not yet been pulled and processed by the consumer thread. It is used to reflect the current system processing pressure and the degree of imbalance between data inflow and outflow. It can be represented by the number of unconsumed messages, or by the partition offset difference, consumption point lag, total number of message bytes, or a comprehensive backlog index.

[0073] Understandably, the execution entity can be a synchronous processing service deployed in a central data processing server cluster, or a combination module of an event listening service and a message delivery service running in a container platform. Data change event capture in the source database can be accomplished by listening to database logs, such as parsing binlogs in a MySQL (MyStructured Query Language) environment, parsing WAL (Write-Ahead Logging) logs in a PostgreSQL environment, and parsing equivalent change log streams in other database environments; alternatively, database triggers can be configured on the target business table to directly trigger message generation logic after insert, update, or delete actions are committed; or database changes can be uniformly subscribed to through a change data capture component.

[0074] For example, taking the operation monitoring scenario of an energy storage power station as an example, the source database can store fields such as power station ID (Identifier), equipment number, sampling time, voltage, current, state of charge, temperature, alarm code and operation mode. When new data uploaded by the data acquisition unit on the power station side is written into the source database, a corresponding insertion event is generated in the database log. The event listening service identifies the affected data table and record range based on this and extracts the data to be processed associated with the event.

[0075] Optionally, to ensure the traceability and aggregability of data written to the message queue, the data to be processed can be encapsulated into a standard message body before being written. This message body may include a business identifier field, a data source field, an event type field, an event occurrence time field, a data payload field, a data verification field, and an idempotency identifier field. Specifically, the business identifier field may include the power plant ID, area code, equipment number, or measurement point number, used for subsequent grouping and aggregation according to business dimensions; the data source field is used to identify the source database, data table, or acquisition channel; the event type field distinguishes between insertion, update, and deletion; the data payload field can directly carry the serialized structured record; the verification field can be a CRC (Cyclic Redundancy Check) value, a digest value, or a hash value, used for verification at the receiving end; and the idempotency identifier field can be generated from the table name, primary key, event timestamp, and version number to prevent duplicate consumption and duplicate writing.

[0076] In one possible implementation, the message body is serialized in JSON (JavaScript Object Notation) format and then delivered to the message queue; in another implementation, the message body uses Avro (Apache Avro), Protobuf (Protocol Buffers), or other compact binary encoding formats to reduce network transmission and storage overhead.

[0077] During message delivery, a synchronous delivery mode can be used to confirm whether the message has successfully entered the message queue, or an asynchronous delivery mode combined with callback confirmation can be used to improve throughput. To reduce the coupling between the database change monitoring module and the message middleware, the data to be processed can be written to a local buffer, memory queue, or intermediate buffer table first, and then written to the message queue in batches by an independent delivery thread.

[0078] For high-concurrency scenarios requiring sequential ordering, partitioning and routing can be performed based on power plant ID, equipment number, or target table primary key. This ensures that data from the same business entity enters the same partition, maintaining local temporal consistency during the subsequent aggregation phase. For exceptional situations, such as message delivery failures, network jitter, or queue service unavailability, failure retries, dead-letter transfer, breakpoint resumption, and local persistence compensation mechanisms can be enabled to prevent the loss of source database change events due to temporary failures.

[0079] Specifically, after writing a message, it is necessary to determine the message queue backlog. For example, the latest write and consumption positions of each topic and partition can be obtained by calling the message queue management interface, and the difference between the two can be summed to obtain the current total backlog of messages. Alternatively, the backlog can be estimated by statistically analyzing the difference between the production rate and consumption rate over a certain period, combined with the current number of unconfirmed messages. Another method is to report consumption receipts on the consumer side and determine the message queue backlog status based on the number of missing receipts.

[0080] In one possible implementation, if the message queue contains multiple topics, the system can only count the backlog of the topic corresponding to the current data processing task. In another implementation, to avoid a single value being insufficient to reflect the true load, the number of message bytes, message dwell time, and maximum partition lag value can be collected simultaneously and normalized to form a unified backlog metric.

[0081] Based on the above analysis, this step establishes a standardized message receiving link around the source database change event and obtains the backlog status of the message queue in real time. This enables the subsequent data aggregation link to be based on quantifiable load awareness, thereby avoiding the problem of excessively frequent triggering or consumption lag caused by continuing to use a fixed rhythm processing when there is high concurrency access. This improves the adaptability of the entire link to sudden traffic and multi-power station concurrent scenarios.

[0082] S102. Determine the open duration of the data aggregation window based on the message queue backlog.

[0083] In this embodiment, the open duration of the data aggregation window refers to the total time from the start of data collection to the cessation of collection, preparation for generating the target file, and execution of the disk write operation. The function of the data aggregation window is to establish a dynamic balance between processing granularity, file generation frequency, and system resource consumption, so that the data to be processed can form a batch data set of a certain scale within a suitable time range.

[0084] Understandably, message queue backlog, as a core input parameter, represents the total number of messages written to the message queue but not yet pulled, parsed, and processed by the consumer threads. It reflects the degree of matching between the current system's data inflow rate and processing capacity. When the backlog is small, it indicates a light system load and smooth data consumption; when the backlog continues to increase, it indicates that the inflow rate exceeds the consumption rate, and the system is under increasing pressure. Dynamically adjusting the window duration based on this indicator allows the data aggregation strategy to maintain adaptive coordination with the system's real-time load, avoiding resource waste or write congestion caused by a fixed window being too long during off-peak periods or too short during peak periods.

[0085] The window duration calculation model can employ nonlinear functions, such as logarithmic functions, square root functions, or sigmoid growth functions, to create a smooth, non-aggressive adaptive relationship between the window length and message backlog. Nonlinear calculation can quickly converge the window duration when there is low backlog, improving the real-time performance of data synchronization; and gently lengthen the window duration when there is high backlog, reducing the write frequency and alleviating database pressure.

[0086] Specifically, the system can use preset formulas for calculation: .in, The duration for which the currently calculated data aggregation window remains open; The baseline window duration (preferably 10 minutes) is used to ensure the basic aggregation cycle under low load and to avoid excessively frequent aggregation. The adjustment coefficient (preferably 5) is used to control the magnitude of the window change with the amount of stacking. This refers to the real-time message queue backlog, which is the total number of unconsumed messages in the message queue. A reference stacking threshold (preferably 1000 rows) is used to normalize the stacking amount, ensuring the formula can stably adapt to different business scales. Using a base-10 logarithmic operation enables non-linear flexible adjustment, guaranteeing smooth window changes without drastic fluctuations. This formula achieves flexible matching between load and window size through a logarithmic relationship, avoiding drastic window fluctuations caused by sudden changes in stacking amount and improving system stability.

[0087] When the accumulation is low When the value is negative, the window duration is automatically shortened, reducing invalid aggregations and improving real-time performance; when the backlog increases, The window size gradually increases, and the window duration is extended gently to avoid high-frequency writes impacting the database. When the logarithmic property causes the backlog to increase exponentially, the window size only increases slightly, and the excessive window size will not cause synchronization delays to exceed the limit.

[0088] For example, read the current number of unconsumed messages Q from the message queue monitoring interface; read system configuration parameters: , , .

[0089] Secondly, calculate the normalized packing ratio, and calculate... The original stacking volume is converted into a relative load level, and a logarithmic operation is performed, taking the base-10 logarithm of the above ratio: .

[0090] Subsequently, the calculation window adjusts the increment. Adjusting the increment To obtain the final window opening duration .

[0091] Optionally, the boundary constraint calculation results can be limited to a reasonable range, for example: To avoid extreme values ​​affecting system stability.

[0092] In another optional embodiment, a window duration calculation model can be pre-configured in the data processing service. This calculation model can employ either a segmented threshold mapping method or a continuous function mapping method. Under the segmented threshold mapping method, multiple backlog intervals can be set. For example, when the message queue backlog is below a first threshold, the open duration of the data aggregation window is set to a shorter duration to reduce waiting latency under low load conditions; when the message queue backlog is between the first and second thresholds, the open duration is set to a medium duration to balance real-time performance and batch processing; when the message queue backlog is above the second threshold, the open duration is extended to a longer duration to absorb more data to be processed in a single aggregation, reducing resource consumption caused by frequent small file generation and high-frequency downstream write triggering. For example, the backlog can be divided into three intervals: "0 to 1000 records," "1001 to 10000 records," and "more than 10001 records," corresponding to window durations of "2 seconds," "5 seconds," and "10 seconds," respectively. The advantages of this approach are that it is simple to implement, the strategy is clear, and it is easy for operation and maintenance personnel to configure parameters based on their experience with business load.

[0093] Furthermore, to ensure that the window duration more accurately reflects the system's actual carrying capacity, message queue backlog can be incorporated into a comprehensive calculation model along with system operating parameters such as CPU (Central Processing Unit) utilization, memory utilization, disk write throughput, network transmission bandwidth utilization, and historical peak fluctuation coefficient. For example, when system resources are idle but the backlog is high, the window can be appropriately extended to increase the amount of data processed in a single batch; when the backlog is high but the CPU is nearing its processing limit, the window growth rate can be limited to prevent excessive single-batch aggregation from causing memory bloat, excessively long sorting time, or disk write timeouts.

[0094] In another possible implementation, a short-term trend prediction model can be established based on the historical backlog sequence to predict the possible trend of the backlog continuing to increase or gradually decreasing in the next time slice. Based on this, the opening duration of the data aggregation window can be adjusted in advance, so that the window adjustment changes from static response to trend-aware adjustment.

[0095] It should be noted that after determining the window duration, the system can write the current window parameters into the context control information. This control information may include the window start time, window end time, target topic, target partition range, maximum number of consumed records, allowed memory usage threshold, and file scrolling conditions, to ensure that the subsequent aggregation process has clear boundaries.

[0096] Meanwhile, to prevent abnormal configurations from causing windows to be too short or too long, minimum and maximum open duration boundaries can be set. For example, the minimum value is used to ensure batch efficiency, and the maximum value is used to control the upper limit of processing latency. For scenarios where multiple tenants or multiple business lines share message queues, different window duration strategies can also be configured for different business priorities. For example, a shorter window can be used for real-time alarm-related data, while a relatively longer window can be used for archived historical operation data.

[0097] Based on the above analysis, this step, by using the message queue backlog as the basis for adjusting the window duration, enables the system to shorten waiting times and reduce unnecessary delays under low load, while increasing the single aggregation capacity and reducing the overhead of frequent file generation and frequent write triggers under high load, thus achieving a dynamic balance between real-time performance and resource utilization. Furthermore, this approach ensures that the file generation rhythm is consistent with the message backlog status, reducing the risk of continuously amplified message queue backlog, frequent fluctuations at the consumer end, and disordered disk write timing, providing a more stable processing foundation for high-concurrency energy storage power station data access scenarios.

[0098] S103. During the open time of the data aggregation window, aggregate and process the data to be processed, generate the target file and the file storage path of the target file in the source database, and store the target file to the file storage path.

[0099] In this embodiment, the aggregation process can be understood as continuously pulling data to be processed from the message queue after the window is opened, and summarizing, filtering, sorting, deduplicating, formatting, and building files according to predetermined rules; the target file can be understood as a data file formed within a data aggregation window that can be archived, shared, or loaded by a downstream analysis system; the file storage path can be understood as the physical or logical storage location identifier corresponding to the target file, which can be a local file system path, a distributed file system path, an object storage path, or a network shared directory path; the file storage path of the target file in the source database indicates that the path information is simultaneously registered, mapped, or associated with the path field, index table, metadata table, or file mapping relationship table in the source database for subsequent querying, tracing, and verification.

[0100] Understandably, at the start of the window opening, the aggregation processing module initiates a message consumption task, pulling data to be processed from the message queue in batches. Consumption can be either long polling or streaming subscription. For each batch of data, the system first performs deserialization and basic verification. Verification may include checking message structure integrity, the existence of required fields, the validity of timestamps, the validity of business identifiers, and the consistency of checksums. After successful verification, the data is written to a memory buffer or temporary aggregation container according to preset aggregation dimensions. Aggregation dimensions can be determined based on business needs, such as organizing data according to source database name, table name, power plant ID, equipment type, sampling period, window time range, or region code, thereby centralizing similar datasets into one or more target files.

[0101] In one optional embodiment, during the aggregation process, to ensure the consistency and availability of the target file content, deduplication and sorting can be performed on multiple change records from the same business primary key within the window. If multiple update records exist for the same primary key within the window, the system can retain the latest version data or retain the full change trajectory in chronological order of events, depending on whether the downstream analysis scenario requires a state snapshot or a change history. For duplicate messages, duplicate detection and filtering can be performed based on the idempotency flag field. For deletion-type events, a deletion flag, operation type field, or tombstone record can be written to the target file so that the downstream can correctly identify the business semantics.

[0102] Furthermore, during the aggregation processing stage, the number of data entries, time span, field summary values, and file-level hash values ​​can be calculated and this information can be synchronously saved as metadata of the target file to support subsequent integrity verification and fast retrieval.

[0103] The format of the generated target file can be set according to subsequent processing requirements. In one possible embodiment, if it is for cross-system sharing or interface import, the target file can be generated into a JSON file or a CSV (Comma-Separated Values) file for fast parsing by general systems; if it is for large-scale analytical processing, the target file can be generated into a columnar file such as Parquet or ORC (Optimized Row Columnar) to improve compression ratio and query efficiency; if it is for archive storage, compression operations can be performed after the file is generated, for example, using GZIP (GNU Zip), Snappy, or LZ4 (Lempel-Ziv 4) compression algorithms.

[0104] File names can be constructed according to business rules, such as including fields like source database identifier, target table identifier, window start time, window end time, power plant ID, batch serial number, and file digest code. This naming convention ensures that data files from the same source and within the same window are clearly identifiable and facilitates subsequent retrieval by time and business scope.

[0105] While aggregating data, the system generates the file storage path of the target file in the source database according to preset structured rules. This embodiment adopts a three-level directory structure: / {database name} / {table name} / {timestamp}_{business feature code}.file format. Here, the database name and table name correspond to the logical space to which the data belongs in the source database, the timestamp is the closing time of the current data aggregation window, and the business feature code is the power plant ID, equipment number, or regional code, used to uniquely identify the data source, achieving automatic isolation and classification of data from multiple power plants, avoiding data confusion, and facilitating subsequent retrieval, backtracking, verification, and cleanup. The system can pre-build a file index table in the source database, which records information such as the file's unique identifier, the business table it belongs to, the window start and end times, the file path, the file size, the number of records, the file summary, and the generation time. After the target file is generated, the aggregation processing module writes the path corresponding to the file into the file index table, or updates the path field of the associated business batch record in the source database, so that the source database can directly reflect the file's disk location. In this way, subsequent archiving services, data verification services, or analysis loading services can quickly locate the target file by querying the source database, without having to search through the scattered directories level by level.

[0106] Optionally, when storing the target file to the file storage path, a temporary file can be written first, followed by atomic renaming to reduce the risk of interruption. Specifically, the system first creates a temporary file with a temporary extension in the target directory, writes the aggregated data stream to this temporary file, and completes the flushing to disk. After verifying that the file size, number of records, and digest value are correct, the temporary file is renamed to the official filename. If a service exception, insufficient disk space, or network interruption occurs during the writing process, the temporary state is retained and compensation logic is triggered to avoid generating a corrupted file with partial writes. For object storage scenarios, the file can be uploaded to a temporary object key first, and then an equivalent atomic publish can be achieved by copying and switching the official object key. After the file is successfully written to disk, the system submits a consumption point to the message queue or sends an acknowledgment receipt, indicating that the corresponding message in the window has been successfully processed; if the file writing fails, no acknowledgment is submitted, and the system waits for a retry or for the compensation task to reprocess the message to ensure consistency between message processing and file writing to disk.

[0107] After data aggregation is complete and the window closes, the system writes the generated target file to the aforementioned file storage path, completing data persistence to disk. The persisted file can be stably stored in the local file system or distributed storage. Even in the event of service restarts, network fluctuations, or processing anomalies, the data will not be lost, providing a reliable basis for anomaly retries, data replenishment, and consistency checks. Simultaneously, the system records metadata information such as file path, generation time, number of data entries, and checksum values ​​for subsequent loading status management and idempotent control, preventing duplicate loading and data inconsistencies.

[0108] In the context of energy storage power station data access scenarios, the above embodiments can be deployed in a processing architecture consisting of a data acquisition layer, a message transmission layer, and a data processing layer. The power station continuously sends up operational data, while the central side generates a data stream to be processed based on data changes in the source database. This data is then received by a message queue, and the aggregation window is adjusted according to the backlog status, thereby making the file generation rhythm more compatible with the business load.

[0109] The data processing method provided in this application uses source database change events as the starting point, message queues as asynchronous buffers, and message queue backlog as the basis for adjusting window duration. Within a controlled window, it aggregates the data to be processed and writes structured files to disk, enabling the data processing chain to dynamically adjust its processing rhythm according to the actual load. This avoids resource waste caused by frequently generating small files in low-load scenarios and alleviates the consumption lag problem caused by a fixed window's inability to process backlogged data in a timely manner in high-load scenarios. Furthermore, by stably associating the target file path with the source database, it improves the feasibility and consistency of subsequent file retrieval, data traceability, and archiving management.

[0110] It should be understood that the above examples are merely illustrative and not limiting. Any implementation method that adjusts the data aggregation window based on the message queue backlog status and completes the generation of aggregation files and path storage within the window can be considered an equivalent implementation of the embodiments of this application.

[0111] Figure 2 A flowchart illustrating a data processing method provided in this application embodiment. Figure 2 .like Figure 2 As shown, the embodiments of this application are in Figure 1 Based on the embodiments, a data processing method is further described in detail; the method includes:

[0112] S201. In the event of a data change event in the source database, write the pending data associated with the data change event into the message queue and determine the message queue backlog.

[0113] Step S201 is similar to step S101 above, and will not be described again here.

[0114] In one possible implementation, data change events are monitored by deploying a listening component in the source database; the listening component monitors log data in the source database to determine whether a data change event has occurred.

[0115] The listening component can be a log subscriber, change capture program, or database agent module deployed on the server where the source database instance is located. It is used to continuously read the log data generated by the source database and identify marker information related to data state changes, such as insertion, update, deletion, and transaction commit, from the log records.

[0116] The log data in the source database can be pre-write logs, binary logs, or archived logs. The monitoring component performs sequential scanning, incremental parsing, and offset memorization of the log data to ensure continuous tracking of change records. When a log entry that meets the preset change characteristics is detected, it determines that the corresponding data change event has occurred.

[0117] Understandably, a dedicated data change monitoring component can be deployed within the runtime environment of the source database. This monitoring component runs as an independent process or service, possessing characteristics of high availability, high fault tolerance, and low resource consumption. It can achieve continuous awareness of data change behavior without requiring any modifications to the source database's business code, table structure, access permissions, or upper-layer application systems. This monitoring component can be accessed, for example, through an interface provided by the database, or by directly accessing log files. In practical applications, other models or implementation forms can also be selected, and this application embodiment does not limit this.

[0118] During operation, once the monitoring component is deployed and started, it can connect to the source database in real time via a subscription method, continuously capturing and parsing incremental log data generated by the database during operation, and determining whether data change events have occurred based on the log content. The source database log data includes, but is not limited to, native incremental logs from databases such as MySQL binlog, PostgreSQL WAL logs, and Oracle redo / undo logs. The monitoring component accurately identifies data change events such as inserts, updates, and deletions in the energy storage power station's operating data tables by parsing and matching key information in the logs, including operation type, operation timestamp, database name, table name, primary key of data rows, field values ​​before and after modification, and transaction status.

[0119] When a data change that conforms to preset rules is detected, the listening component immediately encapsulates the change event into a standard structured message and extracts all business data associated with the event to form a pending data object that can be processed by subsequent processes.

[0120] This monitoring method effectively avoids problems such as data loss, poor real-time performance, and excessive pressure on the source database caused by repeated queries, which are common in traditional business-side proactive reporting and timed polling. It also provides transaction-level consistency guarantees and can fully capture data changes in complex scenarios such as batch changes and concurrent changes. This provides a stable, reliable, and high-quality data foundation for subsequent message queue writing, dynamic window aggregation, file generation, and batch database entry. It is especially suitable for high-concurrency and real-time smart grid energy storage data synchronization scenarios at the level of hundreds of stations.

[0121] S202, Obtain the rules for calculating the window open duration.

[0122] Among them, in the window open duration calculation rules, the calculated open duration and the message queue backlog have a non-linear mapping relationship.

[0123] S203. Determine the open duration of the data aggregation window based on the message queue backlog and window open duration calculation rules.

[0124] In this implementation, the window opening duration calculation rule is a pre-set adaptive scheduling strategy. Its core design principle is to avoid the rigidity inherent in traditional methods such as fixed windows and linear mapping, allowing the data aggregation rhythm to flexibly adapt to system load. The non-linear mapping relationship means that the opening duration of the data aggregation window does not change proportionally with the increase in message queue backlog. Instead, it responds quickly and moderately shortens the window duration when the backlog is low, while increasing it gradually and moderately when the backlog is high. This achieves the effect of improving real-time performance under low load and protecting system stability under high load.

[0125] In a specific implementation, the data processing device can periodically read the current backlog at the message queue consumer and input this backlog into the calculation module corresponding to the window open duration calculation rule. This calculation module can be implemented by a processor executing a duration calculation program stored in memory, or by a rule engine deployed in the policy service. The rule engine outputs the open duration based on a preset nonlinear function relationship. The output result can be further passed to the aggregation module as a data aggregation control parameter. The aggregation module continuously receives data to be processed and generates target files within the open duration. To adapt to different business environments, the calculation rules can be pre-written into a configuration file or policy center, and in practical applications, parameter calibration can be performed based on the message queue's throughput, partition size, and business peak. In practical applications, other models of this component can also be selected; this embodiment does not limit this.

[0126] Specifically, the system first obtains the backlog amount representing queue pressure, and then converts the pressure state into an aggregation window duration based on preset non-linear rules, allowing the window length to be dynamically adjusted according to load changes. Because the calculated open duration and backlog amount have a non-linear relationship, the system can shorten the aggregation window under low load to improve processing timeliness, and extend the aggregation window under high load to increase the amount of data aggregated in a single operation, thereby reducing the resource overhead caused by frequent aggregation triggers and alleviating consumption lag caused by continuous message backlog. During calculation, the system reads the current message queue backlog amount in real time, uses it as a core input parameter, and substitutes it into the calculation rules to obtain the data aggregation window open duration adapted to the current load state.

[0127] By adopting this specific implementation method, the open duration of the data aggregation window can adaptively change with the message queue backlog status, reducing resource waste and processing jitter caused by the fixed window strategy, improving the data aggregation efficiency and stability in high-concurrency scenarios, and enhancing the continuity and controllability of subsequent file generation and disk writing processes.

[0128] S204. During the open time of the data aggregation window, aggregate the data to be processed based on the preset encoding rules to generate the target file.

[0129] S205. Based on the preset file format, the database name of the source database, and the table name, timestamp, and business feature code associated with the data to be processed, generate the file storage path of the target file in the source database.

[0130] S206. Store the target file to the file storage path.

[0131] The timestamp represents the end time of the data aggregation window. Preset encoding rules are used to unify the data structure, field order, numerical format, and missing value imputation strategy, ensuring that data from different energy storage power stations, different acquisition devices, and different time segments can be standardized into a consistent, parsable, and batch-importable standard structure. The target file can be a text file, CSV file, JSON file, or other structured file format suitable for batch loading. In practical applications, other file formats can also be selected; this application embodiment does not limit this.

[0132] The source database name is used to identify the data source system to which the target file belongs, the table name is used to represent the business affiliation of the data to be processed, the business feature code is used to distinguish data sets from different power plants, different acquisition tasks or different business themes, and the timestamp is taken from the end time of the data aggregation window to ensure that the path name corresponds one-to-one with the aggregation batch.

[0133] In the specific implementation, when the data to be processed enters the aggregation processing stage, the system first serializes and concatenates each field in the data to be processed according to the preset encoding rules, and verifies and writes the encoded content to form a target file that can be directly used for disk storage or transmission.

[0134] Subsequently, when generating the file storage path, the system adopts a hierarchical structured path rule, using the database name and table name of the source database as the first and second-level directories, and the timestamp corresponding to the end time of the data aggregation window and the business feature code used to distinguish the data source as the unique identifier of the file, combining them to form a standardized and traceable storage path. The business feature code includes, but is not limited to, power plant ID, equipment number, and area code, used to achieve automatic isolation of multi-source data and avoid confusion between data from different sources and time periods. The timestamp strictly adopts the end time of the data aggregation window, accurately corresponding to the aggregation cycle, facilitating subsequent data backtracking, verification, cleanup, and duplicate loading checks. This file storage path can be written to the path field in the source database or the file index table for subsequent retrieval, backtracking, and batch loading tasks.

[0135] This approach ensures consistency in batch data output through encoding rules and guarantees the correspondence between files, data sources, business themes, and aggregation times through path rules, thereby making target file generation and storage management deterministic and traceable. Since the timestamp is taken from the end time of the aggregation window, each target file corresponds to a specific window batch, facilitating subsequent batch loading, verification, and replay. Adopting this approach reduces the probability of file naming conflicts and path confusion, improves batch archiving efficiency, and enhances the stability and maintainability of file management on the source database side.

[0136] S207. After the preset delay time is closed in the data aggregation window, the dynamic window is launched.

[0137] S208. During the open duration of the dynamic window, load the target files stored in the source database into the target database in batches.

[0138] The preset delay duration is a fixed buffer interval pre-configured by the system. Its function is to achieve peak-shifting scheduling, proactively avoiding data aggregation, file generation, and high-concurrency runtime periods of business systems. This prevents batch loading tasks from competing for CPU, memory, network I / O, and database connection resources with front-end data access, source database writing, and message queue consumption, thereby significantly reducing write conflicts and the probability of loading failures. This delay mechanism upgrades the data processing flow from "real-time execution" to "business-time-aware execution," greatly improving the system's stability in high-concurrency scenarios.

[0139] The dynamic loading window is a dedicated time interval for performing batch import tasks. Its open duration can be adaptively set according to the target database load, the number of files, and the data size, ensuring a smooth and controllable batch loading process. After the dynamic loading window is opened, the system scans the target files that have been written to disk sequentially according to the file storage path, reads and parses them in the order they were generated, and calls the batch import interface provided by the target database to perform data loading, fully leveraging the performance advantages of analytical databases for batch writing.

[0140] Specifically, target files in the source database are typically stored in a file storage path corresponding to business characteristics according to a preset file format. The target database, on the other hand, is used to hold data copies required for subsequent queries, archiving, or analysis. After detecting that the data aggregation window is closed, the system first sets a preset delay time to allow for file handle release, path verification, and file integrity checks. Then, it sends a start command to the loading module, opening the dynamic window. Within this open time, the loading module reads the target files in batches, converts the file content into data records recognizable by the target database, and then performs the write operation, thus avoiding the frequent commit overhead of writing each file line by line.

[0141] Optionally, the target analytical database can be a columnar storage, big data analysis database that supports batch file import, such as ClickHouse, Apache Doris, StarRocks, Hive, Greenplum, etc., which has high-throughput batch writing, massive data storage and fast multidimensional analysis capabilities to adapt to the scenarios of archiving, statistical analysis and scheduling decision calculation of massive operation data of energy storage power stations. The specific database type is not limited in this application.

[0142] Understandably, to improve loading stability, the batch loading process can employ fixed batch size, fixed time slice, or adaptive batch threshold control methods to ensure that multiple target files within the same dynamic window can be continuously loaded into the target database. When receiving batch loading requests, the target database can perform atomic commits based on transaction mechanisms or reduce the impact of single loading failures using segmented commit methods, thereby ensuring that target files stored in the source database can be migrated to the target database in an orderly manner within the dynamic window.

[0143] For example, assuming the system detects a message queue backlog of Q = 5000 messages, and the data aggregation window's open duration is calculated using a non-linear formula to be 13.5 minutes, with the window opening at 17:00, then the data aggregation window will close at 17:13:30. In this embodiment, the preset delay time is set to 1 minute, so the system does not immediately execute loading after the window closes, but waits until 17:14:30 before starting the dynamic loading window.

[0144] During the dynamic loading window's open duration, the system reads the target file already generated in the file storage path, for example: / run-data / config-station-run-data / 20251121171330-sta01.json. Then, through the target database's batch loading interface, this file is loaded into the corresponding data table in the target database all at once. This configuration allows the batch loading task to avoid peak periods for power plant data uploads and source database writes, effectively reducing CPU, network, and database resource contention, and significantly improving write success rate and overall system stability.

[0145] In this way, the target file will not be immediately entered into the target database after the file storage is completed. Instead, it will be loaded in a centralized manner through a dynamic window after a preset delay. This ensures an orderly connection between file writing to disk, index stability and subsequent database entry, thereby reducing write jitter in high-concurrency scenarios, improving batch migration efficiency, and enhancing the data consistency of the target database and the overall processing stability of the system.

[0146] In one possible implementation, loading target files stored in the source database into the target database in batches includes:

[0147] The target files stored in the source database are imported in batches into the target directory deployed in the target database.

[0148] If an import fails during the batch import process, an alarm will be triggered, and the target database will be controlled to re-execute the batch import process.

[0149] The target database is an analytical database designed for high-concurrency analysis of massive amounts of data. It supports large-scale batch loading and high-throughput writing, fully leveraging the performance advantages of file-level batch import. The target directory is a logical storage unit within the target database, divided according to business dimensions. It is used to classify and manage data by database, table, time period, or business source, ensuring a clear data structure after import, facilitating querying and statistical analysis. The batch import process employs streaming loading, transaction-level commit, or sharded parallel loading to improve write efficiency and system stability. The target file refers to the structured file formed by the source database after data aggregation, typically stored in CSV, TXT (Text), JSON, or binary archive formats.

[0150] Understandably, the target database can read the target file list corresponding to the source database through the file transfer component and write each target file to the target directory according to the preset import order. During the import process, file integrity, access permissions, disk space and file lock status can be verified to reduce the risk of import interruption due to insufficient resources or abnormal paths.

[0151] During the import process, the system continuously collects import status, file processing progress, error codes, and exception information, and performs real-time verification of the integrity, format validity, and data consistency of the target file. When import failure occurs due to network interruption, service jitter, database rate limiting, insufficient disk space, or format errors, the system immediately generates an alarm message containing the path of the failed file, the reason for the failure, the time of occurrence, and the business characteristic code, and pushes it to the operations and maintenance side through preset channels for rapid location and handling.

[0152] When an import failure is detected, an alarm can be displayed through the database management interface, message notification service, or log alarm module to inform operations and maintenance personnel of the current batch import anomaly. Simultaneously, the system can automatically trigger a retry mechanism to re-initiate the batch import process for the unsuccessfully imported target files. The target database can re-initiate the batch import based on the identifier of the failed files, retrying only for those failed files. Alternatively, when batch consistency requirements are high, the entire batch of target files can be re-imported to ensure that the file set in the target directory is consistent with the target file set in the source database. Furthermore, the retry process supports strategies such as limiting the number of attempts and exponential backoff to avoid infinite loop imports under abnormal conditions.

[0153] Through the above method, the target files can form a stable batch migration relationship between the source database and the target database. When import fails, timely alerts and automatic retries can be initiated, thereby improving the reliability and continuity of file loading, reducing the frequency of manual intervention, and enhancing the stability of subsequent data querying, archiving, and synchronization processing. The target directory, file format, and retry strategy described above can be configured in other ways in practical applications; this embodiment does not limit these options.

[0154] In one optional embodiment, for example, a multi-energy storage power station network scheduling and monitoring platform is implemented. The platform connects dozens to hundreds of distributed energy storage power stations, collecting real-time operating parameters such as voltage, current, state of charge, equipment temperature, and alarm status of each power station. The source database uses MySQL, and a Binlog log listening component is deployed to capture data change events such as data table additions and updates in a non-intrusive manner in real time. After parsing and extracting the data to be processed, it is written to the RabbitMQ message queue, and the message queue backlog is counted in real time.

[0155] The system dynamically and adaptively determines the open duration of the data aggregation window based on the real-time accumulation volume according to the preset nonlinear calculation rules. Within the window period, the system aggregates the data to be processed according to the unified coding rules, generates a standard format target file, and generates a standardized file storage path by combining the database name, table name, window end timestamp and power plant business feature code, thus completing the file persistence to disk.

[0156] After the data aggregation window is closed, and after a preset delay to avoid peak business periods, the dynamic loading window is launched to batch load target files from the storage path into the target analytical database, such as StarRocks. The batch import process is monitored in real time, and alarms are automatically triggered and a retry mechanism is initiated in case of abnormal failures.

[0157] This application provides a data processing method that captures data changes through non-intrusive monitoring of incremental database logs, dynamically calculates the aggregation window duration using a non-linear formula based on message queue backlog, and generates structured target files and standardized storage paths according to database name, table name, timestamp, and business feature code to achieve automatic isolation of multi-source data. After the aggregation window is closed, a preset delay is allowed before peak-shifting batch loading begins, using an analytical database batch interface for writing, and featuring alarm and automatic retry capabilities. This solution can adapt to system load, reduce resource consumption, improve write efficiency and stability, ensure reliable data without loss, and meet the high-concurrency, high-real-time data synchronization requirements of power grid energy storage at the level of hundreds of stations.

[0158] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0159] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this data processing apparatus is similar to the implementation scheme described in the data processing method above. Therefore, the specific limitations in one or more apparatus embodiments provided below can be found in the limitations of the data processing method above, and will not be repeated here.

[0160] In one embodiment, such as Figure 3 As shown in the embodiment of this application, a data processing apparatus 300 is provided, which includes:

[0161] The writing module 301 is used to write the data to be processed associated with the data change event into the message queue when a data change event occurs in the source database, and to determine the message queue backlog.

[0162] Module 302 is used to determine the open duration of the data aggregation window based on the message queue backlog.

[0163] The generation module 303 is used to aggregate and process the data to be processed within the open time of the data aggregation window, generate the target file and the file storage path of the target file in the source database, and store the target file to the file storage path.

[0164] like Figure 4 As shown, in one possible implementation, the data processing device further includes a monitoring module 304;

[0165] The monitoring module is configured as follows:

[0166] Deploy the listening component in the source database;

[0167] By using a listening component, log data in the source database is monitored to determine whether any data change events have occurred.

[0168] In one possible implementation, the determining module 302 is further configured to:

[0169] Obtain the window open duration calculation rules; in the window open duration calculation rules, there is a non-linear mapping relationship between the calculated open duration and the message queue backlog;

[0170] The open duration of the data aggregation window is determined based on the rules for calculating message queue backlog and window open duration.

[0171] In one possible implementation, the generation module 303 is further configured to:

[0172] The data to be processed is aggregated based on preset encoding rules to generate target files;

[0173] Based on the preset file format, the database name of the source database, the table name associated with the data to be processed, the timestamp, and the business feature code, the file storage path of the target file in the source database is generated; the timestamp is the end time of the data aggregation window.

[0174] like Figure 5 As shown, in one possible implementation, the data processing device further includes: a startup module 305 and a loading module 306;

[0175] The startup module 305 is configured as follows:

[0176] The dynamic window is launched after the data aggregation window has been closed for a preset delay.

[0177] The loading module 306 is configured as follows:

[0178] During the open duration of the dynamic window, target files stored in the source database are loaded into the target database in batches.

[0179] In one possible implementation, the loading module 306 is further configured to:

[0180] The target files stored in the source database are imported in batches into the target directory deployed in the target database.

[0181] If an import fails during the batch import process, an alarm will be triggered, and the target database will be controlled to re-execute the batch import process.

[0182] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 400 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0183] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0184] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0185] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0186] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0187] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0188] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0189] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0190] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0191] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0192] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0195] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0197] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method, characterized by, The method includes: In the event of a data change event in the source database, the data to be processed associated with the data change event is written to the message queue, and the message queue backlog is determined. Based on the message queue backlog, determine the open duration of the data aggregation window; During the open duration of the data aggregation window, the data to be processed is aggregated and processed to generate a target file and the file storage path of the target file in the source database, and the target file is stored in the file storage path.

2. The method of claim 1, wherein, The data change events are monitored in the following ways: Deploy a listening component in the source database; The monitoring component monitors log data in the source database to determine whether a data change event has occurred.

3. The method of claim 1, wherein, Based on the message queue backlog, the open duration of the data aggregation window is determined, including: Obtain the window open duration calculation rule; in the window open duration calculation rule, the calculated open duration and the message queue backlog have a non-linear mapping relationship; The open duration of the data aggregation window is determined based on the message queue backlog and the window open duration calculation rules.

4. The method of claim 1, wherein, The aggregation process, which generates a target file and its storage path in the source database, includes: The data to be processed is aggregated based on preset encoding rules to generate the target file; Based on the preset file format, the database name of the source database, and the table name, timestamp, and business feature code associated with the data to be processed, the file storage path of the target file in the source database is generated; the timestamp is the end time of the data aggregation window.

5. The method of claim 1, wherein, After storing the target file to the file storage path, the method further includes: After a preset delay period after the data aggregation window is closed, the dynamic window is launched; During the open duration of the dynamic window, the target files stored in the source database are loaded into the target database in batches.

6. The method of claim 5, wherein, The step of batch loading the target files stored in the source database into the target database includes: The target files stored in the source database are imported in batches into the target directory deployed in the target database. If an import fails during the batch import process, an alarm is triggered, and the target database is controlled to re-execute the batch import process.

7. A data processing apparatus, characterized by The device includes: The writing module is used to write the data to be processed associated with the data change event into the message queue when a data change event occurs in the source database, and to determine the message queue backlog of the message queue. The determination module is used to determine the open duration of the data aggregation window based on the message queue backlog. The generation module is used to aggregate and process the data to be processed within the open duration of the data aggregation window, generate a target file and the file storage path of the target file in the source database, and store the target file in the file storage path.

8. An electronic device, comprising: include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the steps of the method described in claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images