A real-time data acquisition buffer throughput management method and device and electronic equipment

Abstract

The application provides a real-time data acquisition buffer throughput management method and device and electronic equipment, and relates to the field of data processing. Data objects containing sampling values, time stamps, measurement point numbers and state markers are received through an input memory queue, and a logical mapping identifier is constructed by a mapping agent unit to generate a logical write state; the logical write state is stored in a delay-aware buffer, and a distributed delay monitoring unit listens to a transaction confirmation delay sequence returned by a target database; a time regression model is established based on the delay sequence to generate a prediction projection window of transaction confirmation time; when the window meets a confidence threshold condition, the logical write state is released to an output memory queue; finally, a completion marker table is generated by writing the database through an output thread, and feedback is provided for delay buffer dynamic management. The technical solution provided by the application can coordinate the buffer data release rhythm and the database processing capacity, thereby improving the overall throughput efficiency.

Description

Technical Field

[0001] This application relates to the technical field of data processing, specifically to a real-time data acquisition buffer throughput management method, apparatus, and electronic device. Background Technology

[0002] In high-frequency real-time data acquisition systems, interface programs typically handle the task of transmitting large volumes of continuously acquired data streams from edge acquisition modules to the target database. To ensure data consistency and fault tolerance, target databases in most industrial deployments widely employ distributed consistency protocols, such as Raft or Paxos. These protocols require all write transactions to be replicated, voted on, and confirmed in a strict order among distributed nodes, thereby guaranteeing data consistency and availability. However, this consistency mechanism inherently introduces queuing and distributed confirmation latency during the transaction write process, resulting in transaction confirmation latency significantly higher than the response latency of a single-node system, especially in scenarios with frequent write conflicts, network jitter, or uneven performance among some replica nodes.

[0003] In this type of database architecture, conventional interface programs often employ concurrent release mechanisms to improve write efficiency. This involves immediately batch-distributing cached data to be written after detecting the availability of the output channel. However, existing interface programs generally do not model or provide feedback awareness regarding transaction ordering, replication voting, and consistency confirmation processes in distributed write protocols, neglecting the commit queuing and confirmation bottlenecks on the target database side. When multiple logical write requests are released simultaneously, their physical write transactions experience queuing contention within the database due to sequential commit requirements, resulting in an actual write speed far lower than the data release speed. This mismatch leads to the interface program appearing to have an idle sending channel and unblocked threads, while in reality, write transactions accumulate on the database side, causing released data to be unable to be committed for a long time. This triggers a "throughput reversal" phenomenon, where the interface program's sending thread is idle, while the overall system processing capacity decreases or even stagnates, affecting the stability of the real-time data path.

[0004] Therefore, in current database write scenarios controlled by distributed consistency protocols, it is urgent to solve the problem of how to coordinate the release rhythm of buffered data with the database processing capacity to improve the overall throughput efficiency. Summary of the Invention

[0005] This application provides a real-time data acquisition buffer throughput management method, device, and electronic device, which facilitates the coordination of buffer data release rhythm and database processing capabilities, thereby improving overall throughput efficiency.

[0006] The first aspect of this application provides a real-time data acquisition buffer throughput management method, the method comprising: acquiring a real-time data stream and constructing a data object sequence containing original acquisition values, sampling time, measurement point number, and status marker; writing the data object sequence into an input memory queue; inputting each data object in the input memory queue into a mapping proxy unit; constructing a logical mapping identifier based on the current system clock state and the disk buffer write pointer; and combining the data object with the logical mapping identifier to form a logical write state, wherein the logical mapping identifier includes a unique key value for the data object, a local write logical sequence number, a logical commit prediction window, and a distributed transaction candidate confirmation chain identifier. The logical write state is stored in a latency-aware buffer. A distributed latency monitoring unit pre-deployed in the latency-aware buffer listens to the distributed transaction confirmation latency sequence returned by the target database. A time regression model is constructed based on the distributed transaction confirmation latency sequence to generate a prediction projection window for the transaction confirmation time of the logical write state. If the prediction projection window meets the confidence threshold condition, the corresponding target logical write state is granted release eligibility and written to the output memory queue. The target logical write state in the output memory queue is sent to the target database through an output thread to generate a completion flag table. The latency-aware buffer is managed based on the completion flag table.

[0007] Optionally, the step of acquiring real-time data streams and constructing a data object sequence containing original acquired values, sampling time, measurement point numbers, and status markers, and writing the data object sequence into an input memory queue, specifically includes: receiving original data segments from edge acquisition devices through an acquisition access unit, the original data segments including sampling trigger signals and original acquired values; generating measurement point numbers based on the physical clock time, communication port number, and device code contained in the sampling trigger signal, and establishing a measurement point identifier mapping table to ensure the uniqueness of the measurement point numbers of the data objects; recording the original acquired values ​​as a data value field, recording the synchronized and calibrated physical clock time as a timestamp field, recording the measurement point number as an identifier field, and recording the communication status or verification flag in the original data segment as a status marker field; combining the data value field, the timestamp field, the identifier field, and the status marker field to construct a data object, and writing the data object into the input memory queue in a first-in-first-out order, wherein each data object in the input memory queue is appended with a write pointer and a logical queue position index.

[0008] Optionally, the step of inputting each data object in the input memory queue to the mapping agent unit, constructing a logical mapping identifier based on the current system clock state and the disk buffer write pointer, and combining the data object with the logical mapping identifier to form a logical write state specifically includes: the mapping agent unit sequentially extracting each data object in the input memory queue, reading the current system master clock to generate a high-precision write request timestamp, and synchronously reading the current write pointer position state maintained by the disk write manager in the disk buffer; generating the logical mapping identifier based on the data attribute fields in the data object, the write request timestamp, and the current write pointer position state; and binding and combining the data object with the logical mapping identifier to construct the logical write state.

[0009] Optionally, storing the logical write state in a latency-aware buffer and monitoring the distributed transaction confirmation latency sequence returned by the target database through a pre-deployed distributed latency monitoring unit in the latency-aware buffer specifically includes: writing the logical write state into the latency-aware buffer, where the latency-aware buffer is a dual-index structure with a time-series index and a status identifier mechanism; attaching a logical release status flag and a transaction confirmation status flag to the logical write state after writing; asynchronously monitoring the transaction commit response returned by the target database through the distributed latency monitoring unit to obtain response information including the transaction number, commit timestamp, and write index number; establishing a mapping relationship between the transaction commit response returned by the target database and the corresponding logical write state in the latency-aware buffer based on the unique key value of the data object and the local write logical sequence number; calculating the time interval from commit to confirmation by the target database for each logical write state; and using the time interval as the distributed transaction confirmation latency, writing it into the corresponding record item in the latency-aware buffer to construct the distributed transaction confirmation latency sequence.

[0010] Optionally, the step of constructing a time regression model based on the distributed transaction confirmation delay sequence to generate a prediction projection window for the transaction confirmation time of the logical write state specifically includes: inputting the distributed transaction confirmation delay sequence into a time regression modeling unit; the time regression modeling unit constructing a delay prediction function based on the local write logical sequence number; using a sliding weighted regression or trend fitting algorithm to establish a mapping relationship between the transaction confirmation delay and the logical write order to obtain the time regression model; using the delay prediction function in the time regression model to predict the transaction confirmation time of the currently unreleased logical write state, and outputting the predicted confirmation time point and confidence index of the logical write state; constructing a prediction projection window for the transaction confirmation time based on the predicted confirmation time point and the confidence index; the prediction projection window being centered on the predicted confirmation time point and bounded by the upper and lower limits determined by the confidence index.

[0011] Optionally, the step of sending the target logical write state from the output memory queue to the target database via the output thread and generating a completion flag table specifically includes: sequentially extracting target logical write states eligible for release from the output memory queue, verifying that the logical release status flag of the target logical write state is released and the transaction confirmation status flag is pending confirmation; encapsulating the target logical write state into a data submission request format specified by the target database to obtain data to be submitted; sending the data to be submitted to the distributed write interface of the target database according to the local write logical sequence number via the output thread, listening to and receiving the write response result returned by the target database, the write response result including the submission status, write position index, transaction number, and confirmation time; parsing the write response result and matching it with the target logical write state, confirming that the successfully matched target logical write state is in a completed submission status; registering the target logical write state in a completed submission status in the completion flag table, the completion flag table being used to record the logical mapping identifier, database confirmation time, transaction number, and matching status, and serving as the basis for updating the status of the latency-aware buffer.

[0012] Optionally, the management of the latency-aware buffer based on the completion flag table specifically includes: periodically traversing the completion flag table in the latency-aware buffer and clearing the logical write state that has been written to disk; if a shrinkage of the prediction projection window or a decrease in latency is detected, a buffer collapse mechanism is triggered to release all data objects in the latency-aware buffer that meet the confidence threshold condition to the output memory queue.

[0013] A second aspect of this application provides a real-time data acquisition buffer throughput management device. The device includes an acquisition module and a processing module. The acquisition module is used to acquire a real-time data stream and construct a data object sequence containing original acquisition values, sampling time, measurement point numbers, and status markers, and write the data object sequence into an input memory queue. The processing module is used to input each data object in the input memory queue to a mapping proxy unit, construct a logical mapping identifier based on the current system clock state and the disk buffer write pointer, and combine the data object with the logical mapping identifier to form a logical write state. The logical mapping identifier includes a unique key value for the data object, a local write logical sequence number, a logical commit prediction window, and a distributed transaction candidate confirmation chain identifier. The processing module also uses... The processing module stores the logical write state in a latency-aware buffer and monitors the distributed transaction confirmation latency sequence returned by the target database through a pre-deployed distributed latency monitoring unit in the latency-aware buffer. The processing module is further configured to construct a time regression model based on the distributed transaction confirmation latency sequence to generate a prediction projection window for the transaction confirmation time of the logical write state. If the prediction projection window meets a confidence threshold condition, the processing module grants the corresponding target logical write state release eligibility and writes it to the output memory queue. The processing module is also configured to send the target logical write state in the output memory queue to the target database through an output thread to generate a completion flag table. The processing module is further configured to manage the latency-aware buffer based on the completion flag table.

[0014] A third aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, and both the user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method described above.

[0015] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed, perform the method described above.

[0016] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages:

[0017] 1. Traditional real-time acquisition systems lack modeling and feedback mechanisms for write acknowledgment latency when dealing with databases controlled by Raft or Paxos protocols. This leads to a disconnect between data release and database processing capacity, resulting in the counterintuitive phenomenon of send threads idling while writes accumulate. This technical solution introduces a logical write-state structure on the interface side, coupled with a predictive release mechanism, to construct a proactive rhythm control strategy based on acknowledgment feedback. This dynamically couples the release rhythm with the database processing capacity, fundamentally eliminating throughput inversion and thus improving overall throughput efficiency.

[0018] 2. By deploying a latency-aware buffer and a distributed latency monitoring unit, the system monitors the transaction confirmation responses returned by the database in real time, and constructs a transaction confirmation latency sequence for regression modeling and confidence assessment, thereby forming a predictive projection window. This not only enables the interface program to have latency awareness capabilities, but also transforms the uncontrollable transaction confirmation behavior on the database side into a quantifiable input that the interface can perceive, model, and control.

[0019] 3. By listening to the target database's commit response through the output thread and registering the committed logical write status to the completion flag table, combined with the state synchronization of the delay-aware buffer, it is ensured that every release behavior can be confirmed and every write can be traced back, realizing the state visibility from release to disk, and enhancing the system's transparency and fault tolerance. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a real-time data acquisition buffer throughput management method provided in an embodiment of this application;

[0021] Figure 2 A schematic diagram of a real-time data acquisition buffer throughput management device provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0023] Explanation of reference numerals in the attached figures: 21. Acquisition module; 22. Processing module; 31. Processor; 32. Communication bus; 33. User interface; 34. Network interface; 35. Memory. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0025] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0026] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0027] In high-frequency real-time data acquisition systems, the core responsibility of the interface program is to transmit the high-density data stream continuously generated by the edge acquisition module to the target database in a timely manner. To ensure data consistency and system high availability, target databases in most industrial scenarios are typically built based on distributed consistency protocols, such as Raft or Paxos. These protocols require each write transaction to be replicated, voted on, and confirmed in a strict order across multiple distributed nodes to achieve strong consistency guarantees. However, this mechanism inevitably introduces queuing delays and consistency confirmation delays during transaction execution, resulting in an overall response latency significantly higher than that of traditional single-machine databases, especially when system load is uneven, network fluctuations are frequent, or replica node performance varies significantly.

[0028] In this architecture, existing interface programs typically employ a concurrent release strategy, meaning that once an unused transmission link is detected, the cached data to be processed is immediately released in batches to improve transmission efficiency. However, this strategy ignores the transaction ordering and confirmation bottlenecks caused by the distributed consistency protocol limiting write operations within the target database. A large number of logical write requests are issued simultaneously without transaction scheduling modeling or confirmation capability awareness, leading to a rapid backlog in the database-side transaction processing channel. Transactions must then sequentially enter the internal commit queue to await voting confirmation, resulting in an overall write speed far lower than the release speed of the interface program.

[0029] The direct consequence of this mismatch in timing is that the interface program appears to run smoothly and the output thread is unobstructed, but in reality, the database write transactions continue to accumulate, and the write response is severely delayed, thus affecting the overall throughput efficiency of the system.

[0030] To address the aforementioned technical problems, this application provides a method for managing the throughput of a real-time data acquisition buffer, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a real-time data acquisition buffer throughput management method provided in an embodiment of this application. The method is applied to a server and includes steps S110 to S170, as follows:

[0031] S110. Acquire the real-time data stream and construct a data object sequence containing the original acquisition values, sampling time, measurement point number and status marker, and write the data object sequence into the input memory queue.

[0032] Specifically, server acquisition of real-time data streams refers to the interface program being deployed on the server side to continuously receive real-time generated data inputs from multiple edge acquisition devices, sensor networks, or data access units. This real-time data stream has characteristics such as high frequency, low latency, and strong continuity, and contains unorganized data fragments.

[0033] The data object contains four core fields: Raw acquired value: the actual physical quantity acquired by the sensor or acquisition device, such as temperature, current, pressure, etc.; Sampling time: generated by the system clock of the acquisition device or server, representing the specific timestamp of the sampling action; Measurement point number: used to identify the location and identity of the data acquisition point in the system, usually composed of device number, port number or channel number, etc., to ensure that different data objects can be uniquely traced; Status marker: records the communication integrity, verification result or current operating status of the device for subsequent data quality judgment or anomaly detection.

[0034] Furthermore, writing a sequence of data objects into the input memory queue means that the server sequentially writes a set of constructed data objects into a pre-defined first-in-first-out memory buffer area, forming a continuous data stream channel to be processed. This input memory queue serves as a temporary cache and flow rate buffer, and provides a stable source of input data for the mapping agent unit.

[0035] For example, taking a power monitoring system as an example, a current sensor in a substation collects current values ​​every 10 milliseconds and sends them to the server via an RS485 communication link. The data segment received by the server is as follows: Original collected value: 117.6 (unit: A), sampling time: June 16, 2024, 10:32:11.457, measurement point number: CT_Bus01_PhaseA, status flag: normal communication, CRC check passed. The server packages these four items into a standard data object and writes it sequentially into the input memory queue. As data continues to be received, the input memory queue forms a standardized, indexable, and schedulable data object stream.

[0036] In one possible implementation, acquiring a real-time data stream and constructing a sequence of data objects containing original acquired values, sampling time, measurement point numbers, and status markers, and writing the data object sequence into an input memory queue, specifically includes: receiving original data segments from an edge acquisition device through an acquisition access unit; the original data segments include a sampling trigger signal and original acquired values; generating measurement point numbers based on the physical clock time, communication port number, and device code contained in the sampling trigger signal, and establishing a measurement point identifier mapping table to ensure the uniqueness of the measurement point numbers of the data objects; recording the original acquired values ​​as a data value field, recording the synchronized and calibrated physical clock time as a timestamp field, recording the measurement point number as an identifier field, and recording the communication status or verification flag in the original data segment as a status marker field; combining the data value field, timestamp field, identifier field, and status marker field to construct a data object, and writing the data objects into the input memory queue in a first-in-first-out order, with each data object in the input memory queue having a write pointer and a logical queue position index attached.

[0037] Specifically, this technical solution describes the structured processing procedure on the server side during the real-time data access phase. First, the acquisition access unit receives raw data fragments from the edge acquisition device. These raw data fragments contain a sampling trigger signal and the original acquired values. The sampling trigger signal contains the physical clock time, communication port number, and device code. Based on this, the server generates a unique measurement point number and establishes a measurement point identification mapping table to ensure that each data object has a traceable identity. Subsequently, the server uses the original acquired value as the data value field, records the calibrated sampling time as a timestamp field, uses the generated measurement point number as an identification field, and extracts the communication status or verification result as a status flag field. These four fields together constitute a structurally unified data object.

[0038] By adopting the above technical solution, data collected from different sources, protocols, and devices in the acquisition system is abstracted into standardized units with consistent structure through the unified construction of data objects. This facilitates subsequent logical state generation and status tracking. Standardized coding of measurement point numbers ensures that each data object has a clear source, which can be used for analysis, backtracking, and fault location. The input memory queue acts as a pre-buffer, bridging the gap between the receiving rate and subsequent processing rate, preventing data loss due to short-term computational delays. The status tag field ensures that each data entry carries a quality identifier, helping to build a complete data reliability model and providing a quality screening basis for subsequent delay perception, predictive release, and other mechanisms. Therefore, this step not only effectively transforms real-time data into a structured data stream but also provides a stable, high-quality, and traceable data entry point for the entire throughput management system, serving as the fundamental logical starting point of the entire control chain.

[0039] S120. Input each data object in the input memory queue to the mapping agent unit, construct a logical mapping identifier based on the current system clock state and the disk buffer write pointer, and combine the data object and the logical mapping identifier to form a logical write state. The logical mapping identifier includes the unique key value of the data object, the local write logical sequence number, the logical commit prediction window, and the distributed transaction candidate confirmation chain identifier.

[0040] Specifically, this technical solution describes the logical structure expansion process of data objects in the input memory queue before entering the scheduling and control system. The server sequentially extracts each data object from the input memory queue through a mapping agent unit and constructs a logical mapping identifier based on a high-precision timestamp generated by the current system master clock and the disk buffer write pointer position. The logical mapping identifier uniquely identifies the data object's identity, order, and prediction path in the distributed write process, and includes four fields: a unique key value for the data object (usually composed of a measurement point number and a timestamp), a local write logical sequence number (internal processing order count within the interface), a logical commit prediction window (a dynamic structure reserved for subsequent confirmation time prediction), and a distributed transaction candidate confirmation chain identifier (marking the commit transaction group or write band to which the data object belongs). The above logical mapping identifier, combined with the original data object, forms a logical write state with scheduling, prediction, and tracking attributes.

[0041] In one possible implementation, each data object in the input memory queue is input to the mapping agent unit. A logical mapping identifier is constructed based on the current system clock state and the disk buffer write pointer. The data object and the logical mapping identifier are then combined to form a logical write state. Specifically, the mapping agent unit sequentially extracts each data object from the input memory queue, reads the current system master clock to generate a high-precision write request timestamp, and synchronously reads the current write pointer position state maintained by the disk write manager in the disk buffer. A logical mapping identifier is generated based on the data attribute fields in the data object, the write request timestamp, and the current write pointer position state. The data object and the logical mapping identifier are then bundled and combined to construct the logical write state.

[0042] Specifically, the mapping agent unit sequentially extracts each data object from the input memory queue, reads the current system master clock to generate a high-precision write request timestamp, and synchronously reads the current write pointer position status maintained by the disk write manager in the disk buffer. The mapping agent unit runs in an independent thread, employing a first-in-first-out strategy on the input memory queue to sequentially extract the data objects to be processed. For each data object extracted, the system master clock service is immediately invoked to generate a write request timestamp with millisecond or microsecond precision, ensuring the logical identifier has an accurate time sequence. Simultaneously, this unit interacts with the disk write manager through an interface to read the pointer offset or block address of the area to be written in the current disk buffer, obtaining the real-time position of the write path in the disk space, which is used to construct a structured identifier field associated with the physical write state.

[0043] The mapping agent unit combines the extracted data attribute fields (including data value field, timestamp field, identifier field, and status flag field) with the write request timestamp and hard disk write pointer status to generate a set of logical mapping identifiers to uniquely identify the logical write process. This logical mapping identifier consists of the following components: a unique key value for the data object (e.g., TEMP_Zone4_20250616T151103261 formed by concatenating the measurement point number and the original sampling time), a local write logical sequence number (a unique number for this logical write state in the current runtime cycle, e.g., 14598), a logical commit prediction window (an empty field or a preset estimation range to be filled by the subsequent delay prediction module), and a distributed transaction candidate confirmation chain identifier (e.g., constructed as Group_TEMP_4 based on the output band or device type).

[0044] The process of constructing the logical write state involves structurally merging the original data object with the logical mapping identifier to generate a composite entity with dual information attributes. This logical write state not only retains the original content such as data value fields, timestamp fields, identifier fields, and status flag fields, but also integrates the local write logical sequence number, the distributed transaction candidate confirmation chain identifier, and the window structure required for subsequent predictive scheduling. As the basic unit of the latency-aware buffer, the logical write state serves as the input basis for latency monitoring, time modeling, and release decisions. This technical process, through high-precision time calibration, the introduction of physical write state, and the construction of unique key-value pairs, achieves a semantic upgrade from the original data object to the logical write state, ensuring consistency, predictability, and traceability in subsequent processing stages. It forms the structural foundation of the entire throughput control mechanism.

[0045] S130. Store the logical write state into the latency-aware buffer, and monitor the distributed transaction confirmation latency sequence returned by the target database through the pre-deployed distributed latency monitoring unit in the latency-aware buffer.

[0046] Specifically, this technical solution describes the process of logical write states entering a key relay area with feedback awareness capabilities within the system and establishing a correlation with the write results of the target database. The server writes the logical write states constructed in the mapping proxy unit into a latency-aware buffer. This latency-aware buffer has a dual structural feature: on the one hand, it employs a dual management mechanism of time-series indexes and write sequence indexes to ensure that the logical write states are traceable on the timeline and scheduled in an orderly manner during the write process; on the other hand, it internally deploys a distributed latency monitoring unit to continuously monitor the transaction response results of the target database to each logical write state, including commit confirmation, log replication completion, and transaction number return. Through the unique key value in the logical mapping identifier and the write sequence number, a one-to-one matching relationship is established between the transaction confirmation response returned by the database and the corresponding logical write state, constructing a dynamically updated distributed transaction confirmation latency sequence.

[0047] For example, in an energy control system, the logical write state {key:VOLT_Switch03_20250616T15:20:57.486,seq:12813,predict_window:null,txn_chain_id:Group_VOLT_3} is written to the latency-aware buffer, and the system records its current state as "pending commit". Subsequently, the distributed latency monitoring unit listens for the write response returned by the target database. The response contains the transaction number TXN_98267 and the confirmation timestamp 2025-06-16T15:20:58.112, which successfully matches the key-value pair of the logical write state. The system calculates the transaction confirmation latency of this logical write state to be 626 milliseconds and writes this latency value to the corresponding record in the latency-aware buffer. This latency value also serves as the core input for the subsequent time regression modeling module to predict transaction confirmation trends. This mechanism ensures that the system can perceive changes in the database's write capability in real time and establish a predictive release logic with controllable accuracy based on transaction confirmation behavior.

[0048] In one possible implementation, the logical write state is stored in a latency-aware buffer. A pre-deployed distributed latency monitoring unit in the latency-aware buffer monitors the distributed transaction confirmation latency sequence returned by the target database. Specifically, this includes: writing the logical write state into the latency-aware buffer, which is a dual-index structure with a time-series index and a status identifier mechanism; attaching a logical release status flag and a transaction confirmation status flag to the logical write state after writing; asynchronously monitoring the transaction commit response returned by the target database through the distributed latency monitoring unit to obtain response information including the transaction number, commit timestamp, and write index number; establishing a mapping relationship between the transaction commit response returned by the target database and the corresponding logical write state in the latency-aware buffer based on the unique key value of the data object and the local write logical sequence number; calculating the time interval from commit to confirmation by the target database for each logical write state; and using the time interval as the distributed transaction confirmation latency, writing it into the corresponding record in the latency-aware buffer to construct the distributed transaction confirmation latency sequence.

[0049] Specifically, after a logical write state is generated, it is immediately written to a latency-aware buffer. This buffer is managed using a dual-index mechanism: first, a time-series index is built based on the timestamp dimension to support windowed regression modeling; second, a logical index is built based on the logical write sequence number to quickly locate the transaction status. Each logical write state is appended with two status flags when written to the buffer: the logical release status flag is initially set to "not released," and the transaction confirmation status flag is initially set to "pending confirmation." Together, these flags constitute a continuous identifier of the logical write state's processing status. Example: A logical write state seq=13572, key=TEMP_Node8_20250616T154431117, with appended statuses {release_flag:false,confirm_flag:false}.

[0050] The distributed latency monitoring unit continuously monitors the write acknowledgment responses from the target database in an asynchronous, non-blocking manner. Each time the database completes a write transaction and returns a response, the system extracts key fields from the response, such as the transaction number, commit timestamp, and internal database write position index. These fields logically correspond to the previously issued logical write status, providing a source of responses for subsequent status matching and latency calculation. Example: Receiving a transaction response with the content {txn_id:TXN_50894,commit_time:2025-06-16T15:44:32.004,storage_offset:12.3MB}.

[0051] The latency-aware buffer uses a hash mapping mechanism or composite index mechanism to quickly match the unique key value of the data object in the logical write state with the local write logical sequence number, finding the logical write state corresponding to the transaction response information returned by the target database. Upon successful matching, the system updates the transaction confirmation status flag of the logical write state to "confirmed" and prepares a confirmation timestamp for the next latency calculation. Example: A logical write state is successfully matched using the unique key TEMP_Node8_20250616T154431117 and sequence number 13572, and the status is updated to {release_flag:true,confirm_flag:true}.

[0052] After matching, the system reads the write request timestamp of the logical write state and the commit timestamp returned by the target database. The difference between the two is the distributed transaction confirmation latency of the logical write state. This latency value is expressed in the form of a time interval, with precision typically controlled at the millisecond level, and is used to characterize the dynamic difference between the release speed of the interface program and the processing capacity of the database. Example: The write request time is 15:44:31.443, the database confirmation time is 15:44:32.004, and the calculated confirmation latency is 561 milliseconds.

[0053] The server writes the calculated confirmation latency value into the latency field of the corresponding logical write state, and simultaneously maintains a sequence of latency records sorted by logical write sequence number or timestamp, forming a sliding, updatable distributed transaction confirmation latency sequence. This latency sequence serves as the input basis for a time regression model, reflecting the trend and jitter characteristics of the database write confirmation capability over time, and is used by the subsequent release prediction module for real-time modeling and decision-making. Example: Write a 561ms latency value into the latency field of the logical write state and append it to the global latency sequence {…,534,612,547,561}.

[0054] Through the above five consecutive steps, the logical write state not only receives state management after entering the delay-aware buffer, but also establishes a precise feedback path with the database write behavior, thereby forming the interface program's ability to perceive the database response behavior, and on this basis, supporting the input closed loop of the predictive release and dynamic rhythm control mechanism.

[0055] S140. Construct a time regression model based on the distributed transaction confirmation delay sequence to generate a prediction projection window for the transaction confirmation time of the logical write state.

[0056] Specifically, this technical solution describes how, after obtaining a distributed transaction confirmation delay sequence, the server constructs a time regression model based on this sequence to predict the transaction confirmation time for uncommitted logical write states, forming a predictive projection window to guide release decisions. Specifically, the server extracts the transaction confirmation delay values ​​of multiple recently committed logical write states from the delay-aware buffer, and constructs a time regression model reflecting the current confirmation capability trend of the distributed database, using the logical write sequence number or write timestamp as the independent variable and the corresponding delay value as the dependent variable. This model can be dynamically generated using methods such as sliding weighted regression, locally weighted linear regression, or exponentially weighted moving average, and supports predicting the future confirmation time of any pending logical write state.

[0057] In one possible implementation, a time regression model is constructed based on the distributed transaction confirmation delay sequence to generate a prediction projection window for the transaction confirmation time of the logical write state. Specifically, this includes: inputting the distributed transaction confirmation delay sequence into a time regression modeling unit; the time regression modeling unit constructs a delay prediction function based on the local write logical sequence number; and using a sliding weighted regression or trend fitting algorithm to establish a mapping relationship between the transaction confirmation delay and the logical write order to obtain the time regression model; using the delay prediction function in the time regression model to predict the transaction confirmation time of the currently unreleased logical write state, and outputting the predicted confirmation time point and confidence index of the logical write state; and constructing a prediction projection window for the transaction confirmation time based on the predicted confirmation time point and the confidence index, with the predicted confirmation time point as the center and the upper and lower limits determined by the confidence index as the boundaries.

[0058] Specifically, this technical solution describes how a server, after obtaining a distributed transaction confirmation delay sequence, constructs a time regression model and predicts the transaction confirmation time for logical write states that have not yet been released based on this model. The entire process includes three steps: modeling, prediction, and window generation, forming the core basis for the release scheduling logic. The distributed transaction confirmation delay sequence is input into the time regression modeling unit. The time regression modeling unit constructs a delay prediction function based on the local write logical sequence number and uses a sliding weighted regression or trend fitting algorithm to establish a mapping relationship between transaction confirmation delay and logical write order to obtain the time regression model. The time regression modeling unit first extracts the logical write state sequence number and its corresponding transaction confirmation delay value from the delay-aware buffer over a period of time, constructing the training sample set as follows:

[0059]

[0060] in, This indicates that the logical sequence number is written. This represents the delay (in milliseconds) from submission to confirmation of the logical write state. The modeling unit uses a sliding weighted linear regression function, with the specific formula as follows:

[0061]

[0062] in, The sequence number of the logic to be predicted is the write state. To predict delayed results, The weight represents the degree of influence of neighboring samples on the current prediction point. This is the weight decay coefficient, used to control the decay rate of neighboring sample weights. This model has the ability to slide and update over time, dynamically fitting the fluctuation trend in the delayed sequence.

[0063] The server's local write logical sequence number for the currently unreleased logical write state is: The time regression model applies the delayed prediction function. Calculate the estimated latency from submission to confirmation. The server then adds this latency value to the write request timestamp in the logical write state. The predicted confirmation time point is obtained using the following formula:

[0064]

[0065] Meanwhile, the server constructs a confidence index based on the sliding variance of historical residuals. , representing the fluctuation range of the predicted value, is given by the following formula:

[0066]

[0067] By calculating the standard deviation It can measure the model's fit to the sequence segment and construct error boundaries for the prediction results.

[0068] Ultimately, the server confirms the time point based on the prediction. With confidence index The formula for constructing the prediction projection window is as follows:

[0069]

[0070] in, and This is an adjustable boundary coefficient, for example, 1 / sim2, used to control the width of the prediction window. This window will serve as the basis for determining the release of the logical write state. If the current system clock enters this window and the window width does not exceed the system release tolerance threshold, then the logical write state is determined to be releaseable.

[0071] Example: A logical write state sequence number is 12870, the prediction latency is 540ms, the confidence standard deviation is 35ms, the write time is 15:56:40.012, the prediction confirmation time is 15:56:40.552, and the projection window is... Through the above steps, the server transforms discrete latency data into a continuous prediction function, enabling advance estimation and dynamic release control of logical write-state transaction confirmation behavior. This forms the core decision-making basis of the entire throughput management mechanism.

[0072] S150. If it is determined that the prediction projection window meets the confidence threshold condition, then the corresponding target logic write state release qualification is granted and written to the output memory queue.

[0073] Specifically, after obtaining the predicted confirmation time point of the logical write state and its corresponding predicted projection window from the time regression model, the server analyzes the window width. If the difference between the upper and lower bounds of the window is less than the confidence threshold set by the system, the prediction result is considered sufficiently stable and reliable, and the logical write state is granted the right to be released. This process can prevent premature release when the prediction is inaccurate, thereby effectively avoiding throughput anomalies caused by database write queuing.

[0074] For example, the predicted confirmation time for a logical write state is 15:57:20.340, the predicted projection window is [15:57:20.290, 15:57:20.390], and the window width is 100 milliseconds. The system's current confidence threshold is 120 milliseconds, therefore the window meets the release condition. At this time, the server will set a "release qualified" flag on the logical write state and push it to the output memory queue, entering the subsequent output thread scheduling process. This ensures that data delivery dynamically matches the database's current processing capacity, reducing the risk of latency accumulation caused by logical write states on the database side.

[0075] S160. The target logic in the output memory queue is written to the target database through the output thread to generate a completion mark table.

[0076] Specifically, this technical solution describes how, after determining that a target logical write state is eligible for release, the server uses an output thread to send data to the target database and generates a completion flag table to track the execution status of write transactions. Specifically, the server extracts each target logical write state from the output memory queue, confirms its logical release status as "released" and its transaction confirmation status as "pending confirmation," and then sequentially encapsulates it according to its local write logical sequence number and submits it to the target database's distributed write interface. During the write process, the server listens for the commit response information returned by the target database, parses out fields such as commit status, transaction number, and confirmation time, and matches these fields one by one with the original logical mapping identifier. Logical write states that successfully match are marked as "committed successfully."

[0077] For example, the target logical write state "SEQ-11238" contains the data value "75.6°C" and the measurement point number "T-0319". After being granted release eligibility at 10:45:12.784, it is written to the output memory queue. The server sends it to the database interface using the Raft protocol through the output thread, and receives a response containing "TXN-652321", "confirmed" status, and timestamp "10:45:13.092" a few hundred milliseconds later. The server confirms that the transaction number in the response is completely consistent with the logical mapping identifier corresponding to "SEQ-11238", thus considering the commit complete and registering it in the completion flag table. This completion flag table serves as an important data foundation for logical write state status management and latency modeling feedback, making subsequent predicted releases more robust and possessing closed-loop control capabilities.

[0078] In one possible implementation, the target logical write state in the output memory queue is sent to the target database via an output thread to generate a completion flag table. Specifically, this includes: sequentially extracting target logical write states eligible for release from the output memory queue; verifying that the logical release status flag of the target logical write state is released and the transaction confirmation status flag is pending confirmation; encapsulating the target logical write state into a data commit request format specified by the target database to obtain the data to be committed; sending the data to be committed to the distributed write interface of the target database according to the local write logical sequence number via the output thread; listening to and receiving the write response result returned by the target database, the write response result including the commit status, write position index, transaction number, and confirmation time; parsing the write response result and matching it with the target logical write state, confirming that the successfully matched target logical write state is in a completed commit state; and registering the completed commit target logical write state in the completion flag table, which records the logical mapping identifier, database confirmation time, transaction number, and matching status, and serves as the basis for updating the status of the latency-aware buffer.

[0079] Specifically, first, the server sequentially retrieves each target logical write state from the output memory queue and verifies its status fields to determine whether its logical release status flag is "released" and its transaction confirmation status flag is "pending confirmation." This status verification process ensures that only logical write states that have completed their release decisions but have not yet been confirmed by the target database are processed, avoiding duplicate or erroneous commits and ensuring the consistency and controllability of the entire write chain. Subsequently, the server encapsulates the target logical write state according to the distributed commit format required by the target database, constructing a standardized data packet to be committed containing data value fields, timestamp fields, identifier fields, status flag fields, and logical mapping identifiers. The encapsulation process maintains the consistency between the original data and its context logical identifiers, ensuring that the database can correctly parse and register the logical write relationship.

[0080] Then, the server sends the data to be committed to the target database's distributed write interface in the order of its local write logical sequence number via an output thread, avoiding consistency conflicts caused by out-of-order commits. After sending, the server monitors the write response results returned by the target database in real time, extracting key fields such as commit status, write position index, transaction number, and confirmation time, and establishing a logical receipt data structure for subsequent matching. Next, the server parses the write response results and matches them one by one with the logical mapping identifier in the original logical write status based on the transaction number, confirmation time, and write position index in the response. If a match is successful, the logical write status is marked as "committed," its transaction confirmation status field is updated, and a clear commit confirmation record is formed.

[0081] Finally, the server registers the logical write status of the completed commit in the completion flag table. The completion flag table records information including the logical mapping identifier, database confirmation time, transaction number, and matching status. This completion flag table not only serves as the basis for updating the state of the latency-aware buffer, but also provides feedback support for subsequent transaction prediction modeling and release scheduling, thus forming a closed-loop adaptive data release management mechanism driven by transaction confirmation results.

[0082] S170. Based on the completion flag table, manage the delay-aware buffer.

[0083] Specifically, the server periodically traverses the completion marker table, locates the logical write states that have been confirmed as completed, and removes their corresponding records from the latency-aware buffer to release memory resources and prevent buffer bloat caused by the accumulation of historical data. Simultaneously, the server monitors the overall convergence trend of the prediction projection window. If it detects a decrease in regression model prediction error or a reduction in database confirmation latency, it triggers a buffer collapse mechanism, pushing a batch of logical write states that have not yet been released but have met the confidence threshold conditions to the output memory queue, achieving rapid throughput release under latency-tolerant conditions.

[0084] For example, if the completion marker table records that the three logical write states "SEQ-11021", "SEQ-11022", and "SEQ-11023" have been successfully confirmed and submitted by the database before 10:21:02, the server will delete the corresponding records in the latency-aware buffer and update the latency sequence. If the current time is 10:21:03 and the prediction error decreases, and another batch of logical write states such as "SEQ-11024" to "SEQ-11030" meet the confidence threshold of the current prediction projection window, the server will immediately trigger the buffer collapse mechanism, moving all these logical write states to the output memory queue for distribution at once. This improves overall throughput efficiency, reduces waiting time, and ensures the continuity and balance of high-frequency real-time data writing.

[0085] In one possible implementation, the delay-aware buffer is managed based on a completion flag table, specifically including: periodically traversing the completion flag table in the delay-aware buffer and clearing logical write states that have been written to disk; if a shrinkage of the prediction projection window or a decrease in latency is detected, a buffer collapse mechanism is triggered to release all data objects in the delay-aware buffer that meet the confidence threshold condition to the output memory queue.

[0086] Specifically, firstly, the server performs a periodic traversal of the completion marker table at set time intervals. During the traversal, the server reads the logical mapping identifier of each completion marker record and performs index matching and location in the latency-aware buffer. If it is determined that the logical write state corresponding to the logical mapping identifier has been confirmed and committed by the database, and its commit time is more than a preset safety interval earlier than the current system time, then the logical write state is completely cleared from the latency-aware buffer. This step ensures that the time-series data structure of the latency-aware buffer always maintains a minimum effective working set, preventing storage resource waste and decreased access efficiency caused by historical record retention.

[0087] Subsequently, the server performs regression modeling trend analysis on the latest distributed transaction confirmation latency sequence in the latency-aware buffer. If the latency sequence shows an overall downward trend, or the current window prediction error range significantly converges (i.e., the prediction projection window shrinks), it indicates that the database's current write response performance is in a releaseable state. At this point, the server triggers a buffer collapse mechanism, re-traverses the latency-aware buffer, and filters out logical write states that have not yet been released but whose corresponding transaction prediction projection windows meet the confidence threshold. These logical write states are then extracted in batches from the latency-aware buffer and released as a whole to the output memory queue. This operation significantly reduces the impact of waiting time on system throughput, enables proactive release of data objects under high confidence, and improves overall transmission efficiency and transaction consistency stability.

[0088] This application also provides a real-time data acquisition buffer throughput management device, referring to... Figure 2 , Figure 2 This is a schematic diagram of a real-time data acquisition buffer throughput management device provided in an embodiment of this application. The device is a server, which includes an acquisition module 21 and a processing module 22. The acquisition module 21 acquires real-time data streams and constructs a data object sequence containing original acquisition values, sampling time, measurement point numbers, and status markers, and writes the data object sequence into an input memory queue. The processing module 22 inputs each data object in the input memory queue to a mapping proxy unit, constructs a logical mapping identifier based on the current system clock state and the disk buffer write pointer, and combines the data object and the logical mapping identifier to form a logical write state. The logical mapping identifier includes a unique key value for the data object, a local write logical sequence number, a logical commit prediction window, and a distributed transaction candidate confirmation chain identifier. The processing module 22 then... The logical write state is stored in the latency-aware buffer. The distributed latency monitoring unit pre-deployed in the latency-aware buffer listens to the distributed transaction confirmation latency sequence returned by the target database. The processing module 22 constructs a time regression model based on the distributed transaction confirmation latency sequence and generates a prediction projection window for the transaction confirmation time of the logical write state. If the processing module 22 determines that the prediction projection window meets the confidence threshold condition, it grants the corresponding target logical write state the right to be released and writes it to the output memory queue. The processing module 22 sends the target logical write state in the output memory queue to the target database through the output thread and generates a completion mark table. The processing module 22 manages the latency-aware buffer based on the completion mark table.

[0089] In one possible implementation, the acquisition module 21 acquires the real-time data stream and constructs a sequence of data objects containing the original acquired values, sampling time, measurement point number, and status marker. The sequence of data objects is then written into an input memory queue. Specifically, the acquisition module 21 receives raw data segments from the edge acquisition device through the acquisition access unit. The raw data segments include a sampling trigger signal and the original acquired values. The processing module 22 generates measurement point numbers based on the physical clock time, communication port number, and device code contained in the sampling trigger signal, and establishes a measurement point identifier mapping table to ensure the uniqueness of the measurement point numbers for the data objects. The processing module 22 records the original acquired values ​​as a data value field, records the synchronized and calibrated physical clock time as a timestamp field, records the measurement point number as an identifier field, and records the communication status or verification flag in the original data segment as a status marker field. The processing module 22 combines the data value field, timestamp field, identifier field, and status marker field to construct a data object, and writes the data objects into the input memory queue in a first-in-first-out order. Each data object in the input memory queue is appended with a write pointer and a logical queue position index.

[0090] In one possible implementation, the processing module 22 inputs each data object in the input memory queue to the mapping agent unit, constructs a logical mapping identifier based on the current system clock state and the disk buffer write pointer, and combines the data object and the logical mapping identifier to form a logical write state. Specifically, the processing module 22 extracts each data object in the input memory queue sequentially from the mapping agent unit, reads the current system master clock to generate a high-precision write request timestamp, and synchronously reads the current write pointer position state maintained by the disk write manager in the disk buffer; the processing module 22 generates a logical mapping identifier based on the data attribute fields in the data object, the write request timestamp, and the current write pointer position state; the processing module 22 binds and combines the data object and the logical mapping identifier to construct the logical write state.

[0091] In one possible implementation, the processing module 22 stores the logical write state in a latency-aware buffer. Through a pre-deployed distributed latency monitoring unit in the latency-aware buffer, it monitors the distributed transaction confirmation latency sequence returned by the target database. Specifically, the processing module 22 writes the logical write state into the latency-aware buffer, which is a dual-index structure with a time-series index and a status identifier mechanism. After writing, the logical write state is appended with a logical release status flag and a transaction confirmation status flag. The processing module 22 asynchronously monitors the transaction commit response returned by the target database through the distributed latency monitoring unit, obtaining response information including the transaction number, commit timestamp, and write index number. Based on the unique key value of the data object and the local write logical sequence number, the processing module 22 establishes a mapping relationship between the transaction commit response returned by the target database and the corresponding logical write state in the latency-aware buffer. The processing module 22 calculates the time interval from commit to confirmation by the target database for each logical write state. The processing module 22 uses the time interval as the distributed transaction confirmation latency and writes it into the corresponding record in the latency-aware buffer to construct the distributed transaction confirmation latency sequence.

[0092] In one possible implementation, the processing module 22 constructs a time regression model based on the distributed transaction confirmation delay sequence and generates a prediction projection window for the transaction confirmation time of the logical write state. Specifically, the processing module 22 inputs the distributed transaction confirmation delay sequence into the time regression modeling unit. The time regression modeling unit constructs a delay prediction function based on the local write logical sequence number and uses a sliding weighted regression or trend fitting algorithm to establish a mapping relationship between the transaction confirmation delay and the logical write order to obtain the time regression model. The processing module 22 uses the delay prediction function in the time regression model to predict the transaction confirmation time of the currently unreleased logical write state and outputs the predicted confirmation time point and confidence index of the logical write state. Based on the predicted confirmation time point and the confidence index, the processing module 22 constructs a prediction projection window for the transaction confirmation time. The prediction projection window is centered on the predicted confirmation time point and bounded by the upper and lower limits determined by the confidence index.

[0093] In one possible implementation, the processing module 22 sends the target logical write state from the output memory queue to the target database through the output thread, generating a completion flag table. Specifically, the processing module 22 sequentially extracts the target logical write states that are eligible for release from the output memory queue, verifies that the logical release status flag of the target logical write state is released and the transaction confirmation status flag is pending confirmation; the processing module 22 encapsulates the target logical write state into the data submission request format specified by the target database to obtain the data to be submitted; the processing module 22 sends the data to be submitted to the distributed write interface of the target database according to the local write logical sequence number through the output thread, listens for and receives the write response result returned by the target database, the write response result includes the submission status, write position index, transaction number and confirmation time; the processing module 22 parses the write response result and matches it with the target logical write state, confirming that the successfully matched target logical write state is in the completed submission status; the processing module 22 registers the target logical write state in the completed submission status in the completion flag table, the completion flag table is used to record the logical mapping identifier, database confirmation time, transaction number and matching status, and serves as the basis for updating the status of the latency-aware buffer.

[0094] In one possible implementation, the processing module 22 manages the delay-aware buffer based on the completion mark table, specifically including: the processing module 22 periodically traverses the completion mark table in the delay-aware buffer and clears the logical write state that has been written to disk; if the processing module 22 detects that the prediction projection window shrinks or the delay decreases, it triggers the buffer collapse mechanism to release the data objects in the delay-aware buffer that meet the confidence threshold condition to the output memory queue.

[0095] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0096] This application also provides an electronic device, with reference to... Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: at least one processor 31, at least one network interface 34, a user interface 33, a memory 35, and at least one communication bus 32.

[0097] The communication bus 32 is used to enable communication between these components.

[0098] The user interface 33 may include a display screen and a camera. Optionally, the user interface 33 may also include a standard wired interface and a wireless interface.

[0099] The network interface 34 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0100] The processor 31 may include one or more processing cores. The processor 31 connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in the memory 35, and calling data stored in the memory 35 to perform various server functions and process data. Optionally, the processor 31 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 31 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 31 and may be implemented as a separate chip.

[0101] The memory 35 may include random access memory (RAM) or read-only memory. Optionally, the memory 35 may include a non-transitory computer-readable storage medium. The memory 35 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 35 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 35 may also be at least one storage device located remotely from the aforementioned processor 31. Figure 3As shown, the memory 35, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a real-time data acquisition buffer throughput management method.

[0102] exist Figure 3 In the electronic device shown, the user interface 33 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 31 can be used to call an application program stored in the memory 35 that is a real-time data acquisition buffer throughput management method. When executed by one or more processors, the electronic device executes one or more methods as described in the above embodiments.

[0103] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0104] This application also provides a computer-readable storage medium storing instructions. When executed by one or more processors, these instructions cause an electronic device to perform one or more of the methods described in the above embodiments.

[0105] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0107] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0110] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for managing the throughput of a real-time data acquisition buffer, characterized in that, The method includes: Acquire real-time data stream and construct a data object sequence containing original acquisition values, sampling time, measurement point number and status marker, and write the data object sequence into the input memory queue; Each data object in the input memory queue is input to the mapping agent unit. A logical mapping identifier is constructed based on the current system clock state and the disk buffer write pointer. The data object and the logical mapping identifier are combined to form a logical write state. The logical mapping identifier includes a unique key value of the data object, a local write logical sequence number, a logical commit prediction window, and a distributed transaction candidate confirmation chain identifier. The logical write state is stored in a latency-aware buffer, and the distributed latency monitoring unit pre-deployed in the latency-aware buffer is used to monitor the distributed transaction confirmation latency sequence returned by the target database. A time regression model is constructed based on the distributed transaction confirmation delay sequence to generate a prediction projection window for the transaction confirmation time of the logical write state; If it is determined that the predicted projection window meets the confidence threshold condition, then the corresponding target logic write state is granted the right to be released and written to the output memory queue; The target logic write state in the output memory queue is sent to the target database through the output thread to generate a completion mark table. The delay-aware buffer is managed based on the completion flag table. The step of storing the logical write state into a latency-aware buffer, and monitoring the distributed transaction confirmation latency sequence returned by the target database through a pre-deployed distributed latency monitoring unit in the latency-aware buffer, specifically includes: The logical write state is written to the latency-aware buffer, which is a dual-index structure with timing index and status identification mechanism. After the logical write state is written, a logical release status flag and a transaction confirmation status flag are attached. The distributed latency monitoring unit asynchronously listens to the transaction commit response returned by the target database to obtain response information including the transaction number, commit timestamp, and write index number. Based on the unique key value of the data object and the local write logic sequence number, a mapping relationship is established between the transaction commit response returned by the target database and the corresponding logical write state in the latency-aware buffer; For each of the logical write states, calculate the time interval from submission to confirmation by the target database; The time interval is used as the distributed transaction confirmation delay and written into the corresponding record in the delay awareness buffer to construct the distributed transaction confirmation delay sequence.

2. The real-time data acquisition buffer throughput management method according to claim 1, characterized in that, The process of acquiring real-time data streams and constructing a data object sequence containing original acquired values, sampling time, measurement point numbers, and status markers, and writing the data object sequence into an input memory queue, specifically includes: The acquisition access unit receives raw data segments from the edge acquisition device, the raw data segments including sampling trigger signals and raw acquisition values; Based on the physical clock time, communication port number, and device code contained in the sampling trigger signal, a measurement point number is generated, and a measurement point identifier mapping table is established to ensure that the measurement point number of the data object is unique. The original acquired values ​​are recorded as a data value field, the physical clock time synchronization and calibration are recorded as a timestamp field, the measurement point number is recorded as an identifier field, and the communication status or verification flag in the original data segment is recorded as a status flag field. The data value field, the timestamp field, the identifier field, and the status flag field are combined to construct a data object, and the data object is written to the input memory queue in a first-in-first-out order. Each data object in the input memory queue is appended with a write pointer and a logical queue position index.

3. The real-time data acquisition buffer throughput management method according to claim 1, characterized in that, The process of inputting each data object in the input memory queue to the mapping proxy unit, constructing a logical mapping identifier based on the current system clock state and the disk buffer write pointer, and combining the data object with the logical mapping identifier to form a logical write state specifically includes: The mapping agent unit sequentially extracts each data object from the input memory queue, reads the current system master clock to generate a high-precision write request timestamp, and synchronously reads the current write pointer position status maintained by the disk write manager in the disk buffer. The logical mapping identifier is generated based on the data attribute fields in the data object, the write request timestamp, and the current write pointer position status. The data object and the logical mapping identifier are bundled together to construct the logical write state.

4. The real-time data acquisition buffer throughput management method according to claim 1, characterized in that, The step of constructing a time regression model based on the distributed transaction confirmation delay sequence to generate a prediction projection window for the transaction confirmation time of the logical write state specifically includes: The distributed transaction confirmation delay sequence is input into the time regression modeling unit. The time regression modeling unit constructs a delay prediction function based on the local write logical sequence number and uses a sliding weighted regression or trend fitting algorithm to establish a mapping relationship between transaction confirmation delay and logical write order to obtain the time regression model. The delay prediction function in the time regression model is used to predict the transaction confirmation time of the currently unreleased logical write state, and the predicted confirmation time point and confidence index of the logical write state are output. Based on the predicted confirmation time point and the confidence index, a prediction projection window for the transaction confirmation time is constructed. The prediction projection window is centered on the predicted confirmation time point and bounded by the upper and lower limits determined by the confidence index.

5. The real-time data acquisition buffer throughput management method according to claim 1, characterized in that, The step of sending the target logic write state from the output memory queue to the target database through the output thread and generating a completion marker table specifically includes: Sequentially extract target logical write states that are eligible for release from the output memory queue, verify that the logical release status flag of the target logical write state is released and the transaction confirmation status flag is pending confirmation. The target logic write state is encapsulated into the data submission request format specified by the target database to obtain the data to be submitted; The output thread sends the data to be committed to the distributed write interface of the target database according to the local write logic sequence number, listens for and receives the write response result returned by the target database, and the write response result includes the commit status, write position index, transaction number and confirmation time. The write response result is parsed and matched with the target logical write state to confirm that the successfully matched target logical write state is a completed submission state. The target logic that has been committed is registered in the completion flag table. The completion flag table is used to record the logic mapping identifier, database confirmation time, transaction number and matching status, and serves as the basis for updating the status of the delay-aware buffer.

6. The real-time data acquisition buffer throughput management method according to claim 1, characterized in that, The management of the delay-aware buffer based on the completion flag table specifically includes: The completion flag table is periodically traversed in the delay-aware buffer to clear the logical write state that has been written to disk; If a shrinkage of the prediction projection window or a decrease in latency is detected, a buffer collapse mechanism is triggered, releasing all data objects in the latency-aware buffer that meet the confidence threshold condition to the output memory queue.

7. A real-time data acquisition buffer throughput management device, characterized in that, The device is used to execute the real-time data acquisition buffer throughput management method as described in any one of claims 1 to 6, the device comprising an acquisition module (21) and a processing module (22), wherein, The acquisition module (21) is used to acquire real-time data streams and construct a data object sequence containing original acquisition values, sampling time, measurement point number and status marker, and write the data object sequence into the input memory queue; The processing module (22) is used to input each data object in the input memory queue to the mapping agent unit, construct a logical mapping identifier based on the current system clock state and the disk buffer write pointer, and combine the data object and the logical mapping identifier to form a logical write state. The logical mapping identifier includes a unique key value of the data object, a local write logical sequence number, a logical commit prediction window, and a distributed transaction candidate confirmation chain identifier. The processing module (22) is also used to store the logical write state into the delay awareness buffer and listen to the distributed transaction confirmation delay sequence returned by the target database through the distributed delay monitoring unit pre-deployed in the delay awareness buffer. The processing module (22) is also used to construct a time regression model based on the distributed transaction confirmation delay sequence and generate a prediction projection window for the transaction confirmation time of the logical write state; The processing module (22) is further configured to, if it is determined that the predicted projection window meets the confidence threshold condition, grant the corresponding target logical write state release qualification and write it into the output memory queue; The processing module (22) is also used to send the target logic write state in the output memory queue to the target database through the output thread to generate a completion mark table; The processing module (22) is also used to manage the delay-aware buffer based on the completion flag table.

8. An electronic device, characterized in that, The electronic device includes a processor (31), a memory (35), a user interface (33), and a network interface (34). The memory (35) is used to store instructions. The user interface (33) and the network interface (34) are both used to communicate with other devices. The processor (31) is used to execute the instructions stored in the memory (35) to cause the electronic device to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1 to 6.

Images