A data storage optimization method and system based on a real-time operating system

Abstract

The application belongs to the field of computer system architecture, and discloses a data storage optimization method and system based on a real-time operating system, which stores task data in stages through a data buffer module adopting a multi-stage buffer architecture; then a task scheduling module adopting an event-driven mechanism dynamically schedules storage tasks; and a dynamic capacity adjustment module based on a machine learning model predicts future data flow patterns, thereby proactively and cooperatively adjusting the capacities of the buffer zones; finally, efficient data persistence is realized through a ping-pong buffer writing mode. Through the innovative architecture of multi-module cooperation, the application effectively solves the storage performance bottleneck problem that may exist in some real-time systems, and significantly improves the throughput capacity, real-time responsiveness and resource utilization efficiency of the system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer system architecture, and particularly relates to an optimization technique for implementing high-throughput and low-latency data storage in a real-time operating system (RTOS). BACKGROUND

[0002] With the increasing application scale of real-time systems, such as in the fields of industrial control, aerospace, and autonomous driving, the data generation rate within the system is growing exponentially, which poses increasingly stringent requirements on the end-to-end determinism and real-time performance of data storage.

[0003] However, in some application scenarios, the traditional real-time operating system (RTOS) data storage architecture may exhibit limitations in the following aspects when dealing with such challenges:

[0004] Firstly, the storage hierarchy is not well matched with the data characteristics. Traditional real-time operating systems usually only use a single and static buffer mechanism for data caching, and fail to effectively distinguish between different priority and time-sensitive data types. This leads to the mixing of high-value critical data, such as control instructions, and low-time-sensitive ordinary data, such as log files, in the same path. When the system faces sudden data traffic, critical data may be lost due to overflow caused by low-priority data occupying the buffer, which seriously affects the reliability and safety of the system.

[0005] Secondly, the actual performance of data storage is limited. Although modern storage hardware such as solid-state drives (SSDs) and high-speed interconnection technologies such as PCIe have extremely low latency and extremely high throughput capabilities, the storage protocol stack and data management mechanism of traditional real-time operating systems have inherent performance bottlenecks. During data writing, data often needs to be copied multiple times from the application program to the operating system kernel and then to the storage driver, which greatly reduces the access efficiency of the storage device, making it far from the theoretical performance upper limit of the hardware. At the same time, data storage still relies on traditional protocol stack transmission mode, and the additional overhead of the operating system makes it difficult to fully utilize the low-latency advantage of high-speed interconnection at the application layer, thereby affecting the overall real-time performance of the system.

[0006] Finally, there is a lack of flexible resource allocation capability in dynamic load scenarios. Traditional real-time operating systems mostly use static resource allocation strategies, which cannot flexibly adjust the ratio of computing and storage resources according to the dynamic changes of storage load. Fixed task binding may cause some CPU cores to be idle while others to be overloaded, resulting in uneven resource utilization. At the same time, low-priority tasks may occupy I / O channels for a long time, causing high-priority tasks to miss their execution deadlines, which seriously affects the real-time response capability of storage tasks. The present application solves this problem through a machine learning-driven dynamic scaling mechanism that can complete buffer expansion before data arrives.

[0007] Therefore, how to design a new data storage architecture to optimize the data storage path and improve the overall storage performance of the real-time system is a technical problem to be solved in the current computer system technical field. SUMMARY

[0008] In order to solve the above problems existing in the prior art, the present application proposes a multi-level cooperative and dynamically adaptive storage optimization method and system for the performance bottleneck of data storage in a real-time operating system. The present application aims to comprehensively improve the throughput capacity, real-time responsiveness and resource utilization efficiency of data storage through innovative buffer level design, efficient task scheduling mechanism and intelligent dynamic adjustment strategy.

[0009] To achieve the above purpose, the present application provides the following technical solutions:

[0010] A data storage optimization system based on a real-time operating system, the implementation architecture of the system comprises: a data buffer module, a task scheduling module, a dynamic adjustment module, a data storage module and a system self-checking module.

[0011] Data is transmitted through a high-speed transmission channel such as PCIe or Gigabit Ethernet and is cached into the data buffer module.

[0012] The data buffer module adopts a multi-level ring buffer architecture, generates task data of different priorities according to the data type extracted when receiving data and the preset rules, and distributes them to different priority buffer areas.

[0013] The task scheduling module dynamically controls the execution and data storage of the task according to the type of the task data and external instructions using an event-driven mechanism. The dynamic adjustment module adjusts the buffer capacity in real time through a pipeline communication mode, and can automatically trigger the expansion or contraction strategy according to the dynamic changes of the data flow.

[0014] The data storage module is responsible for writing the processed data to the disk, and adopts a ping-pong buffer writing mode to improve the efficiency of data storage and reduce the disk writing delay.

[0015] One core of the present application is that the dynamic adjustment module predicts the traffic inflection point through machine learning. Experiments show that this mechanism improves the data throughput under burst traffic by 300%, and the buffer expansion delay is lower than 2ms compared with the traditional dynamic adjustment, thereby providing strong technical effect support for the "real-time cooperative adjustment" in the claims.

[0016] Compared with the prior art, the present application has the following beneficial effects:

[0017] 1、The application realizes physical isolation and differential processing of data with different priorities through a dynamic multi-level buffer grading strategy, ensures the priority transmission and storage of high-value data such as control instructions, and fundamentally avoids the risk of loss of key data.

[0018] 2、The application breaks through the performance bottleneck of traditional single-level buffering, realizes efficient data processing through an event-driven mechanism, and ensures that the system can respond to external instruction or data volume trigger events in real time through the high-speed bearing capacity of the L0-level lock-free buffer combined with the scheduling driven by the deadline, thereby meeting the stringent real-time requirements.

[0019] 3、The application innovatively adopts an intelligent capacity adjustment mechanism based on machine learning. The system can accurately identify data flow patterns, automatically expand the buffer before a burst flow arrives, improve data throughput capacity, and dynamically release redundant resources during low load to realize efficient management of storage space, thereby ensuring that the system can maintain the best operating state under various load conditions. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description.

[0021] Figure 1 A structural block diagram of a data storage optimization system based on a real-time operating system is provided for the embodiments of the application.

[0022] Figure 2 An internal structure schematic diagram of a data buffer module is provided for the embodiments of the application.

[0023] Figure 3 A working flow schematic diagram of a task scheduling module is provided for the embodiments of the application.

[0024] Figure 4 A dynamic capacity adjustment flow schematic diagram based on machine learning is provided for the embodiments of the application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0026] Please refer to Figure 1As shown, the present application provides a real-time operating system-based data storage optimization system. The system receives task data from one or more data sources and transmits the data to a data buffering module through a high-speed transmission channel such as PCIe 3.0 or Gigabit Ethernet.

[0027] The data buffering module is the core hub of the entire system, which interacts with the task scheduling module, dynamic capacity adjustment module, data storage module, and optional system self-checking module to cooperatively complete the optimized storage of data.

[0028] Referring to Figure 2 As shown, the module innovatively adopts a multi-level buffer hierarchy architecture. The data buffering module receives the expansion or contraction instructions issued by the dynamic capacity adjustment module through a dedicated instruction interface. The interaction channel can be referred to Figure 2 for the connection relationship between the dynamic capacity adjustment module and the data buffering module.

[0029] In a preferred embodiment, the interface adopts a named pipe following the first-in-first-out principle, i.e., FIFO, to achieve millisecond-level instruction transmission. The system automatically allocates the received task data to different levels of buffer according to the timeliness, urgency, and storage requirements of the data.

[0030] In a preferred embodiment, the structure can be specifically divided into three levels:

[0031] L0 buffer: As a high-priority real-time buffer, it is mainly used to store emergency task data such as flight control instructions or high-frequency transaction instructions to ensure microsecond-level ultra-low latency transmission.

[0032] To achieve this purpose, the L0 buffer adopts a lock-free ring queue data structure, combined with hardware cache line alignment and hardware-level atomic operations, to realize conflict-free concurrent access in a multi-threaded environment. To ensure the atomicity of multi-threaded operations, a mechanism called "compare-and-swap" (CAS) is used to safely update the head and tail pointers of the queue.

[0033] The operation flow details are as follows: when a thread performs an insertion operation, it first reads the current tail pointer value as the expected value, then calculates the new tail pointer position, and finally atomically attempts to update the tail pointer through the CAS operation. Only when the expected value is consistent with the current value, i.e., no other thread has modified the pointer update during this period, the update will be successful, otherwise the process will be retried. The read operation also uses a similar mechanism to update the head pointer.

[0034] This lock-free design avoids the context switching and performance overhead that may be caused by using traditional mutexes (Mutexes), thereby ensuring the rapid response and immediate processing of high-priority tasks.

[0035] L1 level buffer: As a medium priority task buffer, it is used to store commonly used task data, such as periodic service requests or medium important state update information; L1 level buffer adopts a priority-aware scheduling mechanism.

[0036] When performing scheduling, the mechanism will first check and process the tasks in the L0 level buffer. Only when the higher priority queue is empty, the scheduling system will process the tasks in the L1 level buffer.

[0037] Through this pre-emptive processing logic, it ensures that high priority data can be processed first, while maintaining the overall throughput efficiency of the system.

[0038] L2 level buffer: As a low priority persistent buffer, it is mainly used to store data that does not require high timeliness but needs long-term preservation, such as system running logs, historical state records, etc.

[0039] In order to improve storage efficiency and reduce CPU burden, L2 level buffer can adopt zero-copy persistent technology, which realizes efficient data writing through memory mapping (mmap) method, writes data directly from kernel space buffer to persistent storage medium, avoiding additional data copying between kernel space and user space.

[0040] Please refer to Figure 3 The task scheduling module, the scheduling mode of the module mainly depends on two trigger mechanisms, which together constitute an event-driven mechanism: one is external instruction triggering, the system receives remote instructions through TCP or UDP protocol, supports starting, pausing, terminating and updating of storage tasks and other control commands, this mode has the highest priority; the second is data volume triggering, when the data volume of any buffer reaches a preset threshold, the system automatically triggers data storage operation, this process does not require additional instruction intervention, please refer to Figure 3 The diamond judgment box "data volume ≥ threshold?".

[0041] Please refer to Figure 4 Dynamic capacity adjustment module: the embodiment discloses a buffer dynamic capacity adjustment method based on machine learning, which is one of the core innovations of the present application.

[0042] Machine learning model, in order to avoid the risk of "abstract algorithm", the prediction model in the present application is a kind of embedded lightweight hybrid neural network, which is deployed in the kernel module of real-time operating system or tightly coupled user state service.

[0043] Please refer to Figure 4As shown, the prediction model specifically adopts a lightweight hybrid network structure, which organically combines multiple neural network components. Among them, the convolutional neural network (CNN) component is used to extract local spatial features in the data stream, the long short-term memory network (LSTM) component is used to capture its time series dependence, and the Transformer component is responsible for processing long-distance traffic pattern dependence.

[0044] The outputs of each component are fused via a fully connected layer to finally generate the capacity adjustment instruction. The model receives specific physical quantities as input features, including the fill rate of each level of buffer, the time interval of task arrival, the variance of packet size, etc., and its output action is to generate specific expansion or contraction instructions, such as the instruction L0 buffer expansion by 150% or the instruction L2 buffer release by 50% of memory.

[0045] Execution link, the dynamic capacity adjustment module issues capacity adjustment instructions to the data buffer module in real time through the aforementioned pipeline communication method. In a preferred embodiment, the data structure of the instruction can include target buffer level (L0 / L1 / L2), capacity adjustment ratio, and effective timestamp, etc. information, wherein the effective timestamp may, for example, be the start of the next scheduling period.

[0046] Decision algorithm and process: data collection, during system operation, periodically collect key load indicators of the buffer, including L0 / L1 / L2 fill rate, task arrival frequency, packet size distribution, etc., wherein the fill rate may, for example, be 1ms for the sampling window.

[0047] Model prediction, the system inputs the collected feature information into the trained machine learning model according to the preset time interval to predict the traffic trend in the future period, wherein the future period may, for example, be 10ms.

[0048] Expansion and contraction decision, in the expansion and contraction decision phase, that is, the link shown in the diamond judgment box in the flow chart, the system will calculate the flow fluctuation rate FR through the following formula (1), and when FR is greater than the preset threshold, the expansion strategy is triggered. Figure 4

[0049] (1)

[0050] In this formula, represents the task arrival frequency, represents the packet size distribution, represents the overall buffer fill rate. It should be emphasized that the weight coefficients appearing in the formula, such as , ​For example, the parameters can be configured as non-limiting example values of 0.5, 0.3, etc. In actual applications, these parameters can be dynamically optimized by a system calibration module, which does not constitute a limitation of the present application.

[0051] Model optimization: The system continuously monitors the effects of buffer adjustment and records actual load data. By calculating key evaluation indicators such as mean absolute percentage error (MAPE) and coefficient of determination (R²), the system dynamically judges the prediction performance of the current model. When the error exceeds the set threshold, the system will trigger an online learning mechanism to iteratively optimize the model parameters.

[0052] Data storage module and system self-checking module: The data storage module is responsible for writing processed data to persistent storage media, such as a disk.

[0053] In a preferred embodiment, the module uses a ping-pong buffer writing method. This is a technique for optimizing data writing process by alternating the use of two or more buffers.

[0054] When data is written to the currently idle buffer A, another buffer B can be used for data reading or waiting at the same time. When buffer A completes writing, the system switches roles, setting buffer B as the writing buffer and buffer A as the reading or standby buffer.

[0055] This cycle continues, thus achieving parallel execution of data writing and upstream data processing operations, improving data storage efficiency and reducing disk write latency.

[0056] The system self-checking module is responsible for real-time monitoring of disk status, including remaining space, read-write health, etc. When insufficient disk space or abnormalities are detected, the system can trigger an alarm or adjust the storage strategy in a timely manner to ensure the stability and reliability of data storage.

[0057] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application should not be limited to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered within the scope of protection of the present application.

Claims

1. A data storage optimization method based on a real-time operating system, characterized by, The method comprises the following steps: Step 1, data buffering: receiving task data through a data buffering module, the data buffering module adopts a multi-level buffering architecture, and according to the preset type and priority of the task data, the task data is allocated to the buffer area of different priority in the multi-level buffering architecture; Step 2, task scheduling: through a task scheduling module, an event-driven mechanism is adopted to dynamically schedule the storage task for the task data according to external instructions or a preset data volume threshold; Step 3, dynamic capacity adjustment: through a dynamic capacity adjustment module, a prediction model is used to predict the future data flow trend, and the capacity of each buffer area in the multi-level buffering architecture is adjusted in real time according to the prediction result, so that the buffer area resource configuration is dynamically matched with the predicted data flow trend; The dynamic capacity adjustment step specifically comprises: Periodically collecting key load indicators of each buffer area in the multi-level buffering architecture, the key load indicators including the filling rate, task arrival frequency and data packet size distribution of each level buffer area; The collected key load indicators are input into the prediction model, and the prediction model is an embedded lightweight hybrid neural network; The prediction model generates a capacity adjustment instruction, and the data structure of the capacity adjustment instruction includes: target buffer area level, capacity adjustment ratio and effective timestamp; The dynamic capacity adjustment module adjusts the capacity of the corresponding level buffer area at the effective timestamp according to the capacity adjustment instruction; Step 4, data storage: through a data storage module, the task data processed by the buffer area is written into a persistent storage medium.

2. The data storage optimization method based on a real-time operating system according to claim 1, characterized in that, The data storage step specifically adopts a ping-pong buffering writing method, at least two buffer areas are alternately used to enable the data writing operation to be performed in parallel with the upstream data processing operation.

3. The data storage optimization method based on real-time operating system according to claim 1, characterized in that, The multi-level buffering architecture includes: L0 level buffer for storing high-priority real-time data, the L0 level buffer adopts a lock-free ring queue; L1 level buffer for storing medium-priority task data, the L1 level buffer adopts a priority-aware scheduling mechanism; and L2 level buffer for storing low-priority persistent data, the L2 level buffer adopts a zero-copy technology to write data through memory mapping.

4. The data storage optimization method based on a real-time operating system according to claim 3, characterized in that, The lock-free ring queue realizes pointer update through hardware-level atomic operation, specifically adopting a compare-and-swap (CAS) mechanism, wherein: in the insertion operation, the expected value of the tail pointer is compared, and if consistent, the tail pointer is atomically updated and the data is written; in the reading operation, the expected value of the head pointer is compared, and if consistent, the head pointer is atomically updated and the data is removed.

5. The method for data storage optimization based on real-time operating system according to claim 1, characterized in that, The prediction model adopts a lightweight hybrid network structure, including a convolutional neural network component, a long short-term memory network component and a Transformer component.

6. The method of claim 1, wherein the method is based on a real-time operating system. The dynamic capacity adjustment step further comprises: judging the expansion and contraction capacity demand through a flow fluctuation rate (FR) index, the flow fluctuation rate (FR) index is calculated by weighting according to the statistical characteristics of task arrival frequency, data packet size distribution and overall buffer area filling rate through a preset weight coefficient.

7. The method for data storage optimization based on real-time operating system according to claim 1, characterized in that, The event-driven mechanism comprises: responding to external Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) instructions to trigger task scheduling; and / or automatically triggering a storage task when the amount of data in any buffer reaches a preset threshold.

8. A real-time operating system based data storage optimization system, characterized in that, The method comprises: a data buffering module configured to adopt a three-level buffering architecture comprising L0-level buffering, L1-level buffering and L2-level buffering; a task scheduling module configured to dynamically schedule storage tasks using an event-driven mechanism; a dynamic capacity adjustment module connected to the data buffering module and configured to predict future data flow trends using a prediction model and to issue real-time capacity adjustment instructions to the data buffering module based on the prediction results through pipeline communication, so that the capacities of the buffer areas in the three-level buffering architecture are adapted to the data flow trends; characterized in that the dynamic capacity adjustment module is configured to: periodically collect key load indicators of the buffer areas in the three-level buffering architecture, the key load indicators comprising the filling rate, task arrival frequency and data packet size distribution of each level of buffering area; input the collected key load indicators into the prediction model, which is an embedded lightweight hybrid neural network; and generate the capacity adjustment instructions according to the predicted flow trends, the data structure of the capacity adjustment instructions including: target buffer area level, capacity adjustment ratio and effective time stamp; a data storage module configured to write data into a persistent storage medium.

9. The real-time operating system based data storage optimization system of claim 8, wherein, The data storage module is configured to use a ping-pong buffering writing method to enable data writing and processing operations to be performed in parallel.

10. The real-time operating system based data storage optimization system of claim 8, wherein, The prediction model integrated in the dynamic capacity adjustment module adopts a lightweight hybrid network structure, including a convolutional neural network component, a long short-term memory network component and a Transformer component.

Images