Method, system and equipment for adjusting and optimizing log stream of log file system

Abstract

The invention relates to a log stream adjustment optimization method, system and device of a log file system, and relates to the technical field of storage system optimizing.The method comprises the steps that load key index information is periodically obtained from a maintained log stream pool for weighting processing, and an adjustment interval of a current period is determined, then log stream dynamic adjustment of load balancing is carried out according to a multi-level adjustment strategy corresponding to the adjustment interval, and when write-back control optimization is carried out, fine-grained division is carried out according to different types of write-back tasks in the constructed initial write-back queue to obtain multiple queues used for carrying out write-back of the write-back tasks; and performing write-back control optimization on each to-be-written-back task in the multiple queues to obtain a log flow adjustment optimization result. According to the method and the device, the log stream is adjusted according to the load change in a self-adaptive mode, and excessive dependence on the same queue lock is reduced through fine-grained refinement and localization of the queue, so that the concurrency efficiency and the adaptability of the log file system in the multi-core environment are improved.

Description

Technical Field

[0001] This application relates to the technical field of storage system optimization, and particularly to a method, system, and device for optimizing the log stream of a log file system. Background Art

[0002] With the popularization of multi-core architectures and high-performance storage devices (such as NVMe SSDs), file systems often face problems such as insufficient parallelism and decreased write efficiency in high-concurrency scenarios. Many existing log file systems use a single log stream or a fixed number of log streams, and cannot achieve dynamic optimization according to load changes. Higher core counts also lead to serious lock contention in traditional global write-back control (such as a single global waiting queue), resulting in a significant decrease in system throughput and response performance.

[0003] For example, mainstream global log mechanisms (such as the journaling module of some general file systems) can only provide a single or a fixed number of log streams for recording. Existing technical solutions involve allocating separate log streams for each core, but this also causes problems such as scattered writes and severe fragmented I / O. In addition, if the write-back control layer relies on a single global queue, it is prone to bottlenecks in a multi-core environment. Therefore, the lack of adaptive log stream management and the lack of write-back control friendly to concurrency are the main obstacles to improving the concurrency performance of log file systems in the current multi-core architecture. It can be seen that the existing technology lacks adaptability in the multi-core environment for log file systems and has low parallel efficiency. Summary of the Invention

[0004] This application provides a method, system, and device for optimizing the log stream of a log file system. In terms of log stream adjustment, by dynamically adjusting the log stream and reducing fragmented I / O, it is possible to balance log parallelism and write merging efficiency in a high-concurrency environment, and reduce fragmented writes caused by scattered small requests. In terms of optimizing write-back control, by using write-back control friendly to concurrency, it is possible to improve the problem of competition in the global write-back queue, reduce the excessive dependence on the same lock in multiple high-concurrency scenarios, and improve the concurrency efficiency of log submission and write-back, thereby solving the problems of the lack of adaptability of the existing technology in the multi-core environment for log file systems and low parallel efficiency.

[0005] In a first aspect, this application provides a method for optimizing the log stream of a log file system, which is applied to a concurrent log file system with a multi-core architecture, and includes:

[0006] Periodically obtain load key metric information from the maintained log stream pool in the background of the log file system through a preset lightweight monitoring strategy, where the load key metric information includes transaction commit time, queue length, balance, resource utilization rate, and throughput;

[0007] Perform weighted processing based on the load key index information, and determine the adjustment range of the log file system in the current cycle based on the weighted result;

[0008] Perform dynamic adjustment of the log flow for load balancing according to the multi-level adjustment strategy corresponding to the adjustment range, and obtain the log flow adjustment result, where the log flow adjustment result includes the log flow increase adjustment result or the log flow offline adjustment result;

[0009] For the log flow adjustment result, perform fine-grained division according to different types of write-back tasks in the constructed initial write-back queue to obtain multiple queues for writing back write-back tasks, where the multiple queues include tasks to be written back;

[0010] Optimize the write-back control for each task to be written back in the multiple queues to obtain the log flow adjustment optimization result;

[0011] Among them, the write-back control optimization process includes local scheduling processing and dynamic window adjustment processing.

[0012] Optionally, through a preset lightweight monitoring strategy, in the background of the log file system, periodically obtain load key index information from the maintained log flow pool, including:

[0013] When the log file system is initialized and loaded, pre-allocate at least one group of log flows with a preset initial state that can be dynamically managed for a multi-core processor, and maintain a log flow pool created for the allocation of log flow resources. Each log flow independently executes transaction records;

[0014] Through a preset lightweight detection strategy, periodically monitor the log flows of the log file system, capture the load changes of each log flow from the log flow pool, and obtain load key index information;

[0015] Among them, the load key index information is stored in the circular buffer area of the log file system and is used for analysis.

[0016] Optionally, capture the load changes of each log flow from the log flow pool to obtain load key index information, including:

[0017] Determine the log flow queue corresponding to each log flow, and count the number of transactions to be committed or the depth of the request queue for each active log in the log flow queue to obtain queue length information;

[0018] Extract the average commit delay in the most recent or a period of time from the transaction records as the transaction commit time;

[0019] Work measurement is performed on each log stream in the log stream queue to obtain the workload difference degree between the log streams, which is used as the balance degree. In addition, resource utilization information is obtained to analyze the resource utilization rate.

[0020] The regulation result is analyzed according to the number of transactions processed per unit time to obtain the throughput.

[0021] Optionally, weighted processing is performed according to the load key index information, and the adjustment interval of the log file system in the current period is determined based on the weighted result, including:

[0022] Obtain multi-index weighted information, and perform normalization or mapping processing on each index item in the load key index information to obtain a mapping function. The multi-index weighted information includes the weight coefficients corresponding to each index item in the load key index information.

[0023] Within a preset time interval, the mapping functions corresponding to each index item are weighted according to the weight coefficients to obtain a weighted result, which includes index quantization information for characterizing the dynamic measurement of the global load level and balance degree.

[0024] Multi-level dynamic interval classification is performed according to the index quantization information, and the change information of the log stream load is continuously monitored during the dynamic interval classification for dynamic adjustment to obtain the adjustment interval.

[0025] Among them, the adjustment interval includes a low load area, a medium load area, and a high load area.

[0026] Optionally, within a preset time interval, the mapping functions corresponding to the index items are weighted according to the weight coefficients to obtain a weighted result, including:

[0027] According to S = w q × f q + w t × f t + w i × f i + w cpu × f cpu , determine the index quantization information.

[0028] Among them, S is the comprehensive score, representing the index quantization information, w q is the weight of the queue length index, w t is the weight of the transaction submission time index, w i is the weight of the balance degree index, w cpu is the weight of the resource utilization rate index, f q , f t , f i and f cpuIt is a normalization or mapping function for each indicator, used to ensure that the weighted results of each indicator are within a comparable range.

[0029] Optionally, perform dynamic adjustment of the log flow for load balancing according to the multi-level adjustment strategy corresponding to the adjustment interval, and obtain the log flow adjustment result, including:

[0030] Identify the log flow adjustment information based on the adjustment interval, and select a multi-level adjustment strategy according to the log flow adjustment information. The multi-level adjustment strategy includes a transaction allocation strategy and a safe offline strategy;

[0031] When the log flow adjustment information is log flow increase information, select a new log flow from the log flow pool, and perform warm-up initialization and queue core allocation processing on the new log flow;

[0032] According to the log flow increase information, use the transaction allocation strategy to bind the submitted transaction to the new log flow, and perform dynamic adjustment of load balancing on the new log flow until the new log flow tends to be balanced, and obtain the log flow increase adjustment result;

[0033] When the log flow adjustment information is log flow decrease information, use the safe offline strategy to freeze the target log flow, and the target log flow is the log flow with the lowest load;

[0034] After the target log flow completes the currently executed transaction, perform persistent processing and data consistency verification processing on the unsubmitted data of the target log flow, and mark the target log flow as an idle state based on the data consistency verification result, and obtain the log flow offline adjustment result.

[0035] Optionally, for the log flow adjustment result, perform fine-grained division according to different types of write-back tasks in the constructed initial write-back queue, and obtain multiple queues for performing write-back tasks of the write-back tasks, including:

[0036] For the log flow adjustment result, construct an initial write-back queue. The initial write-back queue contains write-back tasks, and each write-back task contains a corresponding task type. The task type includes input / output type and resource node type;

[0037] Separate the log write-back of the write-back task based on the input / output type from the background dirty page refresh and the direct write-back of the application program, and correspond the core or node of the write-back task to the local write-back queue based on the resource node type, and obtain multiple queues corresponding to different task types.

[0038] Optionally, perform write-back control optimization on each write-back task in the multiple queues to obtain the log flow adjustment optimization result, including:

[0039] Perform local optimization based on each of the tasks to be written back in the multiple queues, and perform dynamic window adjustment based on each association queue in the multiple queues to obtain the optimized result of log stream adjustment;

[0040] Among them, the local optimization process includes ensuring that the tasks to be written back only operate on the tasks in the corresponding queue in each stage, and ensuring that there is no mutual exclusion block between different queues. The dynamic window adjustment process includes setting a concurrency limit for each write-back queue and performing real-time adjustment according to the performance of the storage device of the log file system.

[0041] In a second aspect, the present application provides a log stream adjustment optimization system, including:

[0042] An index information periodic monitoring module, configured to periodically obtain load key index information from the maintained log stream pool in the background of the log file system through a preset lightweight monitoring strategy, where the load key index information includes transaction submission time, queue length, balance, resource utilization rate, and throughput;

[0043] A weighted processing module, configured to perform weighted processing according to the load key index information and determine the adjustment interval of the log file system in the current period based on the weighted result;

[0044] A dynamic adjustment module for load balancing, configured to perform dynamic adjustment of log flow for load balancing according to the multi-level adjustment strategy corresponding to the adjustment interval to obtain a log flow adjustment result, where the log flow adjustment result includes a log flow increase adjustment result or a log flow offline adjustment result;

[0045] A write-back task division module, configured to perform fine-grained division on the log flow adjustment result according to each different type of write-back task in the constructed initial write-back queue to obtain a multiple queue for performing write-back of the write-back task, where the multiple queue includes tasks to be written back;

[0046] A write-back control optimization module, configured to perform write-back control optimization on each of the tasks to be written back in the multiple queue to obtain the optimized result of log stream adjustment;

[0047] Among them, the write-back control optimization process includes local scheduling processing and dynamic window adjustment processing.

[0048] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus;

[0049] The memory is used to store a computer program;

[0050] A processor, when executing a program stored in a memory, implements the steps of the log stream adjustment and optimization method of the log file system according to any one of the embodiments of the first aspect.

[0051] In summary, when dynamically adjusting the log stream in the embodiments of the present application, the system periodically obtains load key metric information from the maintained log stream pool for weighted processing, determines the adjustment interval of the current period, and then performs load-balanced log flow dynamic adjustment according to the multi-level adjustment strategy corresponding to the adjustment interval. When optimizing the write-back control, according to the different types of write-back tasks in the constructed initial write-back queue, fine-grained division is performed to obtain multiple queues for writing back the write-back tasks, so as to optimize the write-back control of each write-back task in the multiple queues and obtain the log stream adjustment and optimization result. The purpose of the present application is to improve the parallel efficiency and adaptability of the log file system in a multi-core environment. Aiming at problems such as fixed log stream and global queue lock competition, two improvement objectives are proposed, including adjusting the log stream according to load changes in an adaptive manner, and reducing the over-reliance on the same queue lock by fine-grained refinement and localization of queues. It can be seen that the present application improves the concurrent efficiency and adaptability of the log file system in a multi-core environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 It is a schematic flowchart of a method for adjusting and optimizing the log stream of a log file system provided by an embodiment of the present application;

[0055] Figure 2 It is a schematic flowchart of the steps of a method for adjusting and optimizing the log stream of a log file system provided by an optional embodiment of the present application;

[0056] Figure 3 It is a schematic diagram of the call sequence between an adjustment and optimization process of a log stream and modules provided by an optional example of the present application;

[0057] Figure 4 It is a block diagram of the structure of a log stream adjustment and optimization system provided by an embodiment of the present application;

[0058] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0059] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are only a part rather than all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0060] For the convenience of understanding the embodiments of the present application, further explanatory descriptions will be given below with reference to the accompanying drawings and specific embodiments. The embodiments do not constitute a limitation to the embodiments of the present application.

[0061] Figure 1 FIG. is a schematic flowchart of a method for optimizing log stream regulation of a log file system provided in an embodiment of the present application. In specific implementation, the method for optimizing log stream regulation of the log file system provided in the embodiment of the present application can be mainly applied to a concurrent log file system with a multi-core architecture. By improving the log layer and the write-back control layer of the file system respectively, the system can balance parallel log writing and efficient merging in a high-concurrency scenario, reduce global lock contention and improve the overall throughput. The method for optimizing log stream regulation provided in this embodiment can specifically include the following steps:

[0062] Step 110: Periodically obtain load key metric information from the maintained log stream pool in the background of the log file system through a preset lightweight monitoring strategy.

[0063] Among them, the load key metric information includes transaction commit time, queue length, balance, resource utilization rate, and throughput.

[0064] In this embodiment, the lightweight monitoring strategy refers to a lightweight monitoring logic. In this embodiment, the lightweight monitoring logic is embedded in the log file system, such as in the I / O (Input / Out) write path of the log file system (including: fsync(), journal_submit_commit_record()). The load key metric information includes the usage conditions of various resources in the log file system. Among them, the queue length refers to the log stream queue length; the balance refers to the balance between streams; the resource utilization rate includes CPU and / or I / O utilization rate; the throughput refers to the global log throughput.

[0065] In a specific implementation, this embodiment can preset lightweight monitoring logic in the log file system in advance. When the log file system is loaded, it can periodically and repeatedly monitor and collect corresponding key metrics from the maintained log stream pool as load key metric information. Among them, the log stream pool can contain multiple log streams, each of which can independently execute transaction records. The log stream pool can allocate log stream resources as needed. This embodiment monitors each log stream in the log stream pool to obtain subsequent load changes.

[0066] It should be noted that the log stream adjustment and optimization method provided in this embodiment does not depend on a specific operating system version or file system log format, nor does it require special hardware. It can be combined with other file system structures or operating systems according to actual needs. It is applicable to multi-core CPUs and various types of storage devices (such as different types of storage devices such as HDD, SSD, NVMe, etc.), and at the same time retains a large degree of flexibility for later performance tuning.

[0067] Step 120: Perform weighted processing on the load key metric information, and determine the adjustment interval of the log file system in the current cycle based on the weighted result.

[0068] In related technologies, in technical fields such as log stream adjustment, database resource management and allocation, and cluster resource management and allocation, load balancing is usually involved. However, when performing load balancing in the prior art, multiple metric thresholds are simply preset, and then the obtained multiple load metric data is compared with the metric thresholds to perform load balancing adjustment according to the comparison situation. However, the resource load is dynamically changing in real time. Using fixed thresholds for load balancing adjustment cannot fully adapt to dynamic changes, and it cannot also take into account the requirements of log parallelism and write merge efficiency in high-concurrency scenarios, lacking adaptability.

[0069] To solve the above problems, this embodiment presets weights for each load metric in the obtained load key metric information, and pre-sets different adjustment intervals, where different metrics can correspond to different weights. After obtaining the load key metric information in the current cycle, use the weights to perform weighted processing on each load metric, so as to obtain a weighted result that can best reflect the resource load situation in the current cycle. Then, use the weighted result obtained in the current cycle to match the adjustment interval to obtain the adjustment interval that best adapts to the load situation in the current cycle.

[0070] Step 130: Perform dynamic adjustment of the log flow for load balancing according to the multi-level adjustment strategy corresponding to the adjustment interval to obtain a log flow adjustment result.

[0071] Among them, the log flow adjustment result includes a log flow increase adjustment result or a log flow offline adjustment result.

[0072] In a specific implementation, this embodiment can preset multiple adjustment strategies. As a multi-level adjustment strategy, different adjustment intervals correspond to different adjustment strategies. Specifically, in each period, the corresponding adjustment interval is determined according to the load condition, and the corresponding adjustment strategy is selected according to the adjustment interval, and the adjustment strategy is executed to perform dynamic adjustment of the log flow. Through adaptive log flow adjustment, this embodiment realizes load balancing and increases the adaptability of the system to dynamic loads.

[0073] Exemplarily, the adjustment intervals can be at least divided into a low-load area, a medium-load area, and a high-load area. When the system load enters the high-load area, it indicates that the parallelism of the log flow or the write-back ability is insufficient. At this time, the log flow can be increased or the concurrency of the write-back queue can be improved to avoid a bottleneck formed by a single log flow under high concurrency, and a log flow increase adjustment result is obtained; when the system load enters the high-load area, it indicates that too many log flows cause scattered writes or resource waste. At this time, the log flow can be reduced and some log flows can be safely taken offline to avoid scattered writes and decreased merge efficiency caused by too many log flows, and a log flow offline adjustment result is obtained.

[0074] Thus, in each period, through adaptive log flow adjustment, this embodiment can automatically match the load requirements in a multi-core environment, achieve an appropriate degree of log parallelism. Compared with the existing fixed log flow scheme or the scheme that simply compares the threshold according to the load condition to adjust resource allocation, this embodiment greatly enhances the adaptability of the system to dynamic loads.

[0075] Step 140: For the log flow adjustment result, perform fine-grained division according to different types of write-back tasks in the constructed initial write-back queue to obtain multiple queues for performing write-back tasks.

[0076] Among them, the multiple queues include write-back tasks to be written back.

[0077] In the related art, the existing write-back mechanism usually adopts a global waiting queue. For example, for each write-back task in the write-back queue, the write-back tasks are executed in sequence according to the time when they enter the queue or the priority of the write-back tasks. This write-back mechanism of the global waiting queue easily causes write-back tasks to wait for a long time, and multiple CPUs (processors) compete for the lock (wbt_wait), thereby affecting the parallel efficiency and resulting in low parallel efficiency.

[0078] In order to solve the problems caused by the above-mentioned global waiting queue, this embodiment constructs an initial write-back queue for the log stream adjustment result, and the initial write-back queue contains different types of write-back tasks, and then performs fine-grained division according to each write-back task in the initial write-back queue. Specifically, the fine-grained division mainly includes but is not limited to: division by task type or division by core number. Therefore, this embodiment splits the traditional global write-back queue into finer-grained queues to obtain multiple queues. The multiple queues usually contain a corresponding write-back queue for each type of write-back task, so that only the corresponding queue is operated when the I / O is completed or awakened, thereby reducing lock contention, that is, avoiding the problem of global lock contention and improving parallel efficiency.

[0079] Step 150 , performing write-back control optimization on each of the tasks to be written back in the multiple queues to obtain a log stream regulation optimization result.

[0080] The write-back control optimization process includes local scheduling processing and dynamic window adjustment processing adapted to global queue lock competition.

[0081] In the specific implementation, this embodiment mainly uses the write-back scheduling strategy optimization to achieve write-back control optimization for each write-back task. The write-back control optimization process mainly consists of local scheduling processing and dynamic window adjustment processing. Specifically, for each task to be written back in the queue, a local call-out process is first performed, and a local scheduling mechanism is used to wake up only the thread without affecting the global write-back. Then, a dynamic window adjustment process is performed, and a dynamic window mechanism is used to limit the amount of concurrent write-backs to avoid I / O overload, thereby completing the adjustment and optimization of the log stream during load and write-back.

[0082] It can be seen that when the embodiment of the present application dynamically adjusts the log stream, the system periodically obtains the key load indicator information from the maintained log stream pool for weighted processing, and determines the adjustment interval of the current cycle, and then dynamically adjusts the load-balanced log stream according to the multi-level adjustment strategy corresponding to the adjustment interval. When performing write-back control optimization, fine-grained division is performed according to the different types of write-back tasks in the constructed initial write-back queue, and multiple queues for writing back write-back tasks are obtained, so as to optimize the write-back control of each task to be written back in the multiple queues, and obtain the log stream adjustment optimization result. The present application aims to improve the parallel efficiency and adaptability of the log file system in a multi-core environment, and improves the log layer and write-back control layer of the file system respectively. Specifically, in response to the problems of the existing technology such as the lack of adaptive log stream management and concurrency-friendly write-back control, two major improvement goals are proposed, including adjusting the log stream according to load changes in an adaptive manner, and reducing excessive reliance on the same queue lock through fine-grained refinement and localized queues. This enables the system to take into account both parallel log writing and efficient merging in high-concurrency scenarios, reduce global lock contention and improve overall throughput, thereby improving the concurrency efficiency and adaptability of the system in a multi-core environment.

[0083] Reference Figure 2 , a schematic flowchart of steps of a method for optimizing log stream regulation of a log file system provided by an optional embodiment of the present application is shown, which is applied to a concurrent log file system of a multi-core architecture. The method may specifically include the following steps:

[0084] Step 210, through a preset lightweight monitoring policy, periodically obtain load key metric information from a maintained log stream pool in the background of the log file system.

[0085] Among them, the load key metric information includes transaction commit time, queue length, balance, resource utilization rate, and throughput.

[0086] Optionally, the above-mentioned obtaining load key metric information from a maintained log stream pool in the background of the log file system through a preset lightweight monitoring policy may include the following sub-steps:

[0087] Sub-step 2101, when the log file system is initialized and loaded, pre-allocate at least one group of log streams with a preset initial state that can be dynamically managed for a multi-core processor, and maintain a log stream pool created for the allocation of log stream resources, and each of the log streams independently executes transaction recording.

[0088] Sub-step 2102, through a preset lightweight detection policy, periodically monitor the log streams of the log file system, capture the load changes of each log stream from the log stream pool, and obtain load key metric information;

[0089] Among them, the load key metric information is stored in a circular buffer area of the log file system and is used for analysis.

[0090] A unified description of sub-step 2101 - sub-step 2102:

[0091] In a specific implementation, when the system is initialized, a selectable log stream structure can be reserved for several cores. When the file system is loaded, a group of log streams that can be dynamically managed are pre-allocated for multiple CPUs. Each log stream can independently execute transaction recording. The initial state of the log stream can be set to states such as "idle" or "available", and then a log stream pool can be created through a log stream management module in the system and the log stream pool can be maintained. The log stream pool can be used to allocate log stream resources on demand. This embodiment mainly uses a multi-metric monitoring and comprehensive decision-making method to dynamically adjust the log stream, thereby achieving load balancing. Specifically, lightweight monitoring logic can be embedded at the I / O write path of the system to periodically and repeatedly obtain key metrics as load key metrics, and store the load metric information in a circular buffer for analysis.

[0092] In an alternative embodiment, capturing the load changes of each log stream from the log stream pool to obtain load key metric information may specifically include: determining the log stream queues corresponding to each log stream, and counting the number of transactions to be committed or the request queue depth of each active log in the log stream queue to obtain queue length information; extracting the average commit latency for the most recent time or a period of time from the transaction records as the transaction commit time; measuring the work of each log stream in the log stream queue to obtain the workload difference degree between each log stream as the balance degree, and obtaining resource utilization information to analyze the resource utilization rate; analyzing the regulation result based on the number of transactions processed per unit time to obtain the throughput.

[0093] Specifically, the log stream queue length is usually obtained by counting the number of transactions to be committed or the request queue depth of each active log stream, such as flow_queue_length[i]; the transaction commit time refers to the average commit latency for the most recent time or a period of time recorded, such as avg_commit_time; the balance degree between streams is obtained by measuring the workload difference degree (such as standard deviation or range) between different log streams, such as imbalance_factor; the CPU or I / O utilization rate refers to the CPU core occupancy and IOPS utilization rate, which are optional metrics; the global log throughput refers to the number of transactions processed per unit time for viewing the regulation result, such as journal_throughput.

[0094] Step 220: Obtain multi-metric weighted information, and perform normalization or mapping processing on each metric item in the load key metric information to obtain a mapping function.

[0095] Among them, the multi-metric weighted information includes the weight coefficients corresponding to each metric item in the load key metric information.

[0096] Step 230: Within a preset time interval, perform weighted processing on the mapping functions corresponding to each metric item according to the weight coefficients to obtain a weighted result.

[0097] Among them, the weighted result includes metric quantization information used to characterize the dynamic measurement of the global load level and balance degree.

[0098] A unified description of steps 220 to 230:

[0099] In a specific implementation, to avoid the problem of insufficient parallel efficiency and adaptability of the log file system caused by only using a threshold for load judgment, this embodiment uses a multi-index weighting strategy, presets corresponding weights for different load indicators, and performs weighted processing on each key load indicator. Among them, each weight coefficient includes but is not limited to: the weight of the log queue length indicator, the weight of the average submission time indicator, the weight of the imbalance degree between flows indicator, and the CPU utilization rate weight.

[0100] In this embodiment, to ensure that the weighted result obtained after weighting is within a comparable range, so that the weighted result can reasonably correspond to the actual load interval. For each key load indicator, through processing such as normalization or mapping, a mapping function corresponding to the indicator can be obtained. When performing weighting, the mapping function corresponding to each key indicator is used in combination with the corresponding weight for weighted processing to obtain a weighted result, and this weighted result can include a comprehensive score.

[0101] Furthermore, the resource loads in the log file system are usually variable. To adapt to the load variation situation, this embodiment presets an interval time T, and the system can calculate the comprehensive score every T seconds.

[0102] In an actual implementation, to be able to reasonably and dynamically measure the global load level and balance degree, this embodiment introduces a formula and calculates the comprehensive score using the formula every T seconds. Optionally, within the preset time interval, weighted processing is performed on the mapping function corresponding to the indicator item according to the weight coefficient to obtain a weighted result, which specifically may include: according to S = w q ×f q +w t ×f t +w i ×f i +w cpu ×f cpu , determine the index quantization information; where S is the comprehensive score, representing the index quantization information, w q is the weight of the queue length indicator, w t is the weight of the transaction submission time indicator, w i is the weight of the balance degree indicator, w cpu is the weight of the resource utilization rate indicator, f q , f t , f i and f cpu are the normalization or mapping functions of each indicator, used to ensure that the weighted results of each indicator are within a comparable range. Among them, f q is the normalization or mapping function of the log stream queue length of the indicator item; f t is the normalization or mapping function of the transaction submission time (average submission time) of the indicator item; f i is the normalization or mapping function of the imbalance degree between flows of the indicator item; fcpu It is a normalization or mapping function for the resource utilization rate of the index item.

[0103] Step 240: Perform multi-level dynamic interval classification according to the index quantization information, and continuously monitor the change information of the log stream load during the dynamic interval classification process for dynamic adjustment to obtain an adjustment interval.

[0104] Among them, the adjustment interval includes a low-load area, a medium-load area, and a high-load area.

[0105] In a specific implementation, this embodiment can divide the weighted result into several levels and divide it into different load areas. Through a preset load value, such as T low 、T high , match the load condition of the system in the current cycle with the adjustment interval. Exemplarily, low-load area: S ≤ T low ; medium-load area: T low < S < T high ; high-load area: S ≥ T high .

[0106] In actual processing, the log stream load usually changes continuously. To improve the adaptability of the system, during each dynamic interval classification process, this embodiment also continuously monitors the change information of the log stream load, determines the change situation according to the continuously changing log stream load, dynamically adjusts the index quantization information, and determines the adjustment interval that is most suitable for adjusting the log stream load within the current cycle.

[0107] Step 250: Perform dynamic adjustment of the log flow for load balancing according to the multi-level adjustment strategy corresponding to the adjustment interval to obtain a log flow adjustment result.

[0108] Among them, the log flow adjustment result includes a log flow increase adjustment result or a log flow offline adjustment result.

[0109] In an alternative embodiment, this embodiment performs dynamic adjustment of the log flow for load balancing according to the multi-level adjustment strategy corresponding to the adjustment interval, and obtains the log flow adjustment result, which may specifically include: identifying log flow adjustment information based on the adjustment interval, and selecting a multi-level adjustment strategy according to the log flow adjustment information, where the multi-level adjustment strategy includes a transaction allocation strategy and a safe off-line strategy; when the log flow adjustment information is log flow increase information, selecting a new log flow from the log flow pool, and performing warm-up initialization and queue core allocation processing on the new log flow; according to the log flow increase information, using the transaction allocation strategy to bind the submitted transaction to the new log flow, and performing dynamic adjustment of load balancing on the new log flow until the new log flow tends to be balanced, obtaining the log flow increase adjustment result; when the log flow adjustment information is log flow decrease information, using the safe off-line strategy to freeze the target log flow, where the target log flow is the log flow with the lowest load; after the target log flow completes the currently executed transaction, performing persistent processing and data consistency verification processing on the unsubmitted data of the target log flow, and marking the target log flow as an idle state based on the data consistency verification result, obtaining the log flow off-line adjustment result.

[0110] In a specific implementation, the system executes the corresponding adjustment strategy according to the adjustment interval where the weighted result is located every period. For example, when entering the high load area, it indicates that the log flow parallelism or write-back ability is insufficient. At this time, log flows can be increased, and the adjustment strategy for the log flow is the transaction allocation strategy; when entering the low load area, it indicates that too many log flows cause scattered writing or resource waste. At this time, log flows can be reduced, and some log flows are safely off-line, and the log flow adjustment strategy is the safe off-line strategy.

[0111] In an actual implementation. The dynamic adjustment of the log flow is also called log flow switching. This embodiment mainly uses the comprehensive decision-making module of the log file system to judge whether "log flow increase" or "log flow decrease" is required for the current load level, and then executes the following process:

[0112] When the multi-level adjustment strategy for log flow adjustment is the transaction allocation strategy, log flow increase is performed. It mainly includes: First, the system selects an "idle" log flow from the log flow pool (such as selecting the log flow in the FIFO manner), performs warm-up initialization (the initialization process includes but is not limited to: allocating log space, establishing metadata indexes), and allocates it to the busiest core or queue; then executes the allocation strategy to increase the log flow, obtaining the log flow increase adjustment result. For example, the transaction allocation strategy (such as hash mapping or the minimum flow load strategy): binds the subsequent submitted transactions to the new log flow, reducing the pressure on the already active log flows; or when the imbalance degree between flows is relatively high, part of the log requests of the core or file object can also be migrated to the new log flow to tend to be balanced.

[0113] When the multi-level adjustment strategy for log stream adjustment is the safe offline strategy, it is necessary to reduce the log stream at this time, and the log stream can be safely taken offline. Specifically, first select the target log stream to be taken offline from each log stream and freeze the target log stream (the target log stream can have a certain selection logic, such as the one with the smallest load as the target log stream), and prevent new transactions from being committed. Then wait for all in-flight transactions of this log stream to complete (journal_flush()), and persist the uncommitted data to the storage medium. After all in-flight transactions of the target log stream are committed, perform a consistency check on the data, and after ensuring consistency, mark the log stream as idle for subsequent reuse.

[0114] Thus, through multi-metric analysis and hierarchical strategies, this application avoids the simple approach of blindly adjusting based solely on thresholds, but instead uses more flexible weighting and multi-level interval judgment, improving the efficiency and adaptability of log stream management. When reducing the log stream, it first ensures that the already started log transactions are completed normally before taking it offline to maintain log consistency. Using this mechanism, it can make full use of parallelism in a multi-core environment while avoiding excessive fragmentation caused by over-dispersion of I / O.

[0115] Step 260, construct an initial write-back queue for the log stream adjustment result.

[0116] Among them, the initial write-back queue contains write-back tasks, and each write-back task contains a corresponding task type, and the task type includes input / output type and resource node type.

[0117] Step 270, separate the log write-back of the write-back tasks based on the input / output type from the background dirty page flushing and the direct write-back by the application program respectively, and correspond the cores or nodes of the write-back tasks to the local write-back queue based on the resource node type to obtain multi-queues corresponding to different task types.

[0118] Among them, the multi-queues contain write-back tasks to be processed.

[0119] A unified description of steps 260 - 270:

[0120] To further reduce the write-back bottleneck in a multi-core environment, this embodiment comprehensively improves the traditional global waiting queue method and realizes concurrent-friendly write-back through fine-grained partitioning and local scheduling. In the specific implementation, when performing task write-back, this embodiment constructs an initial write-back queue and then divides the write-back tasks in the initial write-back queue into multiple queues. Specifically, the multi-queue partitioning is mainly divided into: partitioning by I / O type and partitioning by CPU / NUMA node. Among them, partitioning by I / O type means separating journal writeback from background dirty page flushing, direct write-back by application programs, etc.; partitioning by CPU / NUMA node means that each core or node corresponds to a set of local write-back queues, reducing the contention for the global lock (wbt_wait). Thus, this embodiment reasonably divides different types of write-back tasks into multiple queues, improves the traditional global queue into multiple sub-queues or per-core queues, and through different types of partitioning mechanisms, effectively reduces the competition for the same lock in the log submission and dirty page write-back phases, and makes the requests between CPU cores easier to parallelize.

[0121] Step 280, perform write-back control optimization on each of the write-back tasks in the multiple queues to obtain an optimized result of log stream regulation.

[0122] Among them, the write-back control optimization process includes local scheduling processing and dynamic window adjustment processing adapted to the global queue lock contention.

[0123] Optionally, this embodiment performs write-back control optimization on each of the write-back tasks in the multiple queues to obtain an optimized result of log stream regulation, which may specifically include: performing local optimization based on each of the write-back tasks in the multiple queues, and performing dynamic window adjustment based on each association queue in the multiple queues to obtain an optimized result of log stream regulation; where the local optimization process includes ensuring that only the tasks in the corresponding queue are operated in each stage of the write-back process, and ensuring that there is no mutual exclusion blockage between different queues, and the dynamic window adjustment process includes setting a concurrency limit for each write-back queue and performing real-time adjustment according to the performance of the storage device of the log file system.

[0124] In a specific implementation, this embodiment mainly realizes concurrent-friendly write-back through fine-grained partitioning and local scheduling, thereby achieving write-back control optimization. In addition to fine-grained partitioning, the optimization process also involves local scheduling and dynamic window adjustment. Local scheduling mainly operates on tasks in the corresponding queue only during the I / O completion or wake-up phase, and ensures that queues of other cores or task types are not mutually blocked. Dynamic window adjustment involves the setting of the write-back queue and the impact of storage capabilities on regulation, that is, the system sets a concurrency limit for the write-back queue, which can be adjusted in real time based on the IOPS capabilities of the storage device to avoid serious queuing of write-back threads caused by I / O overload. By combining the above-mentioned fine-grained queues and local scheduling, this embodiment not only reduces the competition of multiple cores for a single global lock, but also enables better parallelism between log submission and other write-back operations, improving the overall throughput and write efficiency. The analysis based on the parallelism of multiple queues and the write merge rate shows that this application can more effectively match the parallelism of multi-core CPUs and storage devices, thereby improving system throughput and scalability. Practical verification also shows that this method can maintain high throughput and low response latency in a multi-core concurrent scenario.

[0125] As an example, referring to Figure 3 shown in the figure, this example uses lines and arrows to schematically show the relationship between user-space write operations, the adaptive log stream management module, the concurrent write-back control mechanism, and the underlying storage medium. The specific markings and connections are only used to illustrate the process of the present invention and the call sequence between modules. Among them, (1) User-space write operations mainly represent I / O requests initiated by user processes or applications; (2) Adaptive log stream management module: used to adaptively manage the number of log streams and improve the concurrency of log submission; (3) Concurrent write-back control mechanism: optimize the background write-back task scheduling, improve I / O throughput, and reduce global lock contention; (4) Storage medium: used to persist log data and support multiple types of storage devices. Specifically, user-space write requests are first captured by the file system, and the number or allocation method of the log streams to be used is determined by the adaptive log stream management module, and then the final write-back scheduling is performed by the concurrent write-back control mechanism and written to the underlying storage medium.

[0126] It should be noted that Figure 3 the overall block diagram shown does not include specific implementation logic or parameters and belongs to the schematic description of the embodiments of this application.

[0127] Furthermore, this embodiment can also achieve operation and adaptation. Specifically, the system periodically repeats the above monitoring and decision-making process in the background and performs dynamic optimization of log streams and write-back queues. For example: High-load scenario: After determining to enter the high-load area through weighted analysis, start the process of increasing log streams or increase the concurrency of the write-back queue. Low-load scenario: If the comprehensive score is in the low-load area, reduce the number of log streams to avoid fragmented writing. Medium-load scenario: Maintain the status quo or make minor adjustments to reach a better balance point.

[0128] In summary, the embodiments of the present application use a lightweight monitoring strategy. In the background of the log file system, key load metrics information is periodically obtained from the maintained log stream pool. When adjusting the number of adaptive log streams, based on the dynamically monitored load, the system load is analyzed through weighted processing, and the dynamic adjustment direction of the log stream is determined by using multi-level dynamic interval classification to perform dynamic adjustment of the log stream, automatically activating or recycling the number of log streams, taking into account the multi-core parallelism and write merging efficiency, and avoiding the limitations of traditional single or fixed log streams in different load modes; when performing concurrent write-back control and optimizing the write-back queue, the log submission and background write-back are divided into finer-grained queues or thread groups to reduce the global queue lock competition. Combining local scheduling and dynamic window adjustment enables better parallelism between log submission and other write-back operations. By reducing the scheduling interference across cores, the parallel execution ability of log submission and other write-back operations in a multi-core environment is improved, thereby increasing the overall throughput, response speed, and write efficiency in a multi-core scenario. The solution of this embodiment can be widely applied to file systems with log functions or other storage management software, and can obtain high-concurrency performance improvement in a multi-core environment, providing key technical support for the parallel optimization of modern storage systems.

[0129] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present application are not limited by the described action sequence, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously.

[0130] As Figure 4 shown, the embodiments of the present application also provide a log stream adjustment and optimization system 400, including:

[0131] An index information periodic monitoring module 410, configured to periodically obtain key load metrics information from the maintained log stream pool in the background of the log file system through a preset lightweight monitoring strategy, where the key load metrics information includes transaction submission time, queue length, balance, resource utilization rate, and throughput;

[0132] A weighted processing module 420, configured to perform weighted processing according to the key load metrics information and determine an adjustment interval of the log file system in the current period based on the weighted result;

[0133] A dynamic adjustment module 430 for load balancing, configured to perform dynamic adjustment of the log stream for load balancing according to a multi-level adjustment strategy corresponding to the adjustment interval to obtain a log stream adjustment result, where the log stream adjustment result includes a log stream increment adjustment result or a log stream offline adjustment result;

[0134] The write-back task partitioning module 440 is used to perform fine-grained partitioning on the write-back tasks of different types in the initially constructed write-back queue according to the adjusted result of the log stream, so as to obtain multiple queues for writing back the write-back tasks, and the multiple queues include the tasks to be written back;

[0135] The write-back control optimization module 450 is used to optimize the write-back control of each of the tasks to be written back in the multiple queues to obtain the optimized result of log stream adjustment;

[0136] Among them, the write-back control optimization process includes local scheduling processing and dynamic window adjustment processing.

[0137] Optionally, the metric information periodic monitoring module 410 includes:

[0138] The log stream allocation sub-module is used to pre-allocate at least one group of log streams in a preset initial state that can be dynamically managed for the multi-core processor when the log file system is initialized and loaded;

[0139] The log stream pool maintenance sub-module is used to maintain the created log stream pool for the allocation of log stream resources, and each log stream independently executes transaction records;

[0140] The load periodic monitoring sub-module is used to periodically monitor the log streams of the log file system through a preset lightweight detection strategy, capture the load changes of each log stream from the log stream pool, and obtain the load key metric information; among them, the load key metric information is stored in the circular buffer area of the log file system and is used for analysis.

[0141] Optionally, the load periodic monitoring sub-module includes:

[0142] The statistics unit is used to determine the log stream queue corresponding to each log stream, and count the number of transactions to be committed or the depth of the request queue of each active log in the log stream queue to obtain the queue length information;

[0143] The transaction submission time determination unit is used to extract the average submission delay in the most recent or a period of time from the transaction record as the transaction submission time;

[0144] The workload measurement unit is used to measure the workload of each log stream in the log stream queue to obtain the workload difference degree between each log stream as the balance degree, and obtain the resource utilization information to analyze the resource utilization rate;

[0145] The throughput determination unit is used to analyze the regulation result according to the number of transactions processed per unit time to obtain the throughput.

[0146] Optionally, the weighted processing module 420 includes:

[0147] A weighted information acquisition sub-module, configured to acquire multi-index weighted information.

[0148] A normalization or mapping sub-module, configured to perform normalization or mapping processing according to each index item in the load key index information to obtain a mapping function, where the multi-index weighted information includes weight coefficients corresponding to each index item in the load key index information;

[0149] A weighted processing sub-module, configured to, within a preset time interval, perform weighted processing on the mapping function corresponding to each index item according to the weight coefficient to obtain a weighted result, where the weighted result includes index quantization information for characterizing the dynamic measurement of the global load level and balance degree;

[0150] A dynamic adjustment sub-module, configured to perform multi-level dynamic interval classification according to the index quantization information, and continuously monitor the change information of the log stream load during the dynamic interval classification process for dynamic adjustment to obtain an adjustment interval; wherein, the adjustment interval includes a low load area, a medium load area, and a high load area.

[0151] Optionally, the weighted processing sub-module is specifically configured to: determine the index quantization information according to S = w q ×f q +w t ×f t +w i ×f i +w cpu ×f cpu , where S is the comprehensive score, representing the index quantization information, w q is the weight of the queue length index, w t is the weight of the transaction commit time index, w i is the weight of the balance degree index, w cpu is the weight of the resource utilization rate index, and f q , f t , f i and f cpu are the normalization or mapping functions of each index, used to ensure that the weighted results of each index are within a comparable range

[0152] Optionally, the dynamic adjustment module 430 of the load balancing includes:

[0153] An adjustment strategy selection sub-module, configured to identify log stream adjustment information based on the adjustment interval, and select a multi-level adjustment strategy according to the log stream adjustment information, where the multi-level adjustment strategy includes a transaction allocation strategy and a safe off-line strategy;

[0154] A log stream selection sub-module, configured to select a new log stream from the log stream pool when the log stream adjustment information is log stream addition information, and perform warm-up initialization and queue core allocation processing on the new log stream;

[0155] A load balancing dynamic adjustment sub-module, configured to bind the committed transactions to the new log stream according to the log stream addition information by using a transaction allocation strategy, and perform dynamic adjustment of load balancing on the new log stream until the new log stream tends to be balanced, so as to obtain a log stream increment adjustment result;

[0156] A target log stream freezing sub-module, configured to freeze a target log stream by using the safe off-line strategy when the log stream adjustment information is log stream reduction information, where the target log stream is the log stream with the lowest load;

[0157] A log stream data processing sub-module, configured to perform persistence processing and data consistency verification processing on the uncommitted data of the target log stream after the target log stream completes the currently executed transaction, and mark the target log stream as an idle state based on the data consistency verification result, so as to obtain a log stream off-line adjustment result.

[0158] Optionally, the write-back task partitioning module 440 includes:

[0159] A write-back queue construction sub-module, configured to construct an initial write-back queue for the log stream adjustment result, where the initial write-back queue includes write-back tasks, and each write-back task includes a corresponding task type, and the task type includes input / output type and resource node type;

[0160] A write-back task separation sub-module, configured to separate the log write-back of the write-back tasks based on the input / output type from the background dirty page refresh and the direct write-back of the application program respectively, and correspond the core or node of the write-back tasks to the local write-back queue based on the resource node type, so as to obtain multiple queues corresponding to different task types.

[0161] Optionally, the write-back control optimization module 450 includes:

[0162] A local optimization sub-module, configured to perform local optimization based on each write-back task in the multiple queues;

[0163] A dynamic window adjustment sub-module, configured to perform dynamic window adjustment based on each association queue in the multiple queues to obtain a log stream adjustment optimization result; where the local optimization process includes ensuring that only the tasks in the corresponding queue are operated in each stage during the write-back process, and ensuring that there is no mutual exclusion blocking between different queues, and the dynamic window adjustment process includes setting a concurrency limit for each write-back queue and performing real-time adjustment according to the performance of the storage device of the log file system.

[0164] It should be noted that the log stream adjustment and optimization system provided by the embodiments of the present application can execute the log stream adjustment and optimization method provided by any embodiment of the present application, and has the corresponding functions and beneficial effects of executing the method.

[0165] In a specific implementation, the above-mentioned log stream adjustment and optimization system can be integrated into a device, so that the device can dynamically adjust the log stream according to the load change, and perform fine-grained division and write-back control optimization on the write-back task. As an electronic device, improvements are made separately in the log layer and the write-back control layer of the file system, so that the system can balance parallel log writing and efficient merging in a high-concurrency scenario, reduce global lock contention and improve the overall throughput. The electronic device can be composed of two or more physical entities, or can be composed of a single physical entity. For example, the electronic device can be a personal computer (PC), a computer, a server, etc. The embodiments of the present application do not make specific limitations on this.

[0166] As Figure 5 shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114. Among them, the processor 111, the communication interface 112, and the memory 113 complete mutual communication through the communication bus 114; the memory 113 is used to store a computer program; when the processor 111 executes the program stored on the memory 113, it implements the steps of the log stream adjustment and optimization method of the log file system provided by any one of the foregoing method embodiments. Exemplarily, the steps of the log stream adjustment and optimization method of the log file system may include the following steps: obtaining load key index information from the maintained log stream pool periodically in the background of the log file system through a preset lightweight monitoring strategy, where the load key index information includes transaction submission time, queue length, balance, resource utilization rate, and throughput; performing weighted processing according to the load key index information, and determining the adjustment interval of the log file system in the current period based on the weighted result; performing load-balanced dynamic adjustment of the log flow according to the multi-level adjustment strategy corresponding to the adjustment interval to obtain a log flow adjustment result, where the log flow adjustment result includes a log flow increase adjustment result or a log flow offline adjustment result; for the log flow adjustment result, performing fine-grained division according to different types of write-back tasks in the constructed initial write-back queue to obtain multiple queues for performing write-back tasks, where the multiple queues include write-back tasks to be written back; performing write-back control optimization on each write-back task to be written back in the multiple queues to obtain a log stream adjustment and optimization result; where the write-back control optimization process includes local scheduling processing and dynamic window adjustment processing adapted to global queue lock contention.

[0167] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the log stream adjustment and optimization method of the log file system provided in any of the foregoing method embodiments are implemented.

[0168] It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0169] The foregoing are only specific embodiments of the present application, which enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A log stream adjustment and optimization method for a log file system, characterized in that: A concurrent log file system for multi-core architectures, including: Through a preset lightweight monitoring strategy, in the background of the log file system, key load indicator information is periodically obtained from the maintained log stream pool, and the key load indicator information includes transaction submission time, queue length, balance, resource utilization and throughput; Performing weighted processing according to the load key indicator information, and determining an adjustment interval of the log file system in a current cycle based on the weighted result; Performing dynamic load-balanced log stream adjustment according to the multi-level adjustment strategy corresponding to the adjustment interval to obtain a log stream adjustment result, wherein the log stream adjustment result includes a log stream increase adjustment result or a log stream offline adjustment result; Based on the log stream adjustment result, fine-grained division is performed according to different types of write-back tasks in the constructed initial write-back queue to obtain multiple queues for writing back the write-back tasks, wherein the multiple queues contain tasks to be written back; Performing write-back control optimization on each of the tasks to be written back in the multiple queues to obtain a log stream regulation optimization result; The write-back control optimization process includes local scheduling processing and dynamic window adjustment processing adapted to global queue lock competition.

2. The method according to claim 1, characterized in that: Through the preset lightweight monitoring strategy, in the background of the log file system, the key load indicator information is periodically obtained from the maintained log stream pool, including: When the log file system is initially loaded, at least one set of dynamically manageable log streams in a preset initial state is pre-allocated to the multi-core processor, and a log stream pool created for log stream resource allocation is maintained, and each log stream independently performs transaction recording; By using a preset lightweight detection strategy, the log stream of the log file system is periodically monitored, and the load changes of each log stream are captured from the log stream pool to obtain the key load indicator information; The load key indicator information is stored in a ring buffer of the log file system and is used for analysis.

3. The method according to claim 2, characterized in that Capture the load changes of each log stream from the log stream pool to obtain key load indicator information, including: Determine the log stream queue corresponding to each log stream, and count the number of pending transactions or request queue depth of each active log in the log stream queue to obtain queue length information; Extract the most recent or average commit delay over a period of time from the transaction record as the transaction commit time; Performing work measurement on each log stream in the log stream queue to obtain a workload difference between the log streams as a balance degree, and obtaining resource utilization information to analyze resource utilization rate; The throughput is obtained by analyzing the control results based on the number of transactions processed per unit time.

4. The method according to claim 1, characterized in that: Performing weighted processing according to the load key indicator information, and determining the adjustment interval of the log file system in the current cycle based on the weighted result, including: Acquire multi-index weighted information, and perform normalization or mapping processing on each index item in the load key index information to obtain a mapping function, wherein the multi-index weighted information includes a weight coefficient corresponding to each index item in the load key index information; Within a preset time interval, weighted processing is performed on the mapping function corresponding to each indicator item according to the weight coefficient to obtain a weighted result, wherein the weighted result includes indicator quantification information for characterizing a dynamic measurement of a global load level and a degree of balance; Perform multi-level dynamic interval classification according to the quantitative information of the indicator, and continuously monitor the change information of the log stream load during the dynamic interval classification process to perform dynamic adjustment to obtain the adjustment interval; The adjustment range includes a low load area, a medium load area and a high load area.

5. The method according to claim 4, characterized in that Within a preset time interval, the mapping function corresponding to the indicator item is weighted according to the weight coefficient to obtain a weighted result, including: According to S = w q ×f q +w t ×f t +w i ×f i +w cpu ×f cpu , determine the quantitative information of indicators; Among them, S is the comprehensive score, which represents the quantitative information of the indicator, and w q is the queue length indicator weight, w t is the transaction submission time indicator weight, w i is the balance index weight, w cpu is the resource utilization index weight, f q 、f t 、f i and f cpu It is a normalization or mapping function for each indicator, which is used to ensure that the weighted results of each indicator are within a comparable range.

6. The method according to claim 1, characterized in that According to the multi-level adjustment strategy corresponding to the adjustment interval, the load balancing log stream is dynamically adjusted to obtain the log stream adjustment result, including: Identifying log stream adjustment information based on the adjustment interval, and selecting a multi-level adjustment strategy according to the log stream adjustment information, wherein the multi-level adjustment strategy includes a transaction allocation strategy and a safe offline strategy; When the log stream adjustment information is log stream addition information, a new log stream is selected from the log stream pool, and preheating initialization and queue core allocation processing are performed on the new log stream; According to the log stream addition information, the submitted transaction is bound to the new log stream by using the transaction allocation strategy, and the new log stream is dynamically adjusted for load balancing until the new log stream tends to be balanced, thereby obtaining a log stream increase adjustment result; When the log stream adjustment information is log stream reduction information, freezing the target log stream using the safe offline strategy, the target log stream being the log stream with the lowest load; After the target log stream completes the currently executed transaction, the uncommitted data of the target log stream is persisted and checked for data consistency, and based on the data consistency check result, the target log stream is marked as idle to obtain the log stream offline adjustment result.

7. The method according to claim 1, characterized in that According to the log stream adjustment result, fine-grained division is performed according to different types of write-back tasks in the constructed initial write-back queue to obtain multiple queues for writing back the write-back tasks, including: According to the log stream adjustment result, an initial write-back queue is constructed, wherein the initial write-back queue includes write-back tasks, each write-back task includes a corresponding task type, and the task type includes an input-output type and a resource node type; Based on the input and output type, the log writeback of the writeback task is separated from the background dirty page refresh and the direct writeback of the application, and based on the resource node type, the core or node of the writeback task is corresponded to the local writeback queue to obtain multiple queues corresponding to different task types.

8. The method according to claim 7, characterized in that Performing write-back control optimization on each of the tasks to be written back in the multiple queues to obtain a log stream adjustment optimization result, including: Performing local optimization based on each of the to-be-written-back tasks in the multiple queues, and performing dynamic window adjustment based on each of the association queues in the multiple queues, to obtain a log stream adjustment optimization result; Among them, the local optimization process includes ensuring that only tasks in the corresponding queue are operated in each stage of the write-back process, and ensuring that there is no mutual exclusion blocking between different queues. The dynamic window adjustment process includes setting concurrency limits for each write-back queue and making real-time adjustments based on the performance of the storage device of the log file system.

9. A log stream adjustment and optimization system, characterized in that: include: An indicator information periodic monitoring module is used to periodically obtain key load indicator information from the maintained log stream pool in the background of the log file system through a preset lightweight monitoring strategy, wherein the key load indicator information includes transaction submission time, queue length, balance, resource utilization and throughput; A weighted processing module, used for performing weighted processing according to the load key indicator information, and determining the adjustment interval of the log file system in the current cycle based on the weighted result; A dynamic adjustment module for load balancing, used to dynamically adjust the log stream of load balancing according to the multi-level adjustment strategy corresponding to the adjustment interval, and obtain a log stream adjustment result, wherein the log stream adjustment result includes a log stream increase adjustment result or a log stream offline adjustment result; A write-back task division module is used to perform fine-grained division of the write-back tasks of different types in the constructed initial write-back queue according to the log stream adjustment result, to obtain multiple queues for writing back the write-back tasks, wherein the multiple queues contain tasks to be written back; A write-back control optimization module, used for performing write-back control optimization on each of the tasks to be written back in the multiple queues to obtain a log stream regulation optimization result; The write-back control optimization process includes local scheduling processing and dynamic window adjustment processing.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the log stream adjustment and optimization method of the log file system described in any one of claims 1-8 when executing the program stored in the memory.

Images