Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment, and relates to the technical field of computers.The method comprises the steps that firstly, system state data in the cache data flashing process is collected; detecting the system state data by utilizing triggering conditions corresponding to different flashing strategies, and determining a target condition triggered by the system state data from the triggering conditions; determining a flashing rate corresponding to the cache data based on a target flashing strategy corresponding to the target condition; and finally, flashing the cached data to the persistent storage device according to the flashing rate. In this way, in the cache data flashing process, the system state data can be detected in real time, the target flashing strategy is dynamically selected from the multiple flashing strategies, and the flashing rate of the cache data is adjusted in real time, so that the flashing rate of the cache data adapts to the dynamically changing system state; the matching degree of the flash rate and the system state is improved, and then the system performance is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a data processing method, apparatus, storage medium, and electronic device. Background Art

[0002] For data-intensive applications that need to process a large amount of data, data caching technology can effectively improve system performance. Among them, the flushing of cached data, as the core link of cache management, is directly related to multiple aspects such as data consistency, storage load, and response speed.

[0003] Currently, related technologies usually use static rules for flushing cached data, such as flushing at fixed time intervals. However, this method is difficult to adapt to dynamic system states, easily leads to a mismatch between the flushing rate and the system state, and thus affects system performance. Summary of the Invention

[0004] The present disclosure provides a data processing method, apparatus, storage medium, and electronic device. Its main purpose is to solve the problem that related technologies use static rules for flushing cached data, such as flushing at fixed time intervals. However, this method is difficult to adapt to dynamic system states, easily leads to a mismatch between the flushing rate and the system state, and thus affects system performance.

[0005] In a first aspect, the present application provides a data processing method, including:

[0006] Collecting system state data during the flushing process of cached data;

[0007] Detecting the system state data using the trigger conditions corresponding to different flushing strategies, and determining the target condition triggered by the system state data from the trigger conditions;

[0008] Based on the target flushing strategy corresponding to the target condition, determining the flushing rate corresponding to the cached data;

[0009] Flushing the cached data to the persistent storage device according to the flushing rate.

[0010] In a second aspect, the present application provides a data processing apparatus, including:

[0011] A collection module, configured to collect system state data during the flushing process of cached data;

[0012] A determination module, configured to detect the system state data using the trigger conditions corresponding to different flushing strategies, and determine the target condition triggered by the system state data from the trigger conditions;

[0013] A determination module, configured to determine the flushing rate corresponding to the cached data based on the target flushing strategy corresponding to the target condition;

[0014] A flashing module, configured to flash the cached data to a persistent storage device according to a flashing rate.

[0015] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0016] In a fourth aspect, the present application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and when the processor executes the computer program, the method of the first aspect is implemented.

[0017] In a fifth aspect, the present application provides a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0018] The data processing method, device, storage medium, and electronic device provided by the present disclosure, wherein the method includes: first, collecting system state data during the flashing process of cached data; then, detecting the system state data using the triggering conditions corresponding to different flashing strategies, and determining the target condition triggered by the system state data from the triggering conditions; then, based on the target flashing strategy corresponding to the target condition, determining the flashing rate corresponding to the cached data; and finally, flashing the cached data to the persistent storage device according to the flashing rate. In this way, during the flashing process of cached data, the system state data can be detected in real time, and the system state data can be detected using the triggering conditions corresponding to different flashing strategies, determining the target condition triggered by the current system state data, then using the target flashing strategy corresponding to the target condition to determine the current flashing rate of the cached data, and finally flashing the cached data to the persistent storage device according to the flashing rate, so as to dynamically select the target flashing strategy from multiple flashing strategies according to the real-time detected system state data, adjust the flashing rate of the cached data in real time, make the flashing rate of the cached data adapt to the dynamically changing system state, improve the matching degree between the flashing rate and the system state, and further improve the system performance.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. Description of the Drawings

[0020] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 Shows a schematic flow chart of a data processing method provided by an embodiment of the present application;

[0022] Figure 2 Shows a schematic diagram of an example provided by an embodiment of the present application;

[0023] Figure 3 Shows a schematic diagram of another example provided by an embodiment of the present application;

[0024] Figure 4 Shows a schematic structural diagram of a data processing device provided by an embodiment of the present application. Detailed implementation manners

[0025] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0026] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0027] With the rapid development of data-intensive applications (such as real-time analysis, high-concurrency transaction processing, artificial intelligence (AI) model training, etc.), data caching technology has become the core means to improve system performance. Correspondingly, modern cache systems need to achieve a balance among high throughput, low latency, and resource efficiency, and cache data flushing, as the core link of cache management, is directly related to aspects such as data consistency, storage load, and response speed of the system.

[0028] In the related art, cache flushing strategies mostly adopt static rules (such as timed flushing, fixed capacity threshold triggering). However, this fixed flushing rule method is difficult to adapt to the dynamically changing load characteristics, such as the load changes caused by sudden large traffic operations, which easily leads to the mismatch between the flushing rate (down-flushing rate) and the actual system requirements, thereby affecting system performance.

[0029] Specifically, the current mainstream cache write-back technology mainly focuses on static rules and single-metric triggering mechanisms. Among them, timed write-back is a basic solution that forcibly persists the data in memory to the storage layer at fixed time intervals. However, static time intervals are difficult to cope with sudden writes or data silent periods. An overly long interval may increase the risk of data loss, while an overly short interval is likely to cause unnecessary I / O overhead. Secondly, capacity threshold triggering is another commonly used basic solution. When the data ratio or total amount in the cache reaches a preset threshold, the write-back operation is triggered. However, the static threshold setting is prone to "avalanche-style" write-back due to overly frequent threshold triggering in the face of dynamic loads, especially in high-concurrency scenarios with intensive random writes, which further exacerbates the input / output (I / O) contention problem in the storage layer.

[0030] As a possible implementation, based on the write-combining technology, discrete writes of multiple data blocks can be combined into continuous I / O operations, thereby reducing the number of write-backs and improving the throughput of storage devices, further optimizing the write-back efficiency. Although this method is effective in a hard disk drive (HDD) environment dominated by sequential writes, in a solid-state drive (SSD) or new storage media dominated by random writes, the latency brought by combining may increase the risk of data loss, resulting in limited optimization effects.

[0031] Although the existing cache write-back technologies have achieved certain effects in different scenarios, their core problem lies in the fundamental contradiction between static policies and dynamic loads. Related technologies usually rely on fixed parameters (time intervals, threshold sizes), making it difficult to adapt to load fluctuations and dynamic changes in data access patterns, resulting in waste of resources due to excessive write-backs under low loads, while write-back latency under high loads may cause data loss or a sudden drop in storage performance.

[0032] To address the technical problem that the current related technologies use static rules for caching data write-back, such as writing at fixed time intervals, but this method is difficult to adapt to dynamic system states, easily leading to a mismatch between the write rate and the system state, thereby affecting system performance.

[0033] This embodiment provides a data processing method, as Figure 1 shown, the method includes the following steps:

[0034] Step 101, collect system state data during the caching data write-back process.

[0035] In some embodiments, the caching data may include data temporarily stored in a fast-access storage medium (such as a part of memory or a solid-state drive), which can be used to accelerate data access speed.

[0036] During the cache data writing process, the device status can be detected in real time, and multi-source metrics of the device can be collected. The collected system status data may include, but is not limited to, basic data such as the utilization rate of the Central Processing Unit (CPU), the memory occupancy rate, the IO queue depth, and the network bandwidth ratio, as well as cache-specific data such as the cache data capacity ratio, the cache hit rate, and the read / write request distribution. The system status data can be used to reflect the system load, application requirements, data consistency, and storage system performance, etc., facilitating the adjustment of the writing rate in real time according to the system status, improving the adaptability of the writing rate to the system status, and thus ensuring the stable operation of the system.

[0037] Step 102: Detect the system status data using the trigger conditions corresponding to different writing strategies, and determine the target condition triggered by the system status data from the trigger conditions.

[0038] Among them, the writing strategies may include strategies corresponding to different preset system statuses, and can be constructed based on different writing rate control algorithms; the trigger conditions may include judgment conditions corresponding to multiple writing strategies, used to judge whether to trigger the corresponding writing strategy; the target condition can be the trigger condition that the system status data can satisfy after detecting the system status data using each trigger condition, and serves as the response entry of the target writing strategy.

[0039] In some embodiments, to adapt to the dynamically changing system status, the trigger conditions corresponding to different writing strategies are used to detect the real-time collected system status data to determine the current state of the system and select an appropriate cache data writing method. Specifically, corresponding writing strategies can be configured for different system statuses, and the trigger conditions or response conditions of each writing strategy can be set, so that after the system status data meets a certain trigger condition, the corresponding writing strategy is automatically called to achieve the dynamic regulation of the writing strategy according to the system status. Exemplarily, the system load (such as CPU utilization rate, memory occupancy, I / O waiting time, etc.) reaching a certain threshold can be used as the trigger condition, so as to dynamically adjust the writing rate according to the system load.

[0040] Step 103: Determine the writing rate corresponding to the cache data based on the target writing strategy corresponding to the target condition.

[0041] Exemplarily, the target flashing policy may include a flashing rate control algorithm corresponding to the target condition. In some embodiments, in response to the target condition being triggered, the flashing rate control algorithm corresponding to the target flashing policy may be called to calculate the flashing rate of the cached data corresponding to the current system state, so as to realize real-time adjustment of the flashing rate according to the system state, adapt to load fluctuations and dynamic changes in the data access pattern, reduce resource waste caused by excessive flashing during low load, and avoid situations such as data loss or sudden drop in storage performance caused by flashing delay during high load, thereby improving the stability of the system.

[0042] Step 104: Flash the cached data to the persistent storage device according to the flashing rate.

[0043] In some embodiments, persistent storage can be used to save data to a storage medium that can retain the data even after power-off, to avoid data loss. For example, it can be a hard disk drive, a solid-state drive, a solid-state hybrid drive (SSH), etc. The flashing rate determined according to the target flashing policy can be used to adjust the flashing rate of the cached data in real time, that is, to adjust the speed at which data is flashed from the cache to devices such as hard disks. Correspondingly, after successful flashing, the system status data can be continuously detected, and according to the current system status data, the system performance can be determined. If the system performance deteriorates, the target flashing policy can be re-determined and the flashing rate can be re-adjusted to adapt to the dynamically changing storage environment.

[0044] Compared with the current existing technologies, this embodiment can, during the flashing process of the cached data, detect the system status data in real time, use the trigger conditions corresponding to different flashing policies to detect the system status data, determine the target condition triggered by the current system status data, then use the target flashing policy corresponding to the target condition to determine the current flashing rate of the cached data, and finally flash the cached data to the persistent storage device according to the flashing rate. Thus, according to the real-time detected system status data, the target flashing policy is dynamically selected from multiple flashing policies using multiple trigger conditions, and the flashing rate of the cached data is adjusted in real time, so that the flashing rate of the cached data adapts to the dynamically changing system state, improves the matching degree between the flashing rate and the system state, and further improves the system performance.

[0045] Optionally, to further illustrate the specific implementation process of the method in this embodiment, collecting the system status data during the flashing process of the cached data may specifically include: determining the sampling frequency corresponding to the system status data according to the load information corresponding to the system status data, and collecting the system status data according to the sampling frequency.

[0046] Exemplarily, such as Figure 2As shown, a framework diagram of the monitoring module is presented. The monitoring module is responsible for capturing system status data in real time and may involve modules such as adaptive sampling, basic data collection, cache-specific data collection, noise reduction, and feedback loops. Among them, the adaptive sampling module interacts with the basic data collection and cache-specific data collection modules through the control flow, respectively collecting basic data and cache-specific data to obtain system status data; the basic data collection module can be used to collect the basic operation data of the system, receive the control flow instructions of adaptive sampling, perform basic data collection, and then transmit the collected data to the noise reduction module through the data flow; the cache-specific data collection module can collect the data in the cache for analyzing the usage or performance optimization of the cache, receive the control flow instructions of the adaptive sampling module, and then transmit the collected data to the noise reduction module through the data flow; the noise reduction module can be used to process the system status data collected by the basic data collection and cache-specific data collection, remove noise, improve data quality, and then output the processed data to the feedback loop for trigger condition detection. Exemplarily, a filtering algorithm can be applied to eliminate instantaneous interference (such as occasional I / O delay glitches) to ensure data reliability.

[0047] During the data collection process, the adaptive sampling module can dynamically adjust the sampling frequency according to the system load conditions. For example, when it detects that the system load is too high based on the load information corresponding to the system status data, it can reduce the sampling frequency to reduce the resource consumption brought by the monitoring module; when it detects that the system load is low, it can increase the sampling frequency to improve resource utilization. By this way of dynamically adjusting the sampling frequency, the monitoring accuracy and resource overhead are balanced, ensuring that the system can provide detailed performance data when necessary and will not cause additional burden due to over-monitoring, which helps to maintain the efficient and stable operation of the system.

[0048] Optionally, before detecting the system status data using the trigger conditions corresponding to different flashing strategies and determining the target conditions triggered by the system status data from the trigger conditions, the method of this embodiment may specifically further include: configuring different flashing strategies corresponding to the cache data and the trigger conditions corresponding to different flashing strategies.

[0049] In some embodiments, multiple flashing strategies and their corresponding trigger conditions can be preset to ensure that the system can flexibly respond to different workloads and state changes, perform dynamic adjustment of the flashing strategy, and thus maintain high performance and data consistency. Exemplarily, the flashing strategy can include control algorithms corresponding to different flashing rates, such as fuzzy control algorithms, etc.; the trigger conditions can include trigger conditions corresponding to different control algorithms, such as system load trigger conditions, I / O load trigger conditions, etc.

[0050] Optionally, the system status data is detected using the trigger conditions corresponding to different flashing strategies, and the target condition triggered by the system status data is determined from the trigger conditions. Specifically, it may include: extracting the status trigger information corresponding to different trigger conditions from the system status data; using different trigger conditions to respectively detect the status trigger information corresponding to different trigger conditions, and determining the target condition triggered by the system status data according to the detection results.

[0051] In some embodiments, according to the preset trigger conditions, the status trigger information required for judging each trigger condition is extracted from the system status data, and then each trigger condition is used to respectively detect the extracted status trigger information to determine whether each status trigger information meets the corresponding trigger condition, obtaining the detection results of different trigger conditions.

[0052] Optionally, different trigger conditions are used to respectively detect the status trigger information corresponding to different trigger conditions, and the target condition triggered by the system status data is determined according to the detection results. Specifically, it may include: if it is determined according to the detection results that the target status trigger information meets its corresponding trigger condition, then the trigger condition corresponding to the target status trigger information is determined as the target condition.

[0053] Among them, the target status trigger information is the status trigger information that meets the corresponding trigger condition.

[0054] Exemplarily, if after detecting multiple status trigger information, it is determined that there is target status trigger information in the status trigger information that meets a certain trigger condition, then the trigger condition corresponding to the status trigger information can be determined as the target condition, indicating that the current system status has changed, and it is necessary to use the target flashing strategy corresponding to the target condition for adjustment to obtain a flashing rate adapted to the current system status.

[0055] In some embodiments, the control module can be used to make a judgment based on the data provided by the feedback loop. When the trigger condition is met, calculations are immediately performed according to the selected algorithm, and preset actions are executed according to the results, such as reducing or increasing the downward flashing rate, and at the same time, smooth transition is performed to avoid system oscillations caused by steps. Specifically, the optimal strategy in the control algorithm library can be matched according to the information transmitted by the feedback loop. For example, the fuzzy control algorithm is used to quickly increase the downward flashing rate during a large number of sudden writes, and the reinforcement learning model is switched to optimize the long-term rate during the steady period.

[0056] Exemplarily, for the system load trigger condition, the CPU utilization rate in the system status information can be extracted, and it is detected whether the CPU utilization rate reaches the preset CPU utilization rate threshold corresponding to the trigger condition. By comparing the CPU utilization rate of the current system status with the preset CPU utilization rate threshold, it is determined whether to trigger the flashing strategy corresponding to the system load trigger condition. If it is detected that the CPU utilization rate of the current system status reaches the preset CPU utilization rate threshold, the system load trigger condition can be determined as the target condition. When it indicates that the current system load is too high, the corresponding target flashing strategy can be to reduce the flashing frequency to reduce the additional pressure on the system, and realize the dynamic adjustment of the flashing rate according to the system load.

[0057] Optionally, based on the target flashing strategy corresponding to the target condition, the flashing rate corresponding to the cached data is determined. Specifically, it may include: determining the target control algorithm corresponding to the target flashing strategy from the preset control algorithm library, and the preset control algorithm library includes the flashing rate control algorithms corresponding to different system statuses; using the target control algorithm to adjust the flashing rate corresponding to the cached data.

[0058] Among them, as the core decision provider, the preset control algorithm library can provide a multi-dimensional and extensible set of flashing rate control algorithms (down-flashing rate regulation algorithms), and its functions can specifically include multi-mode algorithm integration and user-defined extension. Specifically, multi-mode algorithm integration can be used to build-in classic control models (such as linear proportional adjustment, Proportional-Integral-Derivative (PID) control) and intelligent decision-making algorithms (such as fuzzy logic control, dynamic programming based on reinforcement learning) to cover different load scenario requirements. For example, when the system status fluctuates violently, the control module can give priority to calling the PID algorithm to achieve fast convergence; while in the long-term steady-state operation stage, switch to the reinforcement learning model to explore the historical data law and optimize the long-term resource utilization rate.

[0059] Optionally, a standardized algorithm interface can also be provided to allow users to inject custom algorithms or adjust the existing algorithm parameters in the algorithm library, realizing the user-defined extension of the preset control algorithm library. Exemplarily, an algorithm extension configuration function can be provided to allow users to add custom algorithms to the preset control algorithm library or adjust the existing algorithm parameters in the algorithm library, so as to meet different user needs, expand the application scenarios of the preset control algorithm library, provide a mechanism for dynamically adjusting the down-flashing rate of the cache, and an extensible preset control algorithm library.

[0060] In this way, it is possible to provide a variety of control algorithms for the flashing rate, meet various system requirements, facilitate the real-time adjustment of the flashing rate, and support users to expand the preset control algorithm library to meet various user needs.

[0061] Optionally, the method of this embodiment may further specifically include: determining whether the target control algorithm meets the downgrade condition based on the system status data after being flashed at the flashing rate; if the target control algorithm meets the downgrade condition, reducing the algorithm level of the target control algorithm and generating a downgrade warning corresponding to the target control algorithm.

[0062] In some embodiments, after adjusting the flashing rate, the system status data can be continuously monitored to obtain the system status data after the flashing rate is adjusted, compare the adjusted system status data with the system status data before the adjustment, detect whether the system performance index has improved, and detect whether the target control algorithm can optimize the system performance according to the safe downgrade mechanism. If it is detected that the target control algorithm is invalid, for example: after multiple (such as 3 times) adjustments, the system performance index has not improved, or the system CPU is severely overloaded due to the operation of the algorithm, it can be determined that the downgrade condition of the target control algorithm is met, and the level of the current control algorithm can be reduced. Exemplarily, it can be automatically downgraded to the level corresponding to the static rule, such as the static threshold mode, the target control algorithm is adjusted to the threshold control algorithm, and an alarm notification is sent. Specifically, the upper and lower limits of the safe threshold of the flashing rate can be set to prevent system paralysis caused by misoperation.

[0063] Optionally, the method of this embodiment may further specifically include: during the flashing process of the cached data, displaying the flashing metrics of the cached data and the mark corresponding to the target control algorithm.

[0064] Among them, the flashing metrics may include key metrics corresponding to the system status data, such as: system load, flashing rate, storage bandwidth occupancy rate, etc. The switching mark corresponding to the target control algorithm can be used to mark when the control algorithm changes, so as to facilitate positioning of the performance bottleneck and optimizing the rate control strategy.

[0065] Exemplarily, during the flashing process, the real-time curve of the key metrics corresponding to the system status data can be dynamically displayed, and the algorithm switching event mark can be superimposed to assist in quickly positioning the performance bottleneck. Based on the historical data recorded by the feedback loop, it supports filtering records according to the time range and event type (such as threshold trigger, algorithm switching, etc.), generating an efficiency report, and the report may include information such as the average flashing delay and the number of I / O conflicts.

[0066] In this way, the user can view the flashing status of the cached data in real time during the flashing process, and provide the mark corresponding to the target control algorithm for subsequent algorithm analysis and optimization, discover and solve potential problems, which helps to improve the overall performance of the system.

[0067] Optionally, the method of this embodiment may further include: receiving the analog information corresponding to the flashing rate, obtaining the system status metrics corresponding to the analog information, and the analog information includes combinations of different control algorithms and different parameters.

[0068] In some embodiments, interactive policy simulation can be applied to provide an offline sandbox environment that allows users to upload rate simulation information for simulation, such as historical load data. By using the historical load data, the system behavior under different time periods or conditions can be simulated, the effects of different combinations of control algorithms and parameters can be obtained, and system state metrics can be output as comparison metrics, such as throughput improvement rate, risk of dirty data backlog, etc. The effects of various rate control strategies can be evaluated, a simulation space can be provided to explore and verify various hypotheses, and simulation results can be obtained to facilitate auxiliary policy tuning and further optimize the rate adjustment effect.

[0069] In addition, the method of this embodiment can also support multi-modal interfaces and system designs that support various user interaction methods, aiming to provide flexible and easy-to-use interfaces for users with different technical levels and meet the needs of different scenarios and users. For example: for an automated system, it supports scripted policy deployment and batch configuration; for advanced users, it provides fine-grained parameter debugging capabilities through the command line mode, such as: forcibly triggering a specified algorithm and dynamically modifying runtime parameters; for non-technical users, it reduces the usage threshold through a graphical interface.

[0070] As a possible implementation, as Figure 3 shown, a data cache processing system architecture diagram based on dynamic adjustment is shown. It mainly includes functional components such as a monitoring module, a feedback loop, a control module, a control algorithm library, and a user interface. Among them, the monitoring module can be used to collect important metrics representing the system state, such as system load, cache occupancy rate, IO waiting latency, cache hit rate, etc.; the feedback loop can timely feedback the data collected by the monitoring module to the control module and record each downbrush rate adjustment and its corresponding system state and the downbrush rate adjustment algorithm used for subsequent analysis and optimization; the control module can obtain the detection metrics from the feedback loop, select a suitable control algorithm from the control algorithm library, dynamically adjust the cache downbrush rate, and transfer the adjustment result to the feedback loop; the control algorithm library can serve as the core decision-making engine, providing a multi-dimensional and extensible set of downbrush rate control algorithms for the control module, supporting multi-mode algorithm integration and user-defined extension; the user interface can, on the one hand, obtain the data collected by the monitoring module from the feedback loop and provide detailed statistical data information and historical record information for the system administrator, and on the other hand, can also provide an algorithm extension configuration interface for the system administrator or developer.

[0071] Among them, the feedback loop timely feeds back the data collected by the monitoring module to the control module. Its main functions include real-time feedback pipeline, historical data lake, and closed-loop effect verification. Real-time feedback pipeline: Adopting shared memory or Remote Direct Memory Access (RDMA) technology, it pushes the metric stream of the monitoring module to the control module in real time to ensure decision-making timeliness; only transmits the change amount (such as the difference in the proportion of dirty data, the amplitude of load fluctuation), reducing data transmission overhead. Historical data lake: Persistently stores all detected raw data, control instruction records, and system response results. Closed-loop effect verification: Conducts post-analysis on each downbrush rate adjustment operation, calculates key benefit indicators such as the reduction amount of dirty data per unit time and the improvement amplitude of storage latency, evaluates the effectiveness of the strategy, and provides it for the control algorithm library to roll back parameters.

[0072] Optionally, a Field-Programmable Gate Array (FPGA) or a Data Processing Unit (DPU) can be used to perform hardware acceleration on computationally intensive tasks such as monitoring data aggregation and control algorithm inference, further improving real-time performance.

[0073] Compared with the existing technologies currently, this embodiment can configure different writing strategies and the corresponding triggering conditions for different writing strategies, select a target control algorithm from the preset algorithm control library to calculate the writing rate, and can also determine whether to downgrade the target control algorithm according to the adjusted system state data, realizing dynamic adjustment of the downbrush rate, optimizing the cache usage efficiency, reducing system pressure, and using the real-time detection and feedback mechanism to ensure the stable operation of the system under dynamic changes, adapting to systems of different scales and complexities, and flexibly coping with load fluctuations.

[0074] An embodiment of the present application also provides a data processing device as Figure 1 a specific implementation of the method shown in Figure 4 As shown, the device includes: an acquisition module 31, a determination module 32, and a writing module 33.

[0075] The acquisition module 31 is configured to acquire system state data during the cache data writing process;

[0076] The determination module 32 is configured to detect the system state data using the triggering conditions corresponding to different writing strategies, and determine the target condition triggered by the system state data from the triggering conditions;

[0077] The determination module 32 is configured to determine the writing rate corresponding to the cache data based on the target writing strategy corresponding to the target condition;

[0078] The flashing module 33 is configured to flash the cached data to the persistent storage device at a flashing rate.

[0079] In some examples of this embodiment, the determining module 32 is specifically configured to configure different flashing policies corresponding to the cached data and the triggering conditions corresponding to different flashing policies; extract the status triggering information corresponding to different triggering conditions from the system status data; use different triggering conditions to respectively detect the status triggering information corresponding to different triggering conditions, and determine the target condition triggered by the system status data according to the detection results.

[0080] In some examples of this embodiment, the determining module 32 is specifically configured to, if it is determined according to the detection results that the target status triggering information satisfies its corresponding triggering condition, determine the triggering condition corresponding to the target status triggering information as the target condition.

[0081] In some examples of this embodiment, the determining module 32 is specifically configured to determine the target control algorithm corresponding to the target flashing policy from a preset control algorithm library, where the preset control algorithm library includes flashing rate control algorithms corresponding to different system states; use the target control algorithm to determine the flashing rate corresponding to the cached data.

[0082] In some examples of this embodiment, the flashing module 33 is further specifically configured to, based on the system status data flashed at the flashing rate, determine whether the target control algorithm satisfies the downgrading condition; if the target control algorithm satisfies the downgrading condition, reduce the algorithm level of the target control algorithm and generate a downgrading alarm corresponding to the target control algorithm.

[0083] In some examples of this embodiment, the flashing module 33 is further specifically configured to display the flashing metrics of the cached data and the mark corresponding to the target control algorithm during the flashing process of the cached data.

[0084] In some examples of this embodiment, the acquisition module 31 is specifically configured to determine the sampling frequency corresponding to the system status data according to the load information corresponding to the system status data; acquire the system status data according to the sampling frequency.

[0085] It should be noted that for other corresponding descriptions of each functional unit involved in the data processing device provided in this embodiment, reference can be made to the corresponding description in Figure 1 and details are not described herein again.

[0086] Based on the method as shown in Figure 1 above, correspondingly, this embodiment further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method as shown in Figure 1 above is implemented.

[0087] Based on the above asFigure 1 The method described above, correspondingly, this embodiment also provides a computer program product, on which a computer program is stored. When the computer program is executed by a processor, it implements the method as described above Figure 1 shown.

[0088] Based on such an understanding, the technical solution of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0089] Based on the method as described above Figure 1 shown, and Figure 4 the virtual device embodiment shown, to achieve the above purpose, this embodiment of the application also provides an electronic device, such as a personal computer, server. This device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the method as described above Figure 1 shown.

[0090] In some embodiments, the above-mentioned physical device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, sensors, an audio circuit, a WI-FI module, and so on. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc. Optionally, the user interface may further include a USB interface, a card reader interface, etc. The network interface may include a standard wired interface, a wireless interface (such as a WI-FI interface), etc. in some embodiments.

[0091] Those skilled in the art can understand that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation to the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0092] The storage medium may further include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to implement communication between components inside the storage medium, and communication between other hardware and software in the information processing physical device.

[0093] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or can also be implemented by hardware. By applying the solution of this embodiment, compared with the current existing technologies, this embodiment can, during the process of flushing cached data, detect system status data in real time, and use the trigger conditions corresponding to different flushing strategies to detect the system status data, determine the target condition triggered by the current system status data, then use the target flushing strategy corresponding to the target condition to determine the current flushing rate of the cached data, and finally flush the cached data to the persistent storage device according to the flushing rate, so as to dynamically select the target flushing strategy from multiple flushing strategies based on the system status data detected in real time by using multiple trigger conditions, and adjust the flushing rate of the cached data in real time, so that the flushing rate of the cached data adapts to the dynamically changing system status, improve the matching degree between the flushing rate and the system status, and further improve the system performance. In addition, different flushing strategies and the trigger conditions corresponding to different flushing strategies can also be configured, the target control algorithm is selected from the preset algorithm control library to calculate the flushing rate, and it can also be determined whether it is necessary to degrade the target control algorithm according to the adjusted system status data, realizing the dynamic adjustment of the flushing rate, optimizing the cache usage efficiency, reducing the system pressure, and ensuring the stable operation of the system under dynamic changes by using real-time detection and feedback mechanisms, adapting to systems of different scales and complexities, and flexibly coping with load fluctuations.

[0094] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0095] The above are only specific embodiments of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that, including: collecting system status data during the caching data flashing process; detecting the system status data by using trigger conditions corresponding to different flashing strategies, and determining a target condition triggered by the system status data from the trigger conditions; determining a flashing rate corresponding to the caching data based on a target flashing strategy corresponding to the target condition; flashing the caching data to a persistent storage device according to the flashing rate; 2. The method according to claim 1, wherein Before the system status data is detected by using trigger conditions corresponding to different flashing strategies and the target condition triggered by the system status data is determined from the trigger conditions, the method further includes: configuring different flashing strategies corresponding to the caching data and trigger conditions corresponding to the different flashing strategies; The detecting the system status data by using trigger conditions corresponding to different flashing strategies and determining the target condition triggered by the system status data from the trigger conditions includes: extracting status trigger information corresponding to different trigger conditions from the system status data; detecting the status trigger information corresponding to the different trigger conditions by using the different trigger conditions, and determining the target condition triggered by the system status data according to the detection results; 3. The method according to claim 2, wherein The detecting the status trigger information corresponding to the different trigger conditions by using the different trigger conditions and determining the target condition triggered by the system status data according to the detection results includes: if it is determined according to the detection results that the target status trigger information meets its corresponding trigger condition, determining the trigger condition corresponding to the target status trigger information as the target condition; 4. The method according to claim 1, wherein The determining the flashing rate corresponding to the caching data based on the target flashing strategy corresponding to the target condition includes: determining a target control algorithm corresponding to the target flashing strategy from a preset control algorithm library, where the preset control algorithm library includes flashing rate control algorithms corresponding to different system statuses; determining the flashing rate corresponding to the caching data by using the target control algorithm; 5. The method according to claim 4, characterized in that, The method further includes: judging whether the target control algorithm meets a downgrading condition based on the system status data flashed at the flashing rate; if the target control algorithm meets the downgrading condition, reducing the algorithm level of the target control algorithm and generating a downgrading alarm corresponding to the target control algorithm; 6. The method according to claim 4, wherein The method further includes: displaying flashing metrics of the caching data and a mark corresponding to the target control algorithm during the flashing process of the caching data; 7. The method according to any one of claims 1 to 6, characterized in that, The collecting the system status data during the caching data flashing process includes: determining a sampling frequency corresponding to the system status data according to load information corresponding to the system status data; collecting the system status data according to the sampling frequency; 8. A data processing device, characterized in that, including: a collecting module configured to collect system status data during the caching data flashing process; a determining module configured to detect the system status data by using trigger conditions corresponding to different flashing strategies and determine a target condition triggered by the system status data from the trigger conditions; A determination module, configured to determine a writing rate corresponding to the cached data based on a target writing policy corresponding to the target condition; A writing module, configured to write the cached data to a persistent storage device according to the writing rate.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein, When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images