A data processing method and a computer storage medium

Abstract

The application relates to a data processing method and a computer storage medium. The method comprises the following steps: acquiring each core of a processor, and creating a corresponding thread for each core, wherein the thread is bound to the corresponding core and is in a silent state; in response to the storage system being in a standby power state, acquiring to-be-processed data, a remaining battery capacity, a battery voltage and an expected available time, and determining a power threshold according to the remaining battery capacity and the expected available time; determining the number of threads to be woken up according to a preset scheduling model, the to-be-processed data and the power threshold; according to a preset allocation strategy, allocating the to-be-processed data to the threads to be woken up to perform a disk-falling operation; in response to the storage system being restarted, detecting whether there is disk-falling data; and in response to the existence of the disk-falling data, loading the disk-falling data into the memory and resuming the execution of a task flow. Through the method, the disk-falling capability can be dynamically adjusted, the energy efficiency ratio can be improved, and the data recovery reliability can be ensured.

Description

Technical Field

[0001] This application relates to the field of computer data storage and recovery technology, and in particular to a data processing method and a computer storage medium. Background Technology

[0002] In modern cloud computing platforms and large-scale storage systems, to ensure the preservation of critical data in memory during sudden power outages, storage systems typically need to securely write data to persistent media within the limited time provided by backup power. This process is called data dumping. Existing storage systems generally use a fixed number of dumping threads to perform data dumping operations. A fixed number of backup threads is preset based on the number of CPU cores, and the data to be dumped is divided into several evenly distributed fragments, which are then written in parallel by different threads to different offset regions of the same dump file.

[0003] However, fixed-thread strategies struggle to balance energy efficiency and performance when dealing with diverse data volumes and storage media. For smaller data volumes, an excessive number of threads leads to high CPU engagement and increased backup battery power consumption. Therefore, existing solutions have significant shortcomings in adaptability, energy consumption control, and implementation reliability, and are also unable to meet the higher requirements of next-generation high-density storage systems for disk write efficiency and energy efficiency ratio. Summary of the Invention

[0004] Therefore, it is necessary to provide a disk writing and recovery method and computer storage medium that can adjust the disk writing capability according to the data scale, improve the energy efficiency ratio, and ensure the reliability of data recovery, in order to address the above-mentioned technical problems.

[0005] Firstly, a data processing method is provided, including:

[0006] Obtain the processor's cores and create corresponding threads for each core. The threads are bound to their corresponding cores and are in a silent state.

[0007] In response to the storage system being in standby power mode, the system acquires the data to be processed, the remaining battery power, the battery voltage, and the estimated available time, and determines the power threshold based on the remaining battery power, the battery voltage, and the estimated available time.

[0008] The number of threads to be woken up is determined based on the preset scheduling model, the data to be processed, and the power threshold.

[0009] According to the preset allocation strategy, the data to be processed is allocated to the awakened thread and the disk write operation is performed;

[0010] In response to a storage system restart, check if any data has been written to disk;

[0011] In response to the existence of data written to disk, the data is loaded into memory and the task execution process is resumed.

[0012] In a second aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the data processing method described in the first aspect.

[0013] By implementing the aforementioned data processing method and computer storage medium, this method pre-creates and binds threads to each processor core during normal operation, keeping these threads silent so they can quickly begin work after entering standby power mode. Upon detecting the entry into standby power mode, the storage system instantly acquires the amount of data to be processed, the current remaining battery power, the battery voltage, and the estimated duration of continuous power supply. Based on this, it calculates an acceptable power threshold to ensure that processing capacity does not exceed the capacity of the backup power supply. After determining the power threshold, the storage system uses a preset scheduling model to dynamically determine the number of threads to be woken up based on the data size, processing requirements, and power consumption limits, thus matching processing capacity with power conditions. According to a preset data allocation strategy, the storage system evenly distributes the data to be processed into fragments to the woken threads, enabling each thread to efficiently execute disk write operations on its corresponding core, achieving a balance between disk write efficiency and power utilization. After power restoration and storage system restart, the system further checks whether there is still data to be written to disk in the persistent storage medium to determine whether the task needs to be resumed. When incomplete disk write operations are detected, the storage system reloads them into memory, resuming the task execution flow. Through the above process, the present invention can not only maintain stable disk writing capability under different power conditions and maximize the effective processing volume per unit power consumption, but also quickly and safely restore the data state when the storage system restarts, thereby improving the operational reliability and overall efficiency of the storage system under abnormal power outage conditions. Attached Figure Description

[0014] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating thread wake-up and disk write-to-data processing in a data processing method provided in this application embodiment;

[0016] Figure 2 A flowchart illustrating thread binding and migration of a data processing method provided in this application embodiment;

[0017] Figure 3 This is a schematic diagram illustrating a multi-threaded data mapping relationship for a data processing method provided in an embodiment of this application.

[0018] Figure 4 This is a diagram showing the internal structure of a computer device in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0020] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or storage system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or storage system. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] In one embodiment, such as Figure 1 As shown, a data processing method is provided, including:

[0023] S100: Obtain the processor's cores and create corresponding threads for each core. The threads are bound to their corresponding cores and are in a silent state.

[0024] S200: In response to the storage system being in standby power mode, acquire the data to be processed, the remaining battery power, the battery voltage, and the estimated available time, and determine the power threshold based on the remaining battery power, the battery voltage, and the estimated available time;

[0025] S300: Determines the number of threads to be woken up based on the preset scheduling model, the data to be processed, and the power threshold;

[0026] S400: Distribute the data to be processed to the awakened thread according to the preset allocation strategy and perform the disk write operation;

[0027] S500: In response to a storage system restart, detects whether there is data written to disk;

[0028] S600: In response to the existence of data written to disk, load the data to disk into memory and resume the execution of the task process.

[0029] Here, a processor core refers to a computing unit within the CPU that can independently execute instructions; each core can run threads independently. A thread is an execution unit that can be independently scheduled within the operating storage system; in this method, it is used to perform data write-to-disk operations. Thread-bound core means fixing a thread to run on a specific core to avoid the overhead of frequent migrations causing context switching. Silent state means that a thread has been created but does not participate in scheduling or consume CPU resources; it is only awakened when needed. Backup power state means that after the main power supply of the storage system is lost, it relies on limited backup power (such as batteries or capacitors) to maintain operation for a short period of time. Power threshold refers to the threshold based on the remaining battery power and battery capacity. The maximum allowable power calculated from the voltage and the estimated available power supply time ensures that the storage system will not lose power due to overload during backup power supply; the preset scheduling model refers to a model based on training, used to infer the optimal number of threads based on task volume and power consumption conditions to maximize energy efficiency; the preset allocation strategy refers to the data partitioning and allocation rules set for data volume and number of threads, used to ensure the balance of parallel task execution; disk write operation refers to writing data in memory to non-volatile storage media (such as SSD) to ensure data security in the event of a power outage; disk written data refers to data content that has been written to persistent storage in backup power state but has not yet completed the complete task process.

[0030] Specifically, after the storage system enters standby power mode, a power threshold is calculated based on the remaining battery power, battery voltage, and estimated power supply time. This threshold, combined with a trained scheduling model, infers the appropriate number of threads based on data scale and power consumption conditions, achieving an adaptive balance between energy consumption and performance during disk write operations. Furthermore, a preset allocation strategy evenly distributes data to the awakened threads, ensuring stable multi-core parallel write operations with controlled energy consumption, thereby maximizing disk write efficiency per unit power consumption. After the storage system restarts, it detects the written data, loads it into memory, and resumes the execution flow, ensuring that tasks can continue to execute in the event of an abnormal power outage, maintaining data consistency and uninterrupted business operations. This significantly improves disk write reliability, energy utilization, and recovery capabilities in sudden power outage scenarios while ensuring data security.

[0031] In one embodiment, determining the power threshold based on the remaining battery capacity, battery voltage, and estimated available time includes:

[0032] The remaining usable energy is obtained by multiplying the remaining battery charge and battery voltage.

[0033] The average power is determined based on the ratio of remaining available energy to expected available time.

[0034] The power threshold is determined by multiplying the average power by the preset power consumption adjustment coefficient.

[0035] Average power consumption refers to the energy consumption level at which the storage system can sustain itself without premature power loss under the constraints of remaining battery power and available time; preset power consumption adjustment coefficient is a parameter used to reflect factors such as actual operating environment, energy decay, and strategy preferences, so that the calculated power threshold is more in line with real needs.

[0036] Specifically, by converting the ratio of remaining battery power to expected available time into average power consumption, real-time quantification of the energy consumption that the storage system can withstand in standby mode is achieved, providing a clear energy consumption constraint basis for subsequent decisions. A preset power consumption adjustment coefficient is further introduced on top of this average power consumption to dynamically correct the power threshold, enabling the storage system to flexibly adjust the available power according to actual scenarios (such as conservative battery life, accelerated disk writes, or leaving a safety buffer). This tiered calculation method allows for reasonable planning of available energy consumption when backup power is limited, avoiding premature power outages due to instantaneous loads exceeding power capacity, and improving the security, stability, and energy efficiency of data writes during standby phases.

[0037] In one embodiment, training a pre-defined scheduling model includes:

[0038] Set up multiple test tasks. Each test task consists of a preset amount of data and a preset number of threads. The amount of data is used to simulate the volume of data to be written, and the number of threads is used to simulate the number of concurrent data writing threads.

[0039] The preset testing tools are used to perform disk write operations on each group of test tasks, and the write time and total power consumption during the execution process are collected for each group of test tasks.

[0040] Based on the collected write time and total power consumption, the write rate and unit power efficiency corresponding to each group of test tasks are calculated, where the write rate is the ratio of data volume to write time, and the unit power efficiency is the ratio of write rate to total power consumption.

[0041] Get test tasks whose total power consumption is less than a preset total power consumption threshold and generate a set of test tasks;

[0042] Sort each test task in the test task set from high to low according to its unit power efficiency, and select the number of threads at the top of the sorting results as the target output value of the test task.

[0043] The training samples are constructed by using the amount of data corresponding to the test task and the preset total power consumption threshold as input features and target output values.

[0044] Multiple training samples are combined into a training sample set, and the machine learning model is trained based on the training sample set to generate a preset scheduling model.

[0045] Among them, the testing tool refers to the software or storage system module used to simulate actual write scenarios and perform disk write operations, which can accurately record performance indicators such as write time and power consumption; the test task refers to the subset obtained from the selected test tasks, which represents the effective sample set that meets the energy consumption conditions within the total power consumption threshold, and is used to further select the optimal number of threads; the training sample set is a set of multiple training samples, which provides a multi-dimensional and multi-condition data input basis for the machine learning model, enabling the model to learn the optimal thread scheduling rules under different scenarios.

[0046] Specifically, by constructing multiple test tasks covering different data volumes and thread counts, and using testing tools to collect write time and energy consumption metrics in real-world disk write scenarios, a reliable foundation of performance and energy consumption data was established. By calculating write rates and unit power efficiency and selecting test tasks with power consumption below a threshold, model training was ensured to be based solely on energy-controlled and effective data samples. Based on unit power efficiency, the number of threads best reflecting the performance-energy balance was selected from each task group as the target output, providing clear optimal scheduling guidance for the training data. These samples were used to construct a training sample set and train a machine learning model, ultimately resulting in a preset scheduling model capable of automatically inferring the optimal thread count based on different data volumes and power consumption constraints. This model achieves a dynamic balance between energy consumption and performance in real-world disk write scenarios, enabling the storage system to maintain efficient writes even with limited backup power, improving data write speed and unit power utilization, while reducing the risk of premature power outages due to excessive energy consumption. This overall enhances the stability and reliability of the storage system under sudden power outage scenarios.

[0047] In one embodiment, before training a machine learning model based on a training sample set to generate a preset scheduling model, the process includes:

[0048] Based on the changes in the number of threads in each training sample in the training sample set, analyze the changing trend of its unit power consumption processing efficiency.

[0049] If the increase in data write rate continues to decrease as the number of threads corresponding to the training samples increases, and the corresponding unit power consumption processing efficiency remains unchanged, then the training samples corresponding to that number of threads in the training sample set are deleted.

[0050] If the test conditions corresponding to the number of threads in the training sample run abnormally and the unit power consumption is lower than the normal trend, then the training sample corresponding to that number of threads in the training sample set will be deleted.

[0051] Among them, the trend of unit power consumption processing efficiency refers to the overall change pattern of unit power consumption efficiency (i.e., the amount of data that can be processed per unit power consumption) as the number of threads gradually increases, showing an increase, stabilization or decrease, which is used to determine whether there is a performance bottleneck or power waste; abnormal test conditions refer to abnormal situations that occur during the execution of test tasks, such as abnormally low write speed, abnormally high power consumption, abnormal thread competition, etc., which cause the collected samples to not truly reflect the normal write performance of the storage system under that number of threads; unit power consumption below the normal trend refers to the unit power consumption efficiency of some training samples being significantly lower than the normal pattern of samples with adjacent number of threads, such as the appearance of a sudden abnormally low efficiency point, which usually means test data deviation, noise interference or instantaneous hardware abnormality.

[0052] Specifically, the storage system analyzes the changing trends of processing efficiency per unit power consumption under different thread counts, identifying and removing samples where the write rate improvement continuously decreases with increasing the number of threads and energy efficiency does not improve. This prevents the model from learning the erroneous pattern of wasted power due to ineffective thread expansion. Simultaneously, samples exhibiting abnormal conditions during testing and deviating from the normal power consumption trend are deleted to avoid model training bias caused by test noise, hardware jitter, or sudden anomalies. Through filtering these two types of invalid samples, the final retained data is more realistic, continuous, and conforms to the performance characteristics of actual thread scheduling, enabling the machine learning model to more accurately learn the mapping relationship between thread count and energy efficiency. The result is a more stable and accurate pre-defined scheduling model that can provide a reasonable number of threads during actual disk writes, thereby improving disk write performance, reducing power consumption, and enhancing the overall reliability and energy efficiency of the storage system in backup power scenarios.

[0053] In one embodiment, such as Figure 2 As shown, the number of threads to be woken up is determined based on a preset scheduling model, the data to be processed, and a power threshold, including:

[0054] Obtain the current data to be processed and the power threshold;

[0055] The data to be processed and the power threshold are used as input features and input to the preset scheduling model;

[0056] The pre-defined scheduling model infers the corresponding number of threads based on the thread number mapping relationship obtained during its training by reasoning about the input features.

[0057] The number of threads is used as the number of threads to be woken up corresponding to the current processing condition.

[0058] The thread number mapping relationship refers to the data pattern learned by the preset scheduling model through training, which is used to map different input features (such as data volume and power threshold) to the most suitable number of threads. Through this mapping relationship, the model can infer the optimal thread configuration according to specific task requirements. Input features refer to the variables or parameters used to train the machine learning model. In this scheme, input features include the size of the data to be processed and the power threshold. Input features are the basis for the model to make inferences.

[0059] Specifically, the storage system acquires the data to be processed and a power threshold as input features and passes them to a pre-defined scheduling model. This model uses the thread count mapping learned during its training to infer the optimal number of threads for the current task. In this way, the model can dynamically adjust the number of threads based on the specific requirements of the task (data size and power consumption limits), achieving more precise and energy-efficient resource scheduling. Under limited power consumption conditions, it ensures that as many threads as possible can be awakened to execute tasks, thereby improving data processing efficiency while avoiding exceeding power consumption limits due to excessive threads. Through this adaptive thread count adjustment, the storage system can achieve efficient data processing even with limited backup power, maximizing resource utilization and ensuring system stability.

[0060] In one embodiment, the data to be processed is allocated to the awakened thread according to a preset allocation strategy, and a disk write operation is performed, including:

[0061] Get the number of currently awakened threads, and divide the data to be processed into multiple data shards based on the number of threads;

[0062] According to the preset allocation strategy, each data shard is allocated to the corresponding awakened thread;

[0063] Create a corresponding dump file for each awakened thread;

[0064] The data is sharded and written to the corresponding dump file according to each awakened thread;

[0065] In response to each awakened thread completing its write operation, the write results of each thread are summarized, and a disk write completion flag is generated and stored in a preset storage location.

[0066] Data sharding refers to dividing the overall data to be processed into multiple smaller data blocks according to certain rules. Each data block can be processed independently by different threads to improve parallel writing efficiency. The specified storage area refers to the storage location that the storage system has specially preset for this disk write, such as a specific logical partition or file area of ​​an SSD, to ensure that the data is written in an orderly manner according to the plan. The disk write completion flag is the status information used to mark that all disk write operations in this round have been completed. It can be a specific flag file, flag bit or metadata, used to determine whether there are completed or incomplete disk write tasks when the storage system restarts.

[0067] Specifically, by dividing the data to be processed into multiple data shards based on the number of awakened threads and allocating these shards to various threads according to a preset allocation strategy, a highly efficient parallel disk write process is achieved. Each thread performs write operations on its assigned data shard, fully utilizing the multi-core parallel capabilities to accelerate the overall write speed. After all threads have completed writing, the storage system automatically summarizes the write results and generates a disk write completion flag, making the status record clear and traceable. In this way, the storage system not only effectively shortens the disk write time but also ensures balanced resource allocation, avoiding thread waiting or uneven load. Simultaneously, the disk write completion flag allows the storage system to quickly determine whether the disk write task has been successfully completed after a restart, enhancing data consistency and reliability. This method comprehensively improves disk write efficiency, reduces energy waste under backup power conditions, and enhances data security and storage system recovery capabilities in abnormal power outage scenarios.

[0068] In one embodiment, after allocating each data shard to its corresponding woken thread, the process includes:

[0069] Acquire the operating status data of each core, including core idle rate and core power consumption;

[0070] Based on the preset energy efficiency calculation model, the operating status data of each core are calculated to obtain the energy efficiency score of each core.

[0071] Based on the energy efficiency score, select cores whose number matches the number of awakened threads as the target core set for the current disk write operation;

[0072] Each awakened thread is bound to a core in the target core set;

[0073] During the disk placement operation, the operating status data of each core is periodically acquired, and the target core set is re-evaluated based on the energy efficiency calculation model.

[0074] In response to the fact that the running status data of any core in the target core set meets the preset migration conditions, the target core set is updated, and migration operations are performed on the threads bound to the non-updated cores.

[0075] Among them, core idle rate refers to the percentage of time a processor core is currently idle, used to measure the load level of the core. The higher the idle rate, the more computing power the core has available. Core power consumption refers to the electrical energy consumed by the processor core in its current operating state, used to measure the energy consumption level of the core. The energy efficiency calculation model is a mathematical model used to comprehensively process multiple indicators such as core idle rate and power consumption, calculate and output the core energy efficiency score, and use it to judge the cost-effectiveness of each core under current conditions (the comprehensive performance and energy consumption). The energy efficiency score refers to the value obtained by the energy efficiency calculation model, which usually reflects the "usable efficiency of the core under the current load and power consumption". The higher the score, the more suitable the core is as a thread execution core. The target core refers to a group of cores selected from all cores with the highest energy efficiency score and the same number as the number of awakened threads, used for thread binding execution of the current disk write operation. The preset migration conditions refer to the conditions that trigger the thread migration operation, such as the core idle rate being too low, the power consumption suddenly increasing, the energy efficiency score decreasing, etc., used to ensure the dynamic optimality of core selection.

[0076] Specifically, the storage system selects target cores based on energy efficiency scores, ensuring that write threads are bound to processor cores with lower load and higher energy efficiency, thereby improving write processing efficiency and energy consumption. During the write process, the storage system periodically monitors the latest operating status of the target cores and reassesses their energy efficiency. If a core is detected to have reached preset migration conditions (such as a significant drop in energy efficiency), the storage system immediately updates the target core list and migrates the threads to new high-efficiency cores, avoiding performance degradation or energy consumption spikes. Through this dynamic core selection combined with real-time migration mechanism, the storage system can continuously maintain the optimal thread execution environment, reducing performance fluctuations caused by single-core congestion or high power consumption. This results in more stable write operations, higher energy efficiency, and more controllable power consumption, while effectively avoiding write speed reductions due to core state changes, thus significantly improving data write reliability and energy efficiency in backup power environments.

[0077] In one embodiment, in response to the presence of data written to disk, the data is loaded into memory and the task execution flow is resumed, including:

[0078] In response to the detection of data being written to disk, the metadata information of the data being written to disk is read, including: multiple data shards and the thread identifier corresponding to each data shard;

[0079] Based on the thread identifier, retrieve the data fragments corresponding to each thread from the dump file at the preset storage location, and load the data fragments into the buffer area in memory associated with the corresponding thread;

[0080] Based on the data slices loaded by each thread, a state reconstruction operation is performed. In response to the completion of the state reconstruction of each thread, the task flow before the interruption is resumed.

[0081] Metadata information refers to control data describing the data structure and associated information written to disk. In this solution, it includes data shards and corresponding thread identifiers. The buffer area refers to a region in memory specifically used to temporarily store data shards, providing read and write space for subsequent thread processing and avoiding additional performance loss caused by direct operation on the storage device. The state reconstruction operation refers to the process by which a thread restores its running context before the interruption based on the loaded data shard content before resuming execution, including write position, processing progress, and internal thread variables, so that it can continue execution from the interruption point without having to start over.

[0082] Specifically, by reading metadata information when disk-written data is detected and reloading data fragments into the corresponding memory buffers based on the metadata, precise recovery of interrupted tasks is achieved. After loading its corresponding data fragment, the thread performs state reconstruction, enabling it to continue working from the interrupted point without reprocessing completed data, thus reducing recovery time. Once state reconstruction is complete, the storage system can resume the task flow before the interruption, achieving smooth continuation of writes. Overall, this method enables efficient and accurate task recovery after abnormal power outages or task interruptions, shortens storage system recovery time, improves data consistency, avoids redundant calculations, and enhances the reliability and recoverability of the storage system under sudden events.

[0083] In one embodiment, the method further includes loading the data to disk into memory before:

[0084] Based on the metadata information of the data written to disk, the integrity of the data written to disk is verified. The verification includes: obtaining the actual number of data fragments, comparing the actual number of data fragments with the number of data fragments recorded in the metadata information, and generating a current check code for each data fragment based on a preset verification algorithm, and comparing the current check code with the corresponding check code recorded in the metadata information to confirm whether each data fragment is complete.

[0085] In response to the fact that the actual number of data fragments is consistent with the number of data fragments recorded in the metadata information, and that the current checksum of each data fragment matches the checksum of the corresponding record, the step of loading the data to disk into memory is executed.

[0086] In response to a discrepancy between the actual number of data fragments and the number of data fragments recorded in the metadata information, an error flag is generated, and the recovery operation based on the data written to disk is terminated;

[0087] In response to the actual number of data fragments matching the number of data fragments recorded in the metadata information, if the current checksum of at least one data fragment does not match the checksum of its corresponding record, an error flag is generated, and the recovery operation based on the data written to disk is terminated.

[0088] The checksum is a feature value generated by a preset verification algorithm (such as CRC, hash value, etc.) based on the data content. It is used to determine whether the data has been tampered with, lost, or damaged during storage and transmission. The preset verification algorithm is an algorithm used to perform integrity verification on the data. It can be CRC, MD5, SHA, etc. Its main function is to generate a checksum based on the data content. Even if the differences between different data are very small, their checksums will be significantly different. The error flag is a status flag generated when incomplete data or verification failure is detected. It is used to notify the storage system that the data currently written to disk cannot be used for recovery, avoiding storage system errors or data pollution caused by loading corrupted data. The actual number of data fragments refers to the number of data fragments contained in the data actually written to disk detected by the storage system. It is used to compare with the number of metadata records to determine whether there is data loss or abnormality in the disk writing process.

[0089] Specifically, by performing rigorous data integrity checks before loading data onto disk, the reliability of the recovery process is effectively guaranteed. The storage system reads the expected number of data shards based on metadata information and compares it with the number of shards detected in the actual data being written to disk, determining from the source whether data is missing. Simultaneously, a preset verification algorithm generates a current checksum for each data shard and compares it with the checksum recorded in the metadata, thereby detecting whether the data was corrupted or incompletely written during the disk writing process. Only when the number of shards matches and all checksums match completely will the storage system continue loading data and executing the recovery task; if the number of shards is inconsistent or the checksums do not match, an error flag is immediately generated and the recovery operation is terminated to avoid introducing corrupted data that could lead to task anomalies or storage system crashes. Through this dual verification mechanism (quantity verification and checksum verification), the storage system can accurately determine whether the data is complete and reliable before recovery, thus ensuring the security and consistency of the recovery process. This improves the reliability of data recovery, prevents execution errors caused by incomplete data, reduces storage system failures and redundant processing, and enhances the data security and recovery capabilities of the storage system in the event of sudden power outages or abnormal interruptions.

[0090] In one embodiment, when the storage system starts, N cores are created based on the number of CPU cores. Each thread is bound to one CPU core and is placed in a silent state after creation, and is woken up during power backup. During power backup, the optimal number of threads N is determined, and the first N threads are woken up to read and write N harden files respectively, from / data / hardendata0 to / data / hardendataN. Each thread processes one file. For example, during power backup, thread 0 writes to file / data / hardendata0 and thread 1 writes to file / data / hardendata1. During recovery, thread 0 reads file / data / hardendata0 and thread 1 reads file / data / hardendata1, and so on. During power backup, the number of pages in memory that need to be written to disk is divided into N parts. Each thread determines its own startpagenum and endpagenum, and finally saves the allocated pages to the file. The numpages saved in each file are recorded in the file header. During recovery, the parameters of each thread are set: the starting position of the page restored by thread 0: threads[0].startpagenum=1, length: numpages in the header file of / data / hardendata0. Thread 1 restores the starting position of the page: threads[1].startpagenum = 1 + numpages from the header file / data / hardendata0; length: numpages from the header file / data / hardendata1. Similarly, different threads operate on different page regions, reading and writing different files, achieving the effect of mutual independence. For example... Figure 3 As shown, when N=8, the thread and CPU core binding relationship, as well as the memory region and dump file being operated on.

[0091] In one embodiment, before resuming the task flow before the interruption, the method further includes:

[0092] Obtain the number of available cores of the current processor and their running status data;

[0093] The number of threads awakened after recovery is calibrated based on the current storage system status;

[0094] Based on the calibrated number of wake-up threads, select the same number of processor cores from the currently available cores as execution cores.

[0095] Rebind each woken-up thread to the execution core;

[0096] In response to the completion of core binding, the task flow resumes execution steps before the interruption.

[0097] Among them, the number of currently available cores refers to the number of cores in the processor that are in an executable state, not occupied by high-priority tasks, and available for use by the current task at the time of task resumption; the number of calibrated wake-up threads refers to the dynamic adjustment of the number of threads obtained after resumption based on the real-time resource status of the storage system (such as core availability, power consumption limits, and load status) before resumption, so as to match the concurrency that the current storage system can support; and the number of execution cores refers to the set of cores reselected from the available cores based on the calibrated number of threads, which are used to actually run each thread when the task is resumed.

[0098] Specifically, by acquiring the current number of available processor cores and their operational status data before resuming the task flow prior to the interruption, the storage system can accurately grasp the real-time hardware resource conditions before task resumption. Based on this, the number of awakened threads obtained after resumption is dynamically calibrated, ensuring that the number of threads matches the actual computing resources available to the storage system. Subsequently, the storage system reselects the corresponding number of execution cores based on the calibrated number of threads and rebinds each awakened thread to the execution core, allowing threads to run on cores with more balanced load and sufficient resources. Through this mechanism, a scheduling optimization and resource adaptation based on the real-time storage system status is completed before entering task resumption execution, avoiding resource conflicts or scheduling imbalances caused by directly using the core binding relationships before the interruption. This effectively improves the stability and reliability of the task resumption process, shortens the resumption time, reduces performance losses caused by resource mismatch, and ultimately significantly enhances the storage system's adaptability and continuous operation capability under abnormal interruption scenarios.

[0099] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments when it is run.

[0100] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0101] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both, such as Figure 4As shown, to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0102] The above provides a detailed description of a signal compatibility method for a graphics processing unit provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain each core of the processor and create a corresponding thread for each core, wherein the thread is bound to the corresponding core and is in a silent state; In response to the storage system being in standby power mode, the system acquires the data to be processed, the remaining battery power, the battery voltage, and the estimated available time, and determines a power threshold based on the remaining battery power, the battery voltage, and the estimated available time. The number of threads to be woken up is determined based on the preset scheduling model, the data to be processed, and the power threshold. The data to be processed is allocated to the awakened thread according to the preset allocation strategy, and the disk write operation is performed. In response to the storage system restarting, check if there is any data written to disk; In response to the presence of data written to disk, the data is loaded into memory, and the task execution process is resumed. The training of the pre-defined scheduling model includes: Multiple test tasks are set up, each consisting of a preset amount of data and a preset number of threads. The amount of data is used to simulate the volume of data to be written, and the number of threads is used to simulate the number of concurrent data writing threads. The test tasks in each group are written to disk using a preset testing tool. The write time and total power consumption during the execution of each group of test tasks are collected. Based on the collected write time and total power consumption, the write rate and unit power efficiency corresponding to each group of test tasks are calculated, wherein the write rate is the ratio of the data volume to the write time, and the unit power efficiency is the ratio of the write rate to the total power consumption. Obtain test tasks whose total power consumption is less than a preset total power consumption threshold and generate a set of test tasks; The test tasks in the test task set are sorted from high to low according to their unit power efficiency, and the number of threads at the top of the sorting result is selected as the target output value of the test task. The training samples are constructed by using the data volume corresponding to the test task and the preset total power consumption threshold as input features and the target output value; The training samples are combined into a training sample set, and the machine learning model is trained based on the training sample set to generate the preset scheduling model.

2. The data processing method according to claim 1, characterized in that, The step of determining the power threshold based on the remaining battery capacity, the battery voltage, and the estimated available time includes: The remaining usable energy is obtained by multiplying the remaining battery charge and the battery voltage. The average power is determined based on the ratio of the remaining available energy to the estimated available time; The power threshold is determined by multiplying the average power by the preset power consumption adjustment coefficient.

3. The data processing method according to claim 1, characterized in that, Before training the machine learning model based on the training sample set to generate the preset scheduling model, the process includes: Based on the changes in the number of threads in each training sample in the training sample set, analyze the changing trend of its unit power consumption processing efficiency. If the increase in the data write rate continues to decrease as the number of threads corresponding to the training sample increases, and the corresponding unit power consumption processing efficiency remains unchanged, then the training sample corresponding to that number of threads in the training sample set is deleted. If the test conditions corresponding to the number of threads in the training sample are running abnormally and the unit power consumption is lower than the normal trend, then the training sample corresponding to that number of threads in the training sample set is deleted.

4. The data processing method according to claim 1, characterized in that, The step of determining the number of threads to be woken up based on the preset scheduling model, the data to be processed, and the power threshold includes: Obtain the current data to be processed and the power threshold; The data to be processed and the power threshold are input as input features into the preset scheduling model; The preset scheduling model infers the corresponding number of threads based on the thread number mapping relationship obtained through its training by reasoning about the input features. The number of threads is used as the number of threads to be woken up corresponding to the current processing condition.

5. The data processing method according to claim 4, characterized in that, The step of allocating the data to be processed to the awakened thread according to a preset allocation strategy and performing a disk write operation includes: Obtain the number of currently awakened threads, and divide the data to be processed into multiple data shards based on the number of threads; According to the preset allocation strategy, each data shard is allocated to the corresponding awakened thread; Create a corresponding dump file for each of the awakened threads; The data fragments are written to the corresponding dump files according to each awakened thread; In response to each of the awakened threads completing the write operation, the write results of each thread are summarized, a disk write completion flag is generated, and the disk write completion flag is stored in a preset storage location.

6. The data processing method according to claim 5, characterized in that, The data fragments are divided separately After being assigned to the corresponding awakened thread, the process includes: Acquire the operating status data of each core, wherein the operating status data includes core idle rate and core power consumption; Based on a preset energy efficiency calculation model, the operating status data of each core are calculated to obtain the energy efficiency score of each core. Based on the energy efficiency score, select the cores whose number is the same as the number of threads that are woken up, as the target core set for the current disk write operation; Each awakened thread is then bound to a core in the target core set. During the disk placement operation, the operating status data of each core is periodically acquired, and the target core set is re-evaluated based on the energy efficiency calculation model. In response to the fact that the running status data of any of the cores in the target core set meets the preset migration conditions, the target core set is updated, and the threads bound to the non-updated cores are migrated.

7. The data processing method according to claim 5, characterized in that, The step of responding to the existence of data written to disk, loading the data to disk into memory, and resuming the task execution flow includes: In response to the detection of the existence of the disk-written data, the metadata information of the disk-written data is read, wherein the metadata information includes: multiple data shards and thread identifiers corresponding to each data shard; Based on the thread identifier, the data fragments in the dump file corresponding to each thread are obtained from the preset storage location, and the data fragments are loaded into the buffer area in memory associated with the corresponding thread; Each thread performs a state reconstruction operation on the data slices it has loaded. In response to the completion of the state reconstruction of each thread, the task flow before the interruption is resumed.

8. The data processing method according to claim 7, characterized in that, Before loading the disk-written data into memory, the process also includes: Based on the metadata information of the disk-written data, the integrity of the disk-written data is verified, and the verification includes: Obtain the actual number of data shards, and compare the actual number of data shards with the number recorded in the metadata information. The number of data fragments recorded is compared, and for each data fragment, a current check code is generated based on a preset check algorithm, and the current check code is compared with the corresponding check code recorded in the metadata information to confirm whether each data fragment is complete. In response to the fact that the actual number of data fragments is consistent with the number of data fragments recorded in the metadata information, and the current checksum of each data fragment matches the checksum of the corresponding record, the step of loading the data written to disk into memory is executed. In response to the discrepancy between the actual number of data fragments and the number of data fragments recorded in the metadata information, an error flag is generated, and the recovery operation based on the data written to disk is terminated. In response to the actual number of data fragments being consistent with the number of data fragments recorded in the metadata information, if the current checksum of at least one data fragment does not match the checksum of its corresponding record, an error flag is generated, and the recovery operation based on the disk-written data is terminated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images