Data processing method and device, electronic device and storage medium

Abstract

The present application discloses a data processing method and apparatus, an electronic device, and a storage medium, relating to the field of data processing technology. The method includes obtaining equivalent values ​​of at least two data to be collected in at least two storage devices, and obtaining the number of available threads of the at least two storage devices; the equivalent value is the number of threads required to collect the data to be collected; based on the equivalent values ​​of the at least two data to be collected, at least one target data to be collected is selected from the at least two data to be collected in a manner that makes the number of available threads approach a minimum value; the number of available threads is negatively correlated with the sum of the equivalent values ​​of the at least one selected target data to be collected; and data collection is performed on the at least one target data to be collected using the number of available threads, thereby reducing thread idleness and improving thread utilization. Therefore, the technical problem of low thread utilization can be solved, achieving the technical effect of improving thread utilization.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device, electronic equipment, and storage medium. Background Art

[0002] There are multiple storage devices in the storage system. In order to ensure the security of data in the multiple storage devices, it is necessary to collect data from the multiple storage devices for centralized management.

[0003] In data processing related technologies, data is usually collected from storage devices one by one in a fixed order. Since the amount of data stored in different storage devices is inconsistent, the threads consumed by different storage devices for data collection are inconsistent, resulting in low thread utilization. Summary of the Invention

[0004] The present application provides a data processing method and apparatus, an electronic device, and a storage medium to at least solve the problem of low thread utilization in related technologies.

[0005] This application provides a data processing method, including:

[0006] Obtaining equivalent values ​​of at least two pieces of data to be collected in at least two storage devices, and obtaining the number of available threads of the at least two storage devices; the equivalent value is the number of threads required to collect the data to be collected;

[0007] Based on equivalent values ​​of the at least two data to be collected, at least one target data to be collected is selected from the at least two data to be collected in a trend that makes the number of available threads approach a minimum value; the number of available threads is negatively correlated with the sum of the equivalent values ​​of the at least one selected target data to be collected;

[0008] Using the available number of threads, data collection is performed on at least one target data to be collected.

[0009] The present application also provides a data processing device, comprising:

[0010] An acquiring unit, configured to acquire equivalent values ​​of at least two pieces of data to be collected from at least two storage devices, and to acquire the number of available threads of the at least two storage devices; the equivalent value is the number of threads required to collect the data to be collected;

[0011] a selection unit configured to select at least one target data to be collected from the at least two data to be collected based on equivalent values ​​of the at least two data to be collected and in a manner that makes the number of available threads approach a minimum value; the number of available threads is negatively correlated with the sum of the equivalent values ​​of the at least one selected target data to be collected;

[0012] The collection unit is used to collect data of at least one target to be collected by utilizing the available number of threads.

[0013] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data processing methods when executing the computer program.

[0014] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0015] The present application also provides a computer program product, comprising a computer program, which implements the steps of any of the above-mentioned data processing methods when executed by a processor.

[0016] Through the present application, by obtaining equivalent values ​​of at least two data to be collected from at least two storage devices and obtaining the number of available threads of the at least two storage devices; the equivalent value is the number of threads required to collect the data to be collected; based on the equivalent values ​​of the at least two data to be collected, at least one target data to be collected is selected from the at least two data to be collected in a manner that makes the number of available threads approach a minimum value; the number of available threads is negatively correlated with the sum of the equivalent values ​​of the at least one selected target data to be collected; and data collection is performed on the at least one target data to be collected using the number of available threads, thereby reducing thread idleness and thereby improving thread utilization. Therefore, the technical problem of low thread utilization can be solved, achieving the technical effect of improving thread utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of the entire data processing process provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0024] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0025] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends, the specific application environment architecture or specific hardware architecture is described here.

[0026] The embodiments of the present application provide a data processing method, and the method is described in detail in conjunction with the execution flow of the data processing method.

[0027] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application.

[0028] like Figure 1 As shown, the method comprises the following steps:

[0029] Step 101: Obtain equivalent values ​​of at least two data to be collected in at least two storage devices, and obtain the number of available threads of the at least two storage devices; the equivalent value is the number of threads required to collect the data to be collected.

[0030] In this embodiment, storage devices can be various physical data storage devices such as hard drives, solid-state drives, and tape libraries. These devices are used to store data of varying types and quantities to meet the data storage needs of various business scenarios. Data to be collected refers to data stored in the storage device that needs to be collected and processed. The collected data can include files, database records, log information, and other information. The equivalent value refers to the computing resources (e.g., number of threads) required to collect a specific piece of data. The equivalent value is determined based on factors such as the data type, data size, and complexity of data collection, reflecting the "workload" required to collect the data. The number of threads refers to the computing resource units used to process data in parallel during the data collection process. Each thread can independently execute tasks, and multiple threads can collect data simultaneously, thereby improving data collection efficiency. The number of available threads refers to the number of thread resources currently available in the system for data collection. The number of available threads is limited by system hardware resources (e.g., number of CPU cores, memory, etc.).

[0031] Identify the multiple storage devices included in the system, for example, storage device A, storage device B, and storage device C. In each storage device, identify the data to be collected, such as data a1 and a2 in storage device A, and data b1 and b2 in storage device B. Record these data in a data list. Calculate the equivalent value for each piece of data to be collected. This calculation can take into account various factors, such as data volume, data type, and business metrics. For example, larger data volumes typically require more time and resources to collect. Therefore, quantify the size of each piece of data to be collected and determine its weight within the data volume dimension. Similarly, assign weights to different data types, such as text, image, and video, based on their different collection methods and complexity. Business metrics, such as the urgency and importance of the data, are scored and quantified based on business needs. Finally, combine the weights of these different dimensions to determine the equivalent value for the data to be collected. For example, for data a1 to be collected, its data volume weight is 0.4, its data type weight is 0.3, and its business indicator weight is 0.3. After corresponding calculations, its equivalent value is 5. The system obtains the number of threads currently available for data collection tasks through real-time monitoring or periodic queries to the task scheduling system. This number changes dynamically based on the overall system task performance. For example, if the system currently has 20 threads, 8 of which are executing other non-data collection tasks, the number of available threads is 12.

[0032] By obtaining the equivalent value of each data point to be collected and the number of available threads, threads can be allocated appropriately based on the actual data requirements and the availability of system resources. This avoids the low thread utilization caused by traditional sequential collection, enabling more efficient and reasonable use of limited thread resources, improving the efficiency of the entire data collection process.

[0033] Step 102 : Based on the equivalent values ​​of the at least two data to be collected, at least one target data to be collected is selected from the at least two data to be collected in a manner that makes the number of available threads approach a minimum value; the number of available threads is negatively correlated with the sum of the equivalent values ​​of the at least one selected target data to be collected.

[0034] Approaching the minimum value means maximizing thread resource utilization by combining tasks to minimize the number of available threads to zero (ideally, the sum of the equivalent values ​​equals the number of available threads). A negative correlation means there is an inverse relationship between the number of available threads and the sum of the equivalent values ​​of the selected target data to be collected. When the sum of the equivalent values ​​of the target data is large, the number of available threads in the system tends to be small, and vice versa.

[0035] First, calculate the equivalent value of the data to be collected on each storage device using the aforementioned method. Simultaneously, obtain the current number of available threads through system monitoring or querying the task scheduling system. For example, the equivalent value of data A to be collected on storage device 1 is calculated to be 5, and the equivalent value of data B to be collected is 3; the equivalent value of data C to be collected on storage device 2 is 4, and the equivalent value of data D to be collected is 6. The current number of available threads in the system is 10.

[0036] From all the data to be collected, find the data to be collected whose equivalent value is less than or equal to the number of available threads, and sort them from large to small according to the equivalent value. If the number of available threads is 10, the equivalent values ​​of the data to be collected in the above example are all less than or equal to 10, and are sorted as D (6), A (5), C (4), B (3). Start selecting from the data with the largest equivalent value, first select data D, whose equivalent value is 6. At this time, the number of used threads is 6, and the number of remaining available threads is 4. Within the remaining available threads, continue to select the data with the largest equivalent value that does not exceed the number of remaining available threads from the remaining data to be collected. For example, at this time, the number of available threads is 4. Among the remaining data to be collected, the equivalent value of data A is 5, which exceeds the number of remaining available threads 4 and cannot be selected; the equivalent value of data C to be collected is 4, which meets the conditions, so data C to be collected is selected, the cumulative number of used threads is 6+4=10, and the number of remaining available threads is 0. If there are still remaining available threads at this time, repeat this step until there are no remaining available threads or all the data to be collected have been considered. If the equivalent value of no data to be collected is less than or equal to the number of available threads, it may mean that the number of available threads is too small to meet the minimum thread requirement of a single data to be collected. In this case, you need to wait until the thread finishes collecting data and then update the number of available threads so that the updated number of threads can process the remaining data to be collected.

[0037] The target data to be collected is selected in such a way that the number of available threads approaches the minimum value, so that as many data collection tasks as possible can be completed within the limited thread resources, thus avoiding idle and wasted threads, improving thread utilization, and enhancing the overall performance and efficiency of the system.

[0038] Step 103: Use the available number of threads to collect data for at least one target to-be-collected data.

[0039] The target data to be collected refers to the data that, after the previous selection process, is identified from the pool of data to be collected and that can be efficiently collected within the constraints of the currently available number of threads. Data collection refers to the process of extracting information from the target data to be collected. Data collection can include multiple steps such as reading, processing, storage, and analysis. The goal is to collect and process data scattered across the system or storage devices for subsequent analysis and use.

[0040] The system first obtains the current number of available threads. This value can be obtained from the operating system's thread management module. Assuming the current system has N CPU cores, each supporting M threads, the number of available threads may be affected by factors such as system load and other application usage. It is typically calculated as N * M minus the number of currently used threads. Based on the data collection requirements and priority, the system selects at least one target data item for collection. This selection may be based on factors such as data size, complexity, storage location, and access frequency. For example, if certain data is stored on a remote server, collection of this data may be prioritized. The system allocates thread resources appropriately based on the current number of available threads. For each target data item, the system allocates an appropriate number of threads based on its complexity, size, and priority. If the data collection task is complex or involves multiple processing steps (such as decompression or large-scale data conversion), the system may allocate more threads to the task. The specific thread allocation algorithm can be based on priority-weighted round-robin or dynamic load balancing. For example, when the data volume is large, the system can adopt a divide-and-conquer strategy, splitting the data into multiple subtasks and assigning different threads to each task for parallel processing. Once thread resources are allocated, the system begins executing the data collection task. Each thread is responsible for collecting a portion of the target data. The order and dependencies between tasks are managed by a scheduling algorithm. If collection tasks have dependencies, the system ensures that the threads execute in the correct order to avoid conflicts and errors. During the data collection process, the system dynamically adjusts the number of available threads based on the progress of the tasks. If some threads complete their tasks, the system can allocate idle threads to other unfinished collection tasks, thereby accelerating the overall collection process. If certain tasks encounter bottlenecks (such as delays) during the collection process, the system will appropriately reduce the number of threads for these tasks and allocate resources to other tasks that can be processed in parallel. When all tasks for the target data to be collected are completed, the system aggregates the collection results, stores the data in a designated storage location, and performs subsequent data processing and analysis. The completion of a collection task also means that available threads are reclaimed, making thread resources available for other tasks.

[0041] By properly allocating the number of available threads, the system can maximize parallelism during the data collection process. Efficient use of thread resources can significantly increase the speed of data collection, especially when processing large amounts of data or frequently accessed data, allowing the system to complete tasks quickly.

[0042] Through the present application, by obtaining equivalent values ​​of at least two data to be collected from at least two storage devices and obtaining the number of available threads of the at least two storage devices; the equivalent value is the number of threads required to collect the data to be collected; based on the equivalent values ​​of the at least two data to be collected, at least one target data to be collected is selected from the at least two data to be collected in a manner that makes the number of available threads approach a minimum value; the number of available threads is negatively correlated with the sum of the equivalent values ​​of the at least one selected target data to be collected; and data collection is performed on the at least one target data to be collected using the number of available threads, thereby reducing thread idleness and thereby improving thread utilization. Therefore, the technical problem of low thread utilization can be solved, achieving the technical effect of improving thread utilization.

[0043] As a refinement of step 101, when executing the acquisition of equivalent values ​​of at least two data to be collected in at least two storage devices, it can be implemented by but not limited to the following methods, including: obtaining the time consumption coefficient of at least two data to be collected, and obtaining the data type weight and business indicator weight of at least two data to be collected; calculating the equivalent value based on the time consumption coefficient, data type weight and business indicator weight.

[0044] The time coefficient reflects the relative amount of time spent collecting the data to be collected. It can be used to measure differences in the time required to collect different types of data. A larger value indicates a longer collection time for the data to be collected. Data type weights refer to the weights assigned to different data types based on their importance or priority. Different types of data have varying degrees of importance in business processes, so each data type requires a different weight. A higher weight indicates a greater impact on the business. Business indicator weights refer to the weights assigned to different business indicators based on business needs. Business indicator weights measure the contribution of data to business objectives and are typically correlated with business goals and key performance indicators. A higher weight indicates a greater impact on business success. Business indicator weights reflect the importance of the data to be collected to the business. Different data types are assigned different weights based on business needs and the purpose of the data to prioritize key data during the collection process.

[0045] Collect historical data collection time records for the data to be collected and calculate their average as the historical average collection time. Also, find the median of all historical collection time records to determine the baseline collection time. Then, divide the historical average collection time by the baseline collection time to obtain the time coefficient. For example, if the historical average collection time for a particular data item is 100 seconds and the baseline collection time is 80 seconds, the time coefficient is 100 ÷ 80 = 1.25. Based on common data types in the storage system, pre-define weights for each data type. For example, assign a weight of 0.3 to text data, 0.5 to image data, and 0.7 to video data. Based on the actual data type to be collected, directly obtain the corresponding data type weight. The business department evaluates and scores the data based on its importance to the business. For example, core business data can be assigned a higher weight, such as 0.8, while auxiliary business data can be assigned a lower weight, such as 0.3. Substitute the obtained time coefficient, data type weight, and business indicator weight into the formula for calculation. First, the data type weight and the business indicator weight are added together to obtain the sum. This sum is then multiplied by the time coefficient to obtain the equivalent value. For example, if the time coefficient of the data to be collected is 1.25, the data type weight is 0.5, and the business indicator weight is 0.6, the sum is 0.5 + 0.6 = 1.1, and the equivalent value is 1.25 × 1.1 = 1.375.

[0046] By comprehensively considering the three key factors of time consumption coefficient, data type weight, and business indicator weight, we can more accurately assess the actual amount of resources required to collect different data to be collected, avoid the inaccuracy caused by estimating the number of threads based on a single factor (such as data volume), and improve the rationality of resource allocation.

[0047] As a refinement of the above embodiment, when calculating the equivalent value based on the time consumption coefficient, data type weight and business indicator weight, it can be implemented in but not limited to the following ways, including: adding the data type weight and the business indicator weight to obtain the sum result; producting the sum result and the time consumption coefficient to obtain the equivalent value.

[0048] To facilitate understanding, let's use an example: assume the data type weight of the data to be collected is 0.3 (for example, text data), the business indicator weight is 0.5 (for example, more important business data), and the time coefficient is 1.25 (for example, calculated based on the ratio of the historical average collection time to the benchmark collection time). Add the data type weight of 0.3 and the business indicator weight of 0.5 to obtain a sum of 0.8. Multiply this sum of 0.8 by the time coefficient of 1.25 to obtain an equivalent value of 1.0. This calculated equivalent value of 1.0 is recorded in the properties or configuration file of the data to be collected, so that it can be used as a basis for thread allocation during subsequent data collection.

[0049] By adding up the comprehensive weights and then multiplying them by the time consumption coefficient to obtain the equivalent value, we can comprehensively consider the time consumption and business importance of data collection and provide a more accurate resource demand assessment for data collection tasks.

[0050] As a refinement of the above embodiment, when executing the acquisition of the time consumption coefficient of at least two data to be collected, it can be implemented by but not limited to the following methods, including: obtaining the historical average collection time of at least two data to be collected, and obtaining the benchmark collection time of at least two data to be collected; the benchmark collection time is the median of all historical collection times of at least two data to be collected; the quotient of the historical average collection time and the benchmark collection time is calculated to obtain the time consumption coefficient.

[0051] The historical average collection time refers to the average time spent on a particular piece of data to be collected, calculated across multiple historical collection records. This value reflects the average time cost of collecting that data in the past. The benchmark collection time refers to the middle value among all the historical collection time records for the data to be collected, sorted in order of magnitude. If the number of records is odd, the middle value is used; if it is even, the average of the two middle values ​​is typically used. The benchmark collection time effectively represents the typical level of collection time for that data and is less affected by extreme values. The time coefficient is the quotient of the historical average collection time and the benchmark collection time, measuring the relative relationship between the collection time of the data to be collected and the typical level. A time coefficient greater than 1 indicates that the data collection time is long; a coefficient less than 1 indicates that the collection time is short.

[0052] After each collection task for pending data on each storage device is completed, the system automatically records the collection time for each pending data item and stores it in a historical collection time database. For example, if the first collection of pending data item 1 on storage device A takes 10 seconds, the second takes 15 seconds, and the third takes 12 seconds, all three collection times will be recorded in the database. To obtain the historical average collection time for a particular pending data item, the system retrieves all historical collection time records for that data item from the database, adds these values, and divides them by the total number of records. For example, the historical average collection time for pending data item 1 is (10 + 15 + 12) / 3 = 12.3 seconds. The system also retrieves all historical collection time records for that data item from the database and sorts them in ascending order. If the number of records is odd, for example, if the five time-consuming records are sorted as [8, 10, 12, 14, 16], the baseline collection time is the middle value, 12 seconds. If the number of records is even, for example, if the four time-consuming records are sorted as [9, 11, 13, 15], the baseline collection time is the average of the two middle values ​​(11 + 13) / 2 = 12 seconds. After obtaining the historical average collection time and the baseline collection time, divide the historical average collection time by the baseline collection time to obtain the time coefficient. For data 1 to be collected, assuming its historical average collection time is 12.3 seconds and the baseline collection time is 12 seconds, the time coefficient = 12.3 / 12 = 1.025.

[0053] By calculating the historical average collection time and the benchmark collection time, we can fully and accurately understand the historical performance and typical level of the collection time of the data to be collected, providing a reliable basis for the subsequent reasonable arrangement of collection tasks and allocation of resources.

[0054] As a refinement of step 102, when executing the equivalent values ​​of at least two data to be collected, and selecting at least one target data to be collected from at least two data to be collected in accordance with a trend of making the number of available threads approach the minimum value, it can be implemented in but not limited to the following manner, including: determining whether there is at least one target equivalent value in the equivalent values; the target equivalent value is an equivalent value less than or equal to the number of available threads; when there is at least one target equivalent value in the equivalent values, selecting the target data to be collected from the data to be collected corresponding to the at least one target equivalent value in descending order of the target equivalent values; wherein the sum of the target equivalent values ​​of the selected target data to be collected does not exceed the number of available threads; and selecting the target data to be collected from the remaining data to be collected in accordance with the trend of making the number of available threads approach the minimum value.

[0055] For ease of understanding, an example is provided. The system determines that the number of available threads is 10 based on the current running state. For multiple data to be collected, their equivalent values ​​are: data A is 5, data B is 3, data C is 4, data D is 6, and data E is 2. The system filters out the data to be collected whose equivalent values ​​are less than or equal to the number of available threads (10), namely data A (5), data B (3), data C (4), data D (6), and data E (2). The system sorts the filtered data from large to small according to the equivalent value: data D (6), data A (5), data C (4), data B (3), data E (2). Select the data in the sorted order, first select data D (6), the number of used threads is 6, and the number of remaining available threads is 4. Then select data A (5), but the number of used threads will reach 11, which exceeds the number of available threads 10, so it cannot be selected. Then try to select data C (4), the number of used threads becomes 10, and the number of remaining available threads is 0. The total target equivalent value of the target data to be collected is 6 + 4 = 10, which is exactly equal to the number of available threads, 10. If new thread resources are released during data collection, increasing the number of available threads, the system can select new target data from the remaining data to be collected according to the above rules. The system continuously monitors data collection progress and thread usage, and repeats the above steps until all data to be collected is collected.

[0056] By reasonably selecting the target data to be collected, we can ensure that within the limit of the number of available threads, we can utilize thread resources as much as possible for data collection and reduce thread idleness.

[0057] In practical applications, the equivalent value of the data to be collected is dynamically adjusted, and the adjustment method can be implemented by, but is not limited to, the following methods, including: obtaining the target mode coefficient of the target performance mode of the storage device where the data to be collected is located; different performance modes correspond to different mode coefficients; the equivalent value of the data to be collected is multiplied by the target mode coefficient to obtain the adjusted equivalent value, so that data collection can be performed on the data to be collected based on the adjusted equivalent value.

[0058] The target performance mode refers to the current performance state mode of the storage device. Different performance modes represent the operating performance of the storage device under different conditions, such as high-speed mode, normal mode, and energy-saving mode. The target mode coefficient refers to the coefficient corresponding to the target performance mode of the storage device. It is used to quantify the performance differences under different performance modes and adjust the equivalent value of data collection. Different performance modes correspond to different mode coefficients. For example, the mode coefficient corresponding to the high-speed mode may be 1.2, the mode coefficient corresponding to the normal mode is 1, and the mode coefficient corresponding to the energy-saving mode is 0.8. The adjusted equivalent value refers to the value obtained by multiplying the original equivalent value of the data to be collected by the target mode coefficient. It reflects the number of thread resources actually required to collect the data under the current performance mode of the storage device.

[0059] The system has the function of detecting the performance status of storage devices. By running a performance detection program, it analyzes real-time operating data of the storage devices, such as CPU usage, memory usage, and data read / write speeds, to determine the current performance mode of the storage device. For example, if a storage device has high CPU usage and fast data read / write speeds, the system determines that it is in high-speed mode. Based on a pre-set mapping table between performance modes and mode coefficients, the system finds the target mode coefficient corresponding to the current target performance mode. Assuming that the mapping table specifies a mode coefficient of 1.2 for high-speed mode, 1 for normal mode, and 0.8 for energy-saving mode, the target mode coefficient obtained for a storage device in high-speed mode is 1.2. The system then obtains the raw equivalent value of the data to be collected. This raw equivalent value is calculated based on the data's attributes (such as data volume, data type, and business metrics) and does not consider storage device performance factors. For example, if the raw equivalent value of the data to be collected is 5, the system multiplies the raw equivalent value by the obtained target mode coefficient (5 × 1.2 = 6), resulting in an adjusted equivalent value of 6. The system decides whether to collect the data to be collected based on the adjusted equivalent value, and uses the adjusted equivalent value as an important reference when scheduling data collection tasks and allocating threads to reasonably arrange data collection tasks and allocate thread resources.

[0060] Considering the current performance mode of the storage device, the equivalent value is adjusted through the target mode coefficient, so that the system can more accurately allocate thread resources for data collection tasks, avoiding uneven or unreasonable resource allocation due to differences in storage device performance.

[0061] As a refinement of the above embodiment, when executing to obtain the target mode coefficient of the target performance mode of the storage device where the data to be collected is located, it can be implemented by, but not limited to, the following methods, including: obtaining the target health of the storage device where the data to be collected is located; and searching for the target mode coefficient corresponding to the target health based on a pre-established mapping relationship between health and performance mode.

[0062] The target health is a quantitative indicator reflecting the current health of a storage device. It is typically derived through a comprehensive assessment of the storage device's hardware status (such as disk wear and memory error rates) and system performance indicators (such as response latency and error code frequency) using specialized tools. A higher value indicates better storage device health. A mapping relationship is a pre-established rule that associates different levels of storage device health with corresponding performance modes. This allows for quick identification of the performance mode that a storage device should match under specific health conditions. The target mode coefficient corresponds to the storage device's target performance mode and is used to adjust the equivalent value of the collected data based on the storage device's performance mode.

[0063] The system uses a professional storage device health assessment tool to regularly monitor the hardware status and system performance indicators of storage devices. Hardware status includes the number of bad blocks on the disk and the memory error rate; system performance indicators include average response time and data transfer rate. The assessment tool comprehensively calculates these indicators based on pre-set weights and calculation rules to determine the target health of the storage device. For example, if the weights are set to 30% for disk wear, 20% for memory error rate, 30% for response latency, and 20% for data transfer rate, the target health of the storage device is 75 after normalization and weighted summation. Establishing a health-performance mode mapping: The system pre-establishes a mapping between health and performance mode based on historical storage device data and performance test results. For example, a health greater than 80 corresponds to high-speed mode; a health between 60 and 80 corresponds to normal mode; and a health below 60 corresponds to energy-saving mode. After obtaining the target health of the storage device, the corresponding performance mode is found based on this mapping to determine the target mode coefficient. If the target health is 75, the mapping corresponds to normal mode, and the target mode coefficient is 1. The target mode coefficients obtained are used to adjust the equivalent value of the data to be collected. The new equivalent value is used to guide the thread allocation and scheduling of data collection tasks. For example, if the original equivalent value of data is 5, the adjusted equivalent value in normal mode will still be 5 × 1 = 5. The system will use this value to appropriately schedule data collection tasks.

[0064] Through target health assessment and mapping relationships, the target mode coefficient is accurately obtained, and then the equivalent value is reasonably adjusted, so that the system can accurately allocate resources and schedule tasks according to the actual performance status of the storage device.

[0065] In order to better understand the entire process of data processing, Figure 2 As shown, Figure 2A flow chart of the entire data processing process provided in an embodiment of the present application, task hierarchical quantification module: 1. Definition of acquisition resources (i.e., data to be acquired): Storage device: Storage device is regarded as the top-level resource. For multi-device management software, each set of storage devices added is a storage resource. Resource type: The lower-level resources of the storage device correspond to specific resources stored in the three business scenarios of blocks, files, and objects, such as basic clusters, storage pools, disks, ports, volumes in block scenarios, file systems and shared directories in file scenarios, buckets in object scenarios, etc. Business indicators: Specific indicators in different business scenarios, such as capacity, performance, alarms, etc.

[0066] Based on the above classification, a "collection resource" is now defined as a single collection resource identifier, formed by combining the storage device, resource type, and business resource. For example, Storage A-Volume-Performance, Storage A-Cluster-Alarm, and Storage B-Volume-Performance are all independent collection resources. The term "collection resource" mentioned below refers to this split definition.

[0067] 2. Fixed tiering benchmark: Based on actual business scenarios, fixed equivalents are defined for resource types and business indicators within the collection resources. Resource type weight (RW) indicates the difference in collection complexity for different resource tiers (such as clusters, storage pools, volumes, and file systems). For example, RW for Volume is 4.0, RW for Pool is 1.0, and RW for FileSystem is 3.0. Business indicator weight (IW) indicates the difference in collection time for different indicators (such as performance and capacity). For example, IW for performance indicators is 3.0, and IW for capacity indicators is 1.0.

[0068] 3. Dynamic Tier Benchmark: In actual collection, due to the varying scale of different resources, the corresponding collection duration can vary significantly. For example, if Storage A has 1,000 volumes and Storage B has 30 volumes, the corresponding collection costs for both volume-performance tasks will differ significantly. Therefore, a dynamic tiering strategy is introduced in addition to the fixed tiering benchmark for collection cost calculation. This strategy's calculation logic is as follows: Historical duration recording: Using a sliding window algorithm, the system independently records the historical average duration (in minutes) for each collection resource. For example: Collection resource 1 (Storage A - Volumes - Performance): Average duration 10 minutes. Collection resource 2 (Storage A - Storage Pool - Performance): Average duration 2 minutes. Duration normalization: The median of all historical durations of all collection tasks in the current system is used as the baseline duration. The standardized duration coefficient for the collection resource is calculated: CT = Average duration / Baseline duration. Assuming the base duration is 2, the CT for collection resource 1 is 5, and the CT for collection resource 2 is 1. Equivalent value calculation: The final thread equivalent value for each collection resource is: Thread equivalent = CT × (RW + IW). Based on the above, the thread equivalent value for collection resource 1 is 5 × (4 + 3) = 35 equivalents, while that for collection resource 2 is 1 × (1 + 3) = 4 equivalents.

[0069] Intelligent combination scheduling module: such as Figure 2 As shown, when the acquisition task is started, the program will start processing according to the following logic: while the thread pool has idle resources: 1. Select the task with the largest equivalent value from the queue; 2. If the task equivalent is ≤ the number of remaining threads (that is, the number of available threads): add it to the current task package; the number of remaining threads after the task with the largest equivalent value is added to the current task package is equal to the number of remaining threads before the task with the largest equivalent value is added to the current task package minus the equivalent value of the task with the largest equivalent value; 3. Otherwise: skip the task and check the task with the second largest equivalent value. In actual execution, the following optimization logic processing is required: 1. Prioritize combining multiple tasks of the same storage device to reduce connection overhead; 2. If there is a task with a higher equivalent than the total number of threads, it is executed separately; the actual execution example is as follows: Now assume that the total thread resources are 80, and there are four collection resources, namely: collection resource A: 35; collection resource B: 28; collection resource C: 20; collection resource D: 15; the corresponding task combination process: select task A (35) → remaining 45; select task B (28) → remaining 17; select task D (15) → remaining 2 (insufficient to execute task C); total utilization: (35+28+15) / 80=97.5%.

[0070] Dynamic Policy Engine: Considering the importance and necessity of the normal operation of storage services, the concept of dynamic storage policies has been added to the above. For storage managed by multiple devices, configurable collection policies include high performance, standard performance, low performance, and dynamic floating. Configuration Description: High performance allows configuration based on standard performance, with a maximum over-provisioning ratio of 1.1 to 1.5 times the number of physical threads (default 1.2). Reduce the sampling interval to obtain more performance data. Standard performance uses the standard equivalent calculated normally without additional processing. Low performance reduces the thread ratio based on standard performance by 40% to 80% (default 60%), increases the sampling interval, and reduces the amount of performance data collected. Dynamic floating links to the system health function and dynamically adjusts based on preset conditions and the linked system health function. This application does not provide further explanation of the storage health function. The storage health branch ranges from 0 to 100 and is affected by multiple dimensions such as storage alarms, performance indicators, and hardware status. It is a visual representation of the storage operation status. Floating adjustment rules: Switching trigger conditions (using storage A as an example): High performance, when the system is in the default window period (e.g., 1:00 AM to 5:00 AM). Manually triggering emergency data collection (e.g., for fault diagnosis). Standard performance: The current mode is high performance and the high performance window period has been exited. The health level has returned to the healthy range (>= 75). Low performance: When the storage health level falls below the healthy range (<75), the storage device triggers an overload alarm. 3. The relationship between dynamic policies and the scheduling module: The equivalent of the data collection resource is reprocessed according to the corresponding storage policy, and only then is the actual equivalent value of the data collection resource determined. The actual execution process is as follows: The mode controller determines the mode based on the device health (e.g., storage A → low performance mode); adjusts the equivalent base: Corrected equivalent = original equivalent × mode coefficient. Mode coefficient examples: High performance mode: 1.2, Standard mode: 1.0, Low power mode: 0.6. The dynamic packager uses the corrected equivalent to sort and package tasks. For example, the original equivalent of the acquisition resource 1 mentioned above is 35. At this time, it is stored in low performance mode with a coefficient of 0.6, and the final corrected equivalent of the acquisition resource 1 is 21.

[0071] This application dynamically analyzes the actual resource consumption characteristics of different collection tasks and intelligently adjusts the thread allocation strategy to effectively solve the core pain points of low resource utilization and uneven collection efficiency in traditional storage monitoring systems. Compared with the fixed polling mechanism, this solution can automatically identify high-load tasks and allocate resources in priority, reducing thread idleness and task queuing. At the same time, it actively limits the resource usage of non-critical tasks in high-concurrency scenarios to avoid interference with the performance of the storage device itself. By intelligently combining collection tasks of different magnitudes, the system can fully tap the potential of the hardware and significantly improve processing throughput while ensuring data integrity. The adaptive mode control mechanism enables the system to flexibly respond to complex scenarios such as business peaks and equipment failures. It can not only quickly complete key data collection in an emergency, but also actively downgrade when the equipment load is high to ensure business continuity.

[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0073] The embodiment of the present application also provides a data processing device, Figure 3 A schematic diagram of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 3 Shown, including:

[0074] An acquiring unit 21 is configured to acquire equivalent values ​​of at least two data to be collected from at least two storage devices, and to acquire the number of available threads of the at least two storage devices; the equivalent value is the number of threads required to collect the data to be collected;

[0075] A selection unit 22 is configured to select at least one target data to be collected from the at least two data to be collected based on equivalent values ​​of the at least two data to be collected and in a manner that makes the number of available threads approach a minimum value; the number of available threads is negatively correlated with the sum of the equivalent values ​​of the at least one selected target data to be collected;

[0076] The collection unit 23 is configured to collect data of at least one target to-be-collected data by utilizing the available number of threads.

[0077] Through the present application, by obtaining equivalent values ​​of at least two data to be collected from at least two storage devices and obtaining the number of available threads of the at least two storage devices; the equivalent value is the number of threads required to collect the data to be collected; based on the equivalent values ​​of the at least two data to be collected, at least one target data to be collected is selected from the at least two data to be collected in a manner that makes the number of available threads approach a minimum value; the number of available threads is negatively correlated with the sum of the equivalent values ​​of the at least one selected target data to be collected; and data collection is performed on the at least one target data to be collected using the number of available threads, thereby reducing thread idleness and thereby improving thread utilization. Therefore, the technical problem of low thread utilization can be solved, achieving the technical effect of improving thread utilization.

[0078] Furthermore, in a possible implementation of this embodiment, as Figure 4 As shown, the acquisition unit 21 includes:

[0079] An acquisition module 211 is configured to acquire time consumption coefficients of at least two pieces of data to be collected, and acquire data type weights and business indicator weights of at least two pieces of data to be collected;

[0080] The calculation module 212 is used to calculate the equivalent value according to the time consumption coefficient, the data type weight and the business indicator weight.

[0081] Furthermore, in a possible implementation of this embodiment, as Figure 4 As shown, the calculation module 212 is further configured to:

[0082] The data type weight and the business indicator weight are added together to obtain the sum result;

[0083] The product of the summation result and the time consumption coefficient is calculated to obtain the equivalent value.

[0084] Furthermore, in a possible implementation of this embodiment, as Figure 4 As shown, the acquisition module 211 is further used for:

[0085] Obtaining the historical average collection time of at least two data to be collected, and obtaining the benchmark collection time of at least two data to be collected; the benchmark collection time is the median of all historical collection times of at least two data to be collected;

[0086] The time coefficient is obtained by dividing the historical average collection time by the benchmark collection time.

[0087] Furthermore, in a possible implementation of this embodiment, as Figure 4 As shown, the selection unit 22 is further used for:

[0088] determining whether there is at least one target equivalent value among the equivalent values; the target equivalent value being an equivalent value that is less than or equal to the number of available threads;

[0089] When there is at least one target equivalent value among the equivalent values, target data to be collected is selected from the data to be collected corresponding to the at least one target equivalent value in descending order of the target equivalent values; wherein the total target equivalent values ​​of the selected target data to be collected does not exceed the number of available threads;

[0090] Target data to be collected is selected from the remaining data to be collected in a trend that makes the number of available threads approach a minimum value.

[0091] Furthermore, in a possible implementation of this embodiment, as Figure 4 As shown, the device also includes:

[0092] The acquisition unit 21 is further configured to acquire a target mode coefficient of a target performance mode of the storage device where the data to be collected is located; different performance modes correspond to different mode coefficients;

[0093] The calculation unit 24 is used to perform product calculation on the equivalent value of the data to be collected and the target mode coefficient to obtain an adjusted equivalent value, so as to collect the data to be collected based on the adjusted equivalent value.

[0094] Furthermore, in a possible implementation of this embodiment, as Figure 4 As shown, the acquisition unit 21 is further used for:

[0095] Obtain the target health of the storage device where the data to be collected is located;

[0096] According to the pre-established mapping relationship between health and performance mode, the target mode coefficient corresponding to the target health is found.

[0097] For the description of the features in the embodiments corresponding to the data processing device, reference can be made to the relevant description of the embodiments corresponding to the data processing method, which will not be repeated here.

[0098] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data processing method embodiments.

[0099] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data processing method embodiments when run.

[0100] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0101] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing method embodiments are implemented.

[0102] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned data processing method embodiments are implemented.

[0103] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The above is a detailed introduction to a data processing method and device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining equivalent values ​​of at least two pieces of data to be collected in at least two storage devices, and obtaining the number of available threads of the at least two storage devices; The equivalent value is the number of threads required to collect the data to be collected; Based on equivalent values ​​of at least two pieces of data to be collected, selecting at least one target piece of data to be collected from the at least two pieces of data to be collected in a manner that makes the number of available threads approach a minimum value; The number of available threads is negatively correlated with the sum of equivalent values ​​of at least one selected target data to be collected; Using the available number of threads, collecting data for the at least one target data to be collected; The acquiring of equivalent values ​​of at least two pieces of data to be collected in at least two storage devices includes: Obtaining time consumption coefficients of the at least two data to be collected, and obtaining data type weights and business indicator weights of the at least two data to be collected; Calculating the equivalent value according to the time consumption coefficient, the data type weight, and the business indicator weight; The selecting at least one target data to be collected from the at least two data to be collected based on equivalent values ​​of the at least two data to be collected and in accordance with a trend of making the number of available threads approach a minimum value includes: determining whether there is at least one target equivalent value among the equivalent values; the target equivalent value being an equivalent value less than or equal to the number of available threads; When the at least one target equivalent value exists in the equivalent values, selecting the target data to be collected from the data to be collected corresponding to the at least one target equivalent value in descending order of the target equivalent values; wherein the total target equivalent values ​​of the selected target data to be collected does not exceed the number of available threads; The target data to be collected is selected from the remaining data to be collected according to a trend of making the number of available threads approach a minimum value.

2. The data processing method according to claim 1, characterized in that: Calculating the equivalent value according to the time consumption coefficient, the data type weight, and the business indicator weight includes: Adding the data type weight and the business indicator weight to obtain a sum result; The product of the summation result and the time consumption coefficient is calculated to obtain the equivalent value.

3. The data processing method according to claim 1, characterized in that: The time consumption coefficient for obtaining the at least two pieces of data to be collected includes: Obtaining a historical average collection time of the at least two data to be collected, and obtaining a benchmark collection time of the at least two data to be collected; the benchmark collection time is the median of all historical collection times of the at least two data to be collected; The time consumption coefficient is obtained by calculating the quotient of the historical average collection time consumption and the benchmark collection time consumption.

4. The data processing method according to claim 1, characterized in that: The method further comprises: Obtaining a target mode coefficient of a target performance mode of the storage device where the data to be collected is located; different performance modes correspond to different mode coefficients; The product of the equivalent value of the data to be collected and the target mode coefficient is calculated to obtain an adjusted equivalent value, so as to collect data for the data to be collected based on the adjusted equivalent value.

5. The data processing method according to claim 4, characterized in that: The step of obtaining a target mode coefficient of a target performance mode of the storage device where the data to be collected is located includes: Obtaining a target health of a storage device where the data to be collected is located; According to a pre-established mapping relationship between health and performance mode, the target mode coefficient corresponding to the target health is searched.

6. A data processing device, characterized in that: include: an acquiring unit, configured to acquire equivalent values ​​of at least two pieces of data to be collected from at least two storage devices, and to acquire the number of available threads of the at least two storage devices; The equivalent value is the number of threads required to collect the data to be collected; a selection unit configured to select at least one target data to be collected from the at least two data to be collected based on equivalent values ​​of the at least two data to be collected and in accordance with a trend of making the number of available threads approach a minimum value; The number of available threads is negatively correlated with the sum of equivalent values ​​of at least one selected target data to be collected; a collection unit, configured to collect data of the at least one target data to be collected by utilizing the available number of threads; The acquisition unit is further configured to: Obtaining time consumption coefficients of the at least two data to be collected, and obtaining data type weights and business indicator weights of the at least two data to be collected; Calculating the equivalent value according to the time consumption coefficient, the data type weight, and the business indicator weight; Wherein, the selection unit is further used for: determining whether there is at least one target equivalent value among the equivalent values; the target equivalent value being an equivalent value less than or equal to the number of available threads; When the at least one target equivalent value exists in the equivalent values, selecting the target data to be collected from the data to be collected corresponding to the at least one target equivalent value in descending order of the target equivalent values; wherein the total target equivalent values ​​of the selected target data to be collected does not exceed the number of available threads; The target data to be collected is selected from the remaining data to be collected according to a trend of making the number of available threads approach a minimum value.

7. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the data processing method according to any one of claims 1 to 5 when executed by a processor.

Images