Data processing method, device, equipment and storage medium

Abstract

The present application provides a data processing method, apparatus, device, and storage medium that can be applied to the fields of computer technology and storage technology. The data processing method includes: in response to receiving a target processing task for data to be processed, obtaining first load information of a central processing unit and second load information of a hard disk processor in a target hard disk; based on the first load information and the second load information, determining a target processor for executing the target processing task from the central processing unit and the hard disk processor; if the target processor is a hard disk processor, sending the data to be processed and a target instruction to the target hard disk, so that the hard disk processor executes the target processing task on the data to be processed in response to the target instruction, obtains the target data, and stores the target data.

Description

Technical Field

[0001] The present application relates to the fields of computer technology and storage technology, and more specifically to a data processing method, apparatus, device, and storage medium. Background Art

[0002] With the development of big data technology, the demand for data processing efficiency has also increased. Traditional data processing methods usually rely on the central processing unit to perform data processing tasks, resulting in frequent data transfer between the hard disk and memory, and low data processing efficiency.

[0003] To improve data processing efficiency, related technologies typically incorporate a hard drive processor with computing power within the hard drive. This allows the hard drive processor to perform data processing tasks and reduce data transmission. However, using only the hard drive processor for data processing fails to effectively utilize the system's computing resources, limiting improvements in overall system performance and efficiency. Summary of the Invention

[0004] In view of the above problems, the present application provides a data processing method, apparatus, device, medium and program product.

[0005] According to the first aspect of the present application, a data processing method is provided, comprising: in response to receiving a target processing task for data to be processed, obtaining first load information of a central processing unit and second load information of a hard disk processor in a target hard disk; based on the above-mentioned first load information and the above-mentioned second load information, determining a target processor for executing the above-mentioned target processing task from the above-mentioned central processing unit and the above-mentioned hard disk processor; in a case where the above-mentioned target processor is the above-mentioned hard disk processor, sending the above-mentioned data to be processed and target instructions to the above-mentioned target hard disk, so that the above-mentioned hard disk processor executes the above-mentioned target processing task on the above-mentioned data to be processed in response to the above-mentioned target instructions, obtains target data, and stores the above-mentioned target data.

[0006] The second aspect of the present application provides a data processing device, including: an acquisition module, used to obtain first load information of a central processing unit and second load information of a hard disk processor in a target hard disk in response to receiving a target processing task for data to be processed; a determination module, used to determine a target processor for executing the target processing task from the central processing unit and the hard disk processor based on the first load information and the second load information; a sending module, used to send the data to be processed and the target instruction to the target hard disk when the target processor is the hard disk processor, so that the hard disk processor executes the target processing task on the data to be processed in response to the target instruction, obtains the target data, and stores the target data.

[0007] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0008] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0009] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0010] According to an embodiment of the present application, when executing data processing tasks for stored data, the target processor is determined based on the first load information of the central processing unit and the load information of the hard disk processor. The actual load conditions of the central processing unit and the hard disk processor can be taken into consideration, and tasks can be dynamically allocated, so that the computing resources in the system can be fully utilized when executing the target processing tasks. Not only can the hard disk processor be used to share the data processing tasks that originally required the central processing unit to execute, but unnecessary data transmission can also be effectively reduced, thereby improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings.

[0012] Figure 1 An application scenario diagram of the data processing method, apparatus, device, medium, and program product according to an embodiment of the present application is shown.

[0013] Figure 2 A flow chart of a data processing method according to an embodiment of the present application is shown.

[0014] Figure 3 A flowchart of storing data according to a specific embodiment of the present application is shown.

[0015] Figure 4 A flowchart of reading data according to a specific embodiment of the present application is shown.

[0016] Figure 5 A flowchart of performing optimization operations according to an embodiment of the present application is shown.

[0017] Figure 6 The figure shows a structural block diagram of a data processing device according to an embodiment of the present application.

[0018] Figure 7A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "comprise," "include," etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0021] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0022] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] A mechanical hard disk drive (HDD) is a hard disk storage device and the most common type of standard hard disk. An HDD uses a rotating hard disk and a read / write head to store and access data. It consists of a hard disk, a read / write head, a motor, and control circuitry. The hard disk is typically a circular disk made of aluminum alloy or glass, coated with a magnetic material. The read / write head reads and writes data to the hard disk. As the motor rotates the hard disk, the read / write head reads and writes data at different locations on the hard disk. The read / write head uses a drive arm to adjust the position of different magnetic tracks. The drive rotates the hard disk, and the read / write head reads data.

[0024] A solid-state drive (SSD) is a high-speed, low-power, silent, and highly reliable storage device that uses flash memory chips to store data. Compared to traditional mechanical hard drives, SSDs contain no moving parts, resulting in faster read and write speeds, a smaller size, lower noise levels, lower energy consumption, and higher reliability. SSDs are suitable for applications requiring high-speed data storage and access, such as gaming, high-definition video processing, databases, and servers. SSDs have a built-in controller and storage array. Through the main controller, data is directly digitally entered into and retrieved from the storage matrix. Using flash memory as the storage medium, SSDs eliminate the need for mechanical components such as motors, platters, and head arms. This reduces the time required for mechanical movement, reduces seek time to almost zero, and allows for high continuous write speeds.

[0025] With the booming development of cloud computing and big data, stronger computing power and faster storage capacity are needed, and the requirements for server storage technology are getting higher and higher. In technical fields such as short videos, shopping websites or artificial intelligence, tens of thousands of users are accessing different content, videos in different fields, and different items for various reasoning calculations at all times. Multiple services running together require access to data on the hard disk, which places high demands on the storage and performance of the hard disk. The traditional model copes with the access and storage of such large amounts of data by continuously stacking the number of hard disks to increase storage space, or replacing high-performance hard disks to improve storage performance. This not only causes a continuous exponential increase in hardware costs, but also makes the storage system larger and more complex, while its stability and reliability are constantly declining. Faced with this high-concurrency, highly complex and low-stability storage system, it is necessary to make system-level improvements to this large and complex system and make intelligent improvements to make the hard disk system more stable and intelligent while being complex.

[0026] Traditional hard drives serve merely as data storage tanks, with a single function that is unable to perform data compression, encryption, and decryption. This means that these tasks require the central processing unit (CPU) to execute before transferring the huge amount of data to the hard drive through the hard drive interface. This has two disadvantages: first, it consumes CPU resources, requiring the CPU to perform routine tasks, which increases the CPU burden; second, the CPU encryption and decryption generates a large amount of data, which needs to be cached in memory and hard drive first. After encryption and decryption are completed, the final data can be cached to the hard drive, resulting in repeated access to the hard drive, which not only occupies bus resources inefficiently, but also has long links and is prone to errors.

[0027] An embodiment of the present application provides a data processing method, comprising: in response to receiving a target processing task for data to be processed, obtaining first load information of a central processing unit and second load information of a hard disk processor in a target hard disk; based on the first load information and the second load information, determining a target processor for executing the target processing task from the central processing unit and the hard disk processor; in the case where the target processor is a hard disk processor, sending the data to be processed and a target instruction to the target hard disk so that the hard disk processor executes the target processing task on the data to be processed in response to the target instruction, obtains the target data, and stores the target data. The embodiment of the present application improves data processing efficiency by considering the actual load conditions of the central processing unit and the hard disk processor, utilizing the hard disk processor to share the data processing tasks that originally needed to be executed by the central processing unit, without the need for additional data transmission.

[0028] Figure 1 An application scenario diagram of the data processing method, apparatus, device, medium, and program product according to an embodiment of the present application is shown.

[0029] like Figure 1 As shown, the application scenario according to this embodiment may include a server 110 , which may include a central processing unit 111 and a hard disk 112 , and the hard disk 112 may include a hard disk processor 1121 , a data storage area 1122 and a read-only storage area 1123 .

[0030] Server 110 may be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices (for example only). The backend management server may analyze and process received data such as user requests, and feed back the processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal device.

[0031] The central processing unit 111 can be used to perform data processing tasks. The hard disk 112 can be used to process and store data. For example, the hard disk 112 can store data sent by the central processing unit 111 in the data storage area 1122, or can use the hard disk processor 1121 to respond to instructions sent by the central processing unit 111 to perform data processing tasks.

[0032] In one specific embodiment, one end of the hard disk processor 1121 is connected to the input / output interface of the hard disk 112, and the other end can be connected to the data storage area 1122 via eight cables. The input / output interface includes A+ Transmit (data transmission positive signal interface), A- Transmit (data transmission negative signal interface), B- Receive (data reception negative signal interface), and B+ Receive (data reception positive signal interface). Because the hard disk processor 1121 is directly connected to the data storage area 1122 via eight cables, the addressing bandwidth is doubled, and data read addressing can be performed with double the bandwidth, greatly improving data reading efficiency.

[0033] The read-only storage area 1123 can be used to store file index information of files stored in the data storage area 1122, such as the boot area, file allocation table, root directory and data area information, so that when an error occurs in the data storage area 1122, the impact on the file index information can be reduced, thereby enabling intelligent rescue and repair of files.

[0034] It should be noted that the data processing method provided in the embodiment of the present application can generally be executed by the central processing unit 111. Accordingly, the data processing device provided in the embodiment of the present application can generally be set in the central processing unit 111.

[0035] For example, in response to receiving a target processing task for the data to be processed, the central processing unit 111 obtains first load information of the central processing unit 111 and second load information of the hard disk processor 1121 in the target hard disk 112; based on the first load information and the second load information, the target processor for executing the target processing task is determined from the central processing unit 111 and the hard disk processor 1121; when the target processor is the hard disk processor 1121, the data to be processed and the target instruction are sent to the target hard disk 112, so that the hard disk processor 1121 executes the target processing task on the data to be processed in response to the target instruction, obtains the target data, and stores the target data in the storage area 1122.

[0036] It should be understood that Figure 1 The numbers of servers, central processing units, hard disks, and hard disk processors are merely illustrative. Any number of servers, central processing units, hard disks, and hard disk processors may be used as needed.

[0037] The following will be based on Figure 1 The scene described by Figures 2 to 5 The data processing method of the application embodiment is described in detail.

[0038] Figure 2 A flow chart of a data processing method according to an embodiment of the present application is shown.

[0039] like Figure 2 As shown, the data of this embodiment includes operations S210 to S230.

[0040] In operation S210 , in response to receiving a target processing task for data to be processed, first load information of a central processing unit and second load information of a hard disk processor in a target hard disk are acquired.

[0041] In operation S220 , a target processor for executing the target processing task is determined from among the central processor and the hard disk processor based on the first load information and the second load information.

[0042] In operation S230, when the target processor is a hard disk processor, the data to be processed and the target instruction are sent to the target hard disk, so that the hard disk processor performs the target processing task on the data to be processed in response to the target instruction, obtains the target data, and stores the target data.

[0043] In an embodiment of the present application, since the target hard disk is configured with a hard disk processor that can execute target processing tasks, for the data to be processed and stored in the target hard disk, the central processing unit and the hard disk processor can be dynamically assigned tasks based on their respective load information to fully utilize the system's computing resources and improve data processing efficiency.

[0044] Target processing tasks may include virtual machine related tasks, storage acceleration tasks, online or offline encryption and decryption of data tasks, data deep inspection tasks, and data compression tasks.

[0045] In some embodiments, the hard disk processor can be pre-configured based on the execution logic of the target processing task to enable it to execute the target processing task. The hard disk processor can be, for example, a field programmable gate array (FPGA) or a microcontroller unit (MCU).

[0046] When acquiring the second load information, the central processing unit may send a load information acquisition instruction to the hard disk processor to interact with the hard disk processor and acquire the second load information returned by the hard disk processor.

[0047] Since the first load information and the second load information can characterize the load status of the central processing unit and the hard disk processor, and thus can characterize whether the central processing unit and the hard disk processor can execute the target processing task in a timely manner, the processor that can execute the target processing task in a timely manner can be determined as the target processor based on the first load information and the second load information.

[0048] In some embodiments, the first load information and the second load information may be compared to determine the target processor, or the first load information and the second load information may be compared to a threshold value to determine the target processor. For example, if the first load information is greater than the threshold value and the second load information is less than the threshold value, the hard disk processor is determined to be the target processor.

[0049] When it is determined that the target processor is a hard disk processor, the central processing unit can send the data to be processed and the target instruction to the hard disk processor so that the hard disk processor can execute the target processing task and store the obtained target data.

[0050] When the target processor is determined to be a central processing unit, the central processing unit can directly perform a target processing task on the data to be processed, obtain target data, and send the target data to the hard disk processor so that the target hard disk can store the target data.

[0051] According to an embodiment of the present application, when executing data processing tasks for stored data, the target processor is determined based on the first load information of the central processing unit and the load information of the hard disk processor. The actual load conditions of the central processing unit and the hard disk processor can be taken into consideration, and tasks can be dynamically allocated, so that the computing resources in the system can be fully utilized when executing the target processing tasks. Not only can the hard disk processor be used to share the data processing tasks that originally required the central processing unit to execute, but unnecessary data transmission can also be effectively reduced, thereby improving data processing efficiency.

[0052] According to an embodiment of the present application, based on the first load information and the second load information, a target processor for executing the target processing task is determined from the central processing unit and the hard disk processor, including: comparing the first load information with the first load threshold to obtain a first comparison result characterizing the load state of the central processing unit; comparing the second load information with the second load threshold to obtain a second comparison result characterizing the load state of the hard disk processor; and determining the target processor from the central processing unit and the hard disk processor based on the first comparison result and the second comparison result.

[0053] Since the CPU and the hard disk processor have different configurations, the load thresholds to be set should also be different. In some embodiments, the first load threshold and the second load threshold can be determined based on the configuration information of the CPU and the hard disk processor or the load status when executing historical tasks.

[0054] When determining the load status of the central processing unit and the hard disk processor, the load information can be compared with the load threshold to obtain a first comparison result and a second comparison result. For example, when the load information is less than the load threshold, it can be indicated that the load of the processor is low, and vice versa, it can be indicated that the load of the processor is high.

[0055] After obtaining the first comparison result and the second comparison result, the target processor can be determined based on the first comparison result and the second comparison result. For example, the processor with the lower load between the central processing unit and the hard disk processor can be determined as the target processor.

[0056] According to an embodiment of the present application, by comparing the load information of the central processing unit and the hard disk processor with their respective load thresholds, the load status of the central processing unit and the hard disk processor can be determined, thereby making the determined target processor more accurate.

[0057] According to an embodiment of the present application, based on the first comparison result and the second comparison result, a target processor is determined from the central processing unit and the hard disk processor, including: when the first comparison result indicates that the first load information is less than the first load threshold, and the second comparison result indicates that the second load information is greater than the second load threshold, determining the central processing unit as the target processor; when the first comparison result indicates that the first load information is less than the first load threshold, and the second comparison result indicates that the second load information is less than the second load threshold, determining the target processor from the central processing unit and the hard disk processor according to the target task type of the target processing task; when the first comparison result indicates that the first load information is greater than the first load threshold, and the second comparison result indicates that the second load information is greater than the second load threshold, determining the target processor from the central processing unit and the hard disk processor according to the target task type and the data volume of the data to be processed.

[0058] When the first load information is less than the first load threshold and the second load information is greater than the second load threshold, it means that the load of the central processing unit is low and the load of the hard disk processor is high. At this time, the central processing unit can be determined as the target processor to use the central processing unit to execute the target processing task.

[0059] When the first load information is greater than the first load threshold and the second load information is less than the second load threshold, it means that the load of the hard disk processor is low and the load of the central processing unit is high. At this time, the hard disk processor can be determined as the target processor to use the hard disk processor to perform the target processing task.

[0060] When the first load information is less than the first load threshold and the second load information is less than the second load threshold, it means that the loads of the central processing unit and the hard disk processor are both low. Further judgment can be made based on the target task type, and a processor suitable for processing the target task type can be selected as the target processor.

[0061] When the first load information is greater than the first load threshold and the second load information is greater than the second load threshold, it means that the loads of the central processing unit and the hard disk processor are both high. Further judgment can be made based on the target task type and the amount of data to be processed to determine that the processor that consumes less resources or has higher processing efficiency when executing the target processing task is the target processor.

[0062] According to an embodiment of the present application, by considering various situations of the first comparison result and the second comparison result, the target processor is further selected according to parameters such as the target task type and data volume, so that the determined target processor is more suitable for processing the target processing task, further improving the processing efficiency.

[0063] Figure 3 A flowchart of storing data according to a specific embodiment of the present application is shown.

[0064] like Figure 3 As shown, storing data includes operations S310 to S340.

[0065] In operation S310, the target processor is determined. The central processing unit may determine that the target processor for executing the data encryption task is the hard disk processor based on the first comparison result and the second comparison result.

[0066] In operation S320, a target instruction and the data to be processed are sent. The target instruction can instruct the target hard disk to use the hard disk processor to perform a data encryption task on the data to be processed, obtain the target data, and store the target data.

[0067] In operation S330 , the hard disk processor performs a data encryption task on the data to be processed to obtain target data.

[0068] In operation S340, the target data is stored.

[0069] According to an embodiment of the present application, a target processor is determined from a central processing unit and a hard disk processor according to a target task type of a target processing task, including: when the target task type is a data compression type, determining the central processing unit as the target processor.

[0070] When both the central processing unit and the hard disk processor are in a low-load state and the target task type is a data compression type, since executing the data compression task will reduce the amount of data, the amount of target data is smaller than the amount of data to be processed. The central processing unit can be determined as the target processor, so that the central processing unit can be used to execute the data compression task, thereby only needing to transmit the compressed target data to the target hard disk. Compared with directly transmitting the data to be processed to the target hard disk, the transmission efficiency can be improved.

[0071] Correspondingly, when the central processing unit reads uncompressed data in the target hard disk, the hard disk processor may also compress the uncompressed data and then send it to the central processing unit to improve transmission efficiency.

[0072] In other embodiments, when the first load information is less than a first preset threshold, the second load information is less than a second preset threshold, and the target task type is a data encryption type, since executing the data encryption task in the central processing unit requires frequent data transmission between the memory and the target hard disk, the hard disk processor can be determined as the target processor to avoid unnecessary data transmission, thereby improving transmission efficiency.

[0073] According to the embodiments of the present application, by giving priority to using the central processing unit to execute data compression type tasks, the central processing unit only needs to transmit the compressed target data to the target hard disk, thereby improving data transmission efficiency and further improving overall data processing efficiency.

[0074] Figure 4 A flowchart of reading data according to a specific embodiment of the present application is shown.

[0075] like Figure 4 As shown, reading data includes operations S410 to S440.

[0076] In operation S410, a data read instruction is sent. When the central processing unit needs to read data from a target hard disk, it can send a data read instruction to the target hard disk.

[0077] In operation S420, the hard disk processor is used to accelerate the search for data and perform data compression to obtain compressed data. After receiving the data read instruction, the target hard disk can use the hard disk processor to perform data compression to reduce the amount of data.

[0078] In operation S430, the compressed data is transmitted.

[0079] In operation S440, the compressed data is decompressed. After receiving the compressed data, the central processing unit can decompress the compressed data to obtain the required data.

[0080] According to an embodiment of the present application, the hard disk processor is also used to: determine the ratio between the data volume of the data to be processed and the remaining storage capacity of the target hard disk when the target task type does not include a data compression type; when the ratio is greater than a preset threshold, obtain initial target data after executing the target processing task on the data to be processed; perform data compression on the initial target data to obtain target data, and store the target data.

[0081] The hard drive processor automatically determines whether data compression is necessary based on the target hard drive's remaining storage capacity and the amount of data to be processed. If the amount of data to be processed is large, the ratio will also be large, and storing the data will require a large amount of storage space on the target hard drive. Therefore, compression can be used to reduce the storage space occupied.

[0082] According to the embodiment of the present application, by utilizing the hard disk processor to automatically compress the data to be processed, not only the storage space of the target hard disk can be saved, but also the consumption of the computing resources of the central processing unit can be reduced.

[0083] According to an embodiment of the present application, a target processor is determined from a central processing unit and a hard disk processor based on the target task type and the amount of data to be processed, including: determining a first computing resource consumption required for the central processing unit to execute the target processing task and a second computing resource consumption required for the hard disk processor to execute the target processing task based on the target task type and the amount of data; and determining a target processor from the central processing unit and the hard disk processor based on the first computing resource consumption and the second computing resource consumption.

[0084] Since different target task types, different data volumes, and different processor configurations may result in different computing resource consumptions, the first computing resource consumption may be determined based on the target task type, data volume, and configuration parameters of the central processing unit.

[0085] Correspondingly, the second computing resource consumption may also be determined based on the target task type, data volume, and configuration parameters of the hard disk processor.

[0086] When determining the target processor based on the first computing resource consumption and the second computing resource consumption, the processor with less computing resource consumption can be determined as the target processor. For example, when the first computing resource consumption is less than the second computing resource consumption, the central processing unit can be determined as the target processor.

[0087] In other embodiments, the first computing resource consumption may be compared with the computing resource amount of the central processing unit to obtain a third comparison result, and the second computing resource consumption may be compared with the computing resource amount of the hard disk processor to obtain a fourth comparison result. Based on the third and fourth comparison results, a processor having computing resources capable of supporting the target processing task is determined as the target processor.

[0088] According to an embodiment of the present application, the target processor is determined based on the computing resource consumption of the central processing unit and the hard disk processor in executing the target processing task, so that the computing resource consumption of executing the target processing task is reduced.

[0089] According to an embodiment of the present application, a target processor is determined from a central processing unit and a hard disk processor according to the target task type and the amount of data to be processed, including: determining the first computing resource sub-consumption required for the central processing unit to execute the target processing subtask and the second computing resource sub-consumption required for the hard disk processor to execute the target processing subtask according to the data amount and the target task sub-type of each target processing subtask; determining the target processor corresponding to each target processing subtask from the central processing unit and the hard disk processor according to the execution order of multiple target processing subtasks, the first computing resource sub-consumption and the second computing resource sub-consumption.

[0090] According to an embodiment of the present application, a target processing task may include multiple target processing subtasks. Accordingly, a target task type may also include target task subtypes corresponding to each of the multiple target processing subtasks.

[0091] In an embodiment of the present application, when a target processing task includes multiple target processing subtasks, the multiple target processing subtasks can be allocated, and the central processing unit can be used to execute some target processing subtasks, and the hard disk processor can be used to execute the remaining target processing subtasks.

[0092] Before allocating multiple target processing subtasks, the first computing resource sub-consumption of the central processing unit to execute the target processing subtask and the second computing resource sub-consumption of the hard disk processor to execute the target processing subtask can be determined first, so as to allocate multiple target processing subtasks based on the first computing resource sub-consumption and the second computing resource sub-consumption.

[0093] When determining the first computing resource sub-consumption, the data volume, target task sub-type, and CPU configuration parameters can be calculated according to a preset computing resource consumption formula to obtain the first computing resource sub-consumption. Similarly, the data volume, target task sub-type, and CPU configuration parameters can also be calculated according to a preset computing resource consumption formula to obtain the second computing resource sub-consumption.

[0094] Since the hard disk processor can only process the data to be processed after the central processor sends it to the target hard disk, when allocating target processing subtasks, it is necessary to consider the execution order of multiple target processing subtasks and determine the target processor for executing the target processing subtasks.

[0095] For example, it may be determined that a plurality of target processing subtasks with a higher execution order are executed by the central processing unit, and the other target processing subtasks are executed by the hard disk processor.

[0096] According to an embodiment of the present application, when a target processing task includes multiple target processing subtasks, the execution order of the multiple target processing subtasks is considered, and the multiple target processing subtasks are respectively assigned to the hard disk processor and the central processing unit for execution, thereby improving data processing efficiency.

[0097] According to an embodiment of the present application, the hard disk processor is also used to: obtain the interface status of the target hard disk; when the interface status indicates that the target hard disk has no instructions to be executed, perform optimization operations on the stored data in the target hard disk, and the optimization operations include at least one of data cleaning operations, error repair operations, and defragmentation operations.

[0098] Traditional hard drive optimization typically relies on external tools or operating system commands. However, this optimization method only works for simple hard drives and cannot scale well with large-scale array storage drives. Therefore, for large-scale array storage drives, the only option is to purchase specialized storage servers and rely on the manufacturer's storage scheduling system for optimization, which incurs additional costs and fees.

[0099] In an embodiment of the present application, the hard disk processor can automatically identify the status of the interface and optimize the data in the hard disk when the hard disk has no commands to be executed, thereby improving the performance and life of the hard disk.

[0100] The optimization operation may include at least one of a data cleanup operation, an error repair operation, and a defragmentation operation. The data cleanup operation may be used to clean up unnecessary files on the disk to free up disk space; the error repair operation may be used to check and repair disk errors and mask bad blocks on the hard disk; and the defragmentation operation may be used to reduce disk fragmentation and improve system performance and hard disk life.

[0101] During the optimization process, the hard disk processor can monitor the interface status in real time. If the target hard disk has pending instructions, the hard disk processor can terminate the current optimization operation and perform data storage, data query, and data processing in response to the pending instructions.

[0102] According to an embodiment of the present application, by utilizing a hard disk processor to optimize the target hard disk when the target hard disk is idle, the storage space of the target hard disk can be released and the life of the target hard disk can be increased.

[0103] Figure 5 A flowchart of performing optimization operations according to an embodiment of the present application is shown.

[0104] like Figure 5 As shown, performing the optimization operation includes operations S510 to S540.

[0105] In operation S510, an interface status of a target hard disk is acquired.

[0106] In operation S520, it is determined whether the interface status indicates that the target hard disk has no instructions to be executed. If the interface status indicates that the target hard disk has no instructions to be executed, operation S530 is performed, otherwise operation S540 is performed.

[0107] In operation S530, an optimization operation is performed.

[0108] In operation S540 , the instruction to be executed is processed.

[0109] Based on the above data processing method, this application also provides a data processing device. Figure 6 The device is described in detail.

[0110] Figure 6 The figure shows a structural block diagram of a data processing device according to an embodiment of the present application.

[0111] like Figure 6 As shown, the data processing device 600 of this embodiment includes an acquisition module 610 , a determination module 620 and a sending module 630 .

[0112] The acquisition module 610 is configured to, in response to receiving a target processing task for the data to be processed, acquire first load information of the central processing unit and second load information of the hard disk processor in the target hard disk. In one embodiment, the acquisition module 610 may be configured to execute operation S210 described above, which will not be further described herein.

[0113] The determination module 620 is used to determine the target processor for executing the target processing task from the central processor and the hard disk processor based on the first load information and the second load information. In one embodiment, the determination module 620 can be used to execute the operation S220 described above, which will not be repeated here.

[0114] The sending module 630 is configured to send the to-be-processed data and the target instruction to the target hard disk, if the target processor is a hard disk processor, so that the hard disk processor, in response to the target instruction, performs the target processing task on the to-be-processed data, obtains the target data, and stores the target data. In one embodiment, the sending module 630 can be configured to perform operation S230 described above, and will not be further described here.

[0115] According to an embodiment of the present application, the determination module 620 includes a first comparison submodule, a second comparison submodule, and a determination submodule.

[0116] The first comparison submodule is used to compare the first load information with a first load threshold to obtain a first comparison result representing the load state of the central processing unit.

[0117] The second comparison submodule is used to compare the second load information with the second load threshold to obtain a second comparison result representing the load state of the hard disk processor.

[0118] The determination submodule is used to determine a target processor from the central processing unit and the hard disk processor based on the first comparison result and the second comparison result.

[0119] According to an embodiment of the present application, the determination submodule includes a first determination unit, a second determination unit, and a third determination unit.

[0120] The first determining unit is configured to determine the central processor as the target processor if the first comparison result indicates that the first load information is less than a first load threshold and the second comparison result indicates that the second load information is greater than a second load threshold.

[0121] The second determination unit is used to determine the target processor from the central processing unit and the hard disk processor according to the target task type of the target processing task when the first comparison result indicates that the first load information is less than the first load threshold and the second comparison result indicates that the second load information is less than the second load threshold.

[0122] The third determination unit is used to determine the target processor from the central processing unit and the hard disk processor according to the target task type and the amount of data to be processed when the first comparison result indicates that the first load information is greater than the first load threshold and the second comparison result indicates that the second load information is greater than the second load threshold.

[0123] According to an embodiment of the present application, the second determining unit includes a first determining subunit.

[0124] The first determining subunit is configured to determine the central processing unit as the target processor when the target task type is a data compression type.

[0125] According to an embodiment of the present application, the third determining unit includes a consumption determining subunit and a second determining subunit.

[0126] The consumption determination subunit is used to determine the first computing resource consumption required for the central processing unit to execute the target processing task and the second computing resource consumption required for the hard disk processor to execute the target processing task according to the target task type and data volume.

[0127] The second determining subunit is configured to determine a target processor from the central processing unit and the hard disk processor based on the first computing resource consumption and the second computing resource consumption.

[0128] According to an embodiment of the present application, the target processing task includes multiple target processing subtasks, and the target task type includes target task subtypes corresponding to each of the multiple target processing subtasks; the third determination unit also includes a sub-consumption determination subunit and a third determination subunit.

[0129] The sub-consumption determination subunit is used to determine the first computing resource sub-consumption required for the central processing unit to execute the target processing subtask and the second computing resource sub-consumption required for the hard disk processor to execute the target processing subtask based on the data volume and the target task subtype of each target processing subtask.

[0130] The third determining subunit is used to determine a target processor corresponding to each target processing subtask from the central processing unit and the hard disk processor according to the execution order of the multiple target processing subtasks, the first computing resource sub-consumption and the second computing resource sub-consumption.

[0131] According to an embodiment of the present application, the hard disk processor is also used to: obtain the interface status of the target hard disk; when the interface status indicates that the target hard disk has no instructions to be executed, perform optimization operations on the stored data in the target hard disk, and the optimization operations include at least one of data cleaning operations, error repair operations, and defragmentation operations.

[0132] According to embodiments of the present application, any multiple modules among the acquisition module 610, determination module 620, and sending module 630 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present application, at least one of the acquisition module 610, determination module 620, and sending module 630 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of these. Alternatively, at least one of the acquisition module 610, determination module 620, and sending module 630 may be at least partially implemented as a computer program module that, when executed, performs the corresponding functionality.

[0133] Figure 7 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present application is shown.

[0134] like Figure 7As shown, an electronic device 700 according to an embodiment of the present application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0135] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in one or more memories.

[0136] According to an embodiment of the present application, electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to bus 704. Electronic device 700 may also include one or more of the following components connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or modem. Communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 710 as needed, so that computer programs read from the removable media can be installed into storage section 708 as needed.

[0137] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0138] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0139] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided in the embodiments of the present application.

[0140] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the computer program is executed by the processor 701. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0141] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0142] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0143] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

[0146] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: In response to receiving a target processing task for the data to be processed, obtaining first load information of the central processing unit and second load information of the hard disk processor in the target hard disk; Comparing the first load information with a first load threshold to obtain a first comparison result representing a load state of the central processing unit; Comparing the second load information with a second load threshold to obtain a second comparison result representing a load state of the hard disk processor; If the first comparison result indicates that the first load information is less than the first load threshold, and the second comparison result indicates that the second load information is greater than the second load threshold, determining the central processor as the target processor; When the first comparison result indicates that the first load information is less than the first load threshold, and the second comparison result indicates that the second load information is less than the second load threshold, determining the target processor from the central processing unit and the hard disk processor according to the target task type of the target processing task; If the first comparison result indicates that the first load information is greater than the first load threshold, and the second comparison result indicates that the second load information is greater than the second load threshold, determining the target processor from the central processing unit and the hard disk processor according to the target task type and the amount of data to be processed; When the target processor is the hard disk processor, the data to be processed and the target instruction are sent to the target hard disk, so that the hard disk processor performs the target processing task on the data to be processed in response to the target instruction, obtains the target data, and stores the target data.

2. The method according to claim 1, characterized in that The step of determining the target processor from the central processing unit and the hard disk processor according to the target task type of the target processing task includes: In a case where the target task type is a data compression type, the central processing unit is determined to be the target processor.

3. The method according to claim 1, characterized in that The step of determining the target processor from the central processing unit and the hard disk processor according to the target task type and the amount of data to be processed includes: Determining, based on the target task type and the data volume, a first computing resource consumption required for the central processing unit to execute the target processing task and a second computing resource consumption required for the hard disk processor to execute the target processing task; The target processor is determined from the central processing unit and the hard disk processor based on the first computing resource consumption and the second computing resource consumption.

4. The method according to claim 1, wherein The target processing task includes a plurality of target processing subtasks, and the target task type includes target task subtypes corresponding to each of the plurality of target processing subtasks; The step of determining the target processor from the central processing unit and the hard disk processor according to the target task type and the amount of data to be processed includes: Determining, according to the data volume and the target task subtype of each target processing subtask, a first computing resource sub-consumption required for the central processing unit to execute the target processing subtask and a second computing resource sub-consumption required for the hard disk processor to execute the target processing subtask; According to the execution order of multiple target processing subtasks, the first computing resource sub-consumption, and the second computing resource sub-consumption, a target processor corresponding to each target processing subtask is determined from the central processing unit and the hard disk processor.

5. The method according to claim 1, characterized in that The hard disk processor is further configured to: Obtaining the interface status of the target hard disk; When the interface status indicates that the target hard disk has no instructions to be executed, an optimization operation is performed on the stored data in the target hard disk, where the optimization operation includes at least one of a data cleaning operation, an error repair operation, and a defragmentation operation.

6. A data processing device, characterized in that: The device comprises: an acquisition module, configured to acquire first load information of a central processing unit and second load information of a hard disk processor in a target hard disk in response to receiving a target processing task for the data to be processed; a determination module, configured to determine, from the central processing unit and the hard disk processor, a target processor for executing the target processing task based on the first load information and the second load information; a sending module, configured to send the data to be processed and the target instruction to the target hard disk when the target processor is the hard disk processor, so that the hard disk processor performs the target processing task on the data to be processed in response to the target instruction, obtains target data, and stores the target data; The determination module includes: a first comparison submodule, configured to compare the first load information with a first load threshold to obtain a first comparison result representing a load state of the central processing unit; a second comparison submodule, configured to compare the second load information with a second load threshold to obtain a second comparison result representing a load state of the hard disk processor; a determination submodule, configured to determine the target processor from the central processing unit and the hard disk processor based on the first comparison result and the second comparison result; The determination submodule includes: a first determining unit, configured to determine that the central processor is the target processor if the first comparison result indicates that the first load information is less than the first load threshold and the second comparison result indicates that the second load information is greater than the second load threshold; a second determining unit configured to determine, if the first comparison result indicates that the first load information is less than the first load threshold and the second comparison result indicates that the second load information is less than the second load threshold, the target processor from the central processing unit and the hard disk processor according to a target task type of the target processing task; A third determination unit is used to determine the target processor from the central processing unit and the hard disk processor according to the target task type and the amount of data to be processed when the first comparison result indicates that the first load information is greater than the first load threshold and the second comparison result indicates that the second load information is greater than the second load threshold.

7. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images