Data processing method, device, equipment and storage medium

Abstract

The present disclosure provides a data processing method, apparatus, device and storage medium, wherein the method comprises: receiving a data processing request sent by a first data party; parsing the data processing request to obtain a block address of the basic logic block, an offset in the basic logic block and a length of data to be processed; determining a starting operation position of a data processing operation in the basic logic block according to the block address and the offset; and performing a data processing operation on data of the length of data to be processed from the starting operation position in the basic logic block.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data storage, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In the process of deep learning model training, it is necessary to provide high-speed data input / output to the GPU or AI (Artificial Intelligence) chip so that the GPU or AI chip can quickly obtain data in the data set used for model training. In related technologies, memory caching software such as MemCache (a distributed high-speed cache system developed by Brad Fitzpatrick of LiveJournal) is usually used to cache the data set in memory. Since the data set is usually large in size, this method consumes a lot of memory, and the high memory cost brings a lot of cost consumption to large-scale model training. Summary of the invention

[0003] The present disclosure provides a data processing method, apparatus, device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, which is applied to a data storage device, wherein the data storage device runs a file system implemented based on a solid-state hard disk, and the file system includes at least one basic logic block, the method comprising: receiving a data processing request sent by a first data party; parsing the data processing request to obtain a block address of the basic logic block, an offset in the basic logic block, and a length of data to be processed; determining a starting operation position of a data processing operation in the basic logic block according to the block address and the offset; and performing a data processing operation on data of the length of data to be processed in the basic logic block from the starting operation position.

[0005] In some optional embodiments, the first data entity includes: a GPU, an AI chip or a CPU.

[0006] In some optional embodiments, when the data processing request includes a data read request, the length of the data to be processed includes the length of the data to be read; and in the basic logic block, a data processing operation is performed on the data of the length of the data to be processed from the starting operation position, including: determining the readable data length in the basic logic block based on the offset and the total length of the basic logic block; when the readable data length is less than or equal to the length of the data to be read, performing a reading operation on the data corresponding to the readable data length in the basic logic block; or, when the readable data length is greater than the length of the data to be read, performing a reading operation on the data corresponding to the length of the data to be read in the basic logic block.

[0007] In some optional embodiments, when the data processing request includes a data write request, the data write request includes data to be written, and the length of the data to be processed includes the length of the data to be written; in the basic logic block, a data processing operation is performed on the data of the length of the data to be processed from the starting operation position, including: determining the length of writable data in the basic logic block based on the offset and the total length of the basic logic block; when the length of the writable data is less than or equal to the length of the data to be written, writing the data corresponding to the length of the writable data in the data to be written into the basic logic block; or, when the length of the writable data is greater than the length of the data to be written, writing the data corresponding to the length of the data to be written into the basic logic block.

[0008] In some optional embodiments, after parsing the data processing request to obtain the block address of the basic logic block, it also includes: determining the basic logic block corresponding to the block address as the target logic block, and discarding the data processing request if the device identifier in the target logic block does not belong to the data storage device.

[0009] In some optional embodiments, the method further includes: when writing data in the basic logic block, obtaining a first verification value based on the written data processing; when reading data in the basic logic block, obtaining a second verification value based on the read data processing; and when the first verification value is the same as the second verification value, determining that the reading is successful.

[0010] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing apparatus, which is applied to a data storage device, wherein the data storage device runs a file system implemented based on a solid-state hard disk, wherein the file system includes at least one basic logic block, and the apparatus includes: a request receiving module, which is used to receive a data processing request sent by a first data party; a parsing module, which is used to parse the data processing request to obtain a block address of the basic logic block, an offset in the basic logic block, and a length of data to be processed; a starting position determining module, which is used to determine a starting operation position of a data processing operation in the basic logic block according to the block address and the offset; and a processing module, which is used to perform a data processing operation on data of the length of data to be processed from the starting operation position in the basic logic block.

[0011] In some optional embodiments, the first data entity includes: a GPU, an AI chip or a CPU.

[0012] In some optional embodiments, when the data processing request includes a data read request, the length of the data to be processed includes the length of the data to be read; the processing module is used to perform a data processing operation on the data of the length of the data to be processed from the starting operation position in the basic logic block, including: determining the readable data length in the basic logic block based on the offset and the total length of the basic logic block; when the readable data length is less than or equal to the length of the data to be read, performing a read operation on the data corresponding to the readable data length in the basic logic block; or, when the readable data length is greater than the length of the data to be read, performing a read operation on the data corresponding to the length of the data to be read in the basic logic block.

[0013] In some optional embodiments, when the data processing request includes a data write request, the data write request includes data to be written, and the length of the data to be processed includes the length of the data to be written; the processing module is used to perform a data processing operation on the data of the length of the data to be processed from the starting operation position in the basic logic block, including: determining the length of writable data in the basic logic block based on the offset and the total length of the basic logic block; when the length of the writable data is less than or equal to the length of the data to be written, writing the data corresponding to the length of the writable data in the data to be written into the basic logic block; or, when the length of the writable data is greater than the length of the data to be written, writing the data corresponding to the length of the data to be written in the data to be written into the basic logic block.

[0014] In some optional embodiments, the device also includes: a request discarding module, which is used to determine the basic logical block corresponding to the block address as the target logical block, and discard the data processing request if the device identifier in the target logical block does not belong to the data storage device.

[0015] In some optional embodiments, the device also includes: a first verification value module, which is used to obtain a first verification value based on the written data when writing data in the basic logic block; a second verification value module, which is used to obtain a second verification value based on the read data when reading the data in the basic logic block; and a verification module, which is used to determine that the reading is successful when the first verification value is the same as the second verification value.

[0016] According to a third aspect of an embodiment of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method described in any one of the first aspects is implemented.

[0017] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the data processing method described in any one of the first aspects is implemented.

[0018] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, wherein when the program is executed by a processor, the data processing method described in any one of the first aspects is implemented.

[0019] In the disclosed embodiment, the data set required for model training can be directly stored in the solid-state drive, and the GPU or AI chip can read the data set stored in the solid-state drive based on the file system running on the data storage device. In this way, the data set does not need to be stored in the memory, which reduces the consumption of memory and saves a lot of training costs.

[0020] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] Figure 1 is a flow chart of a data processing method according to an exemplary embodiment;

[0023] Figure 2 is a flow chart of a reading operation according to an exemplary embodiment;

[0024] Figure 3 is a flowchart of a write operation according to an exemplary embodiment;

[0025] Figure 4 is a schematic diagram of a data processing device according to an exemplary embodiment;

[0026] Figure 5 is a schematic diagram of another data processing device according to an exemplary embodiment;

[0027] Figure 6 is a schematic diagram of another data processing device according to an exemplary embodiment;

[0028] Figure 7 The diagram is a schematic structural diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The specific methods described in the following exemplary embodiments do not represent all schemes consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] In the model training process of deep learning, the data in the data set required for GPU or AI chip calculation is usually stored in a separate device. In the embodiment of the present disclosure, the device storing the data in the data set is referred to as a data storage device. The data storage device can manage the data in the data set by running a file system based on a solid-state drive (SSD). Among them, a solid-state drive is a hard disk made of a solid-state electronic storage chip array, using NAND (non-AND) FLAAH (flash memory) as a storage medium. The file system is a method and data structure used by a data storage device to clarify the stored files, that is, a method of organizing files in storage.

[0033] The data processing method provided by the embodiment of the present disclosure can directly store the data set required for model training in the solid-state drive, and enable the GPU or AI chip to read the data set stored in the solid-state drive based on the file system running on the data storage device. In this method, there is no need to store the data set in the memory, which reduces the consumption of memory and saves a lot of training costs.

[0034] In order to enable those skilled in the art to better understand the technical solution provided by the present disclosure, the method and data structure of the file system implemented based on the solid state drive in the present disclosure are briefly described below.

[0035] In the disclosed embodiment, the SSD may be pre-divided into one or more disks, that is, one or more disk logic blocks may be obtained. Optionally, the size of each disk logic block may be set according to the actual storage data requirements. For example, the size of each disk logic block may be set to 4G, 8G, or 16G, etc. Since the storage space of the SSD is usually large, the disk logic blocks obtained by the division are also larger than the memory.

[0036] For any disk logic block, it may include one or more thread logic blocks. Each thread logic block corresponds to a dedicated input / output thread. Therefore, different thread logic blocks can input / output data in parallel based on different input / output threads.

[0037] For any thread logic block, it may include one or more basic logic blocks. The basic logic block is the basic processing unit obtained by pre-dividing the SSD. The space of each basic logic block is an integer multiple of 4k. For example, the size of a basic logic block can be 4k, 8k, 12k, etc., and there will be no basic logic block such as 13k. The size of the specific basic logic block can be set according to the characteristics of the actual stored data. For example, the data included in the data set is usually 6-8k in size, and the size of the basic logic block can be set to 4k.

[0038] In some optional embodiments, the disk logic block includes disk description information. The disk description information is used to describe the actual usage information of the current disk logic block. Exemplarily, the disk description information may include but is not limited to the number of reads and writes of the current disk logic block, the total number of basic logic blocks, the size of used space, the size of unused space, etc.

[0039] Since a disk logic block includes one or more thread logic blocks, in order to facilitate the search for different thread logic blocks, in some optional embodiments, the disk logic block may include a thread logic block list. When it is necessary to search for a thread logic block in a disk logic block, a quick search can be performed in the disk logic block based on the thread logic block list. In a possible implementation, the thread logic block list can be used as the starting address for searching for a thread logic block in the disk logic block, and each time a thread logic block in a disk logic block needs to be queried, the query can be performed from the starting address.

[0040] In some optional embodiments, the thread logic block includes thread logic block description information. The thread logic block description information is used to describe the relevant usage information of the current thread logic block. Exemplarily, the thread logic block description information can be used to describe whether the current thread logic block is available. For example, the read and write errors of the current thread logic block can be counted, and when the read and write errors of the current thread logic block are too many, the thread logic block description information can be used to record that the current thread logic block is unavailable. Exemplarily, the thread logic block description information can be used to describe whether the current thread logic block has been stopped. For example, the user can stop using the current thread logic block by actively modifying the configuration file, that is, the thread logic block description information can be used to record that the current thread logic block has been stopped.

[0041] In some optional embodiments, the thread logic block includes thread logic block device information. The thread logic block device information is used to describe information about a physical device associated with the current thread logic block. Exemplarily, the thread logic block device information can be used to record the number or ID of the disk to which the current thread logic block belongs. Since solid-state drives usually rely on certain physical devices, in this embodiment, the thread logic block device information can also be used to record the number or ID of the physical device to which the current thread logic block belongs.

[0042] In some optional embodiments, the thread logic block includes thread logic block statistics information. The thread logic block statistics information is used to describe the statistics information of the current thread logic block being used. Exemplarily, the thread logic block statistics information can be used to record the space capacity of the current thread logic block, the size of the space used by the current thread logic block, the number of abnormal basic logic blocks in the current thread logic block, or the number of read and write failures of the current thread logic block.

[0043] In some optional embodiments, the thread logic block includes a free logic block list. The free logic block list is used to describe the list information of the basic logic blocks in the current thread logic block that are in an idle state. Since the thread logic block usually includes a large number of basic logic blocks, and different basic logic blocks may be in different states. For example, some basic logic blocks may be in an idle state, and some basic logic blocks may be in use. In the embodiment of the present disclosure, the free logic block list can be used to display the basic logic blocks in the current thread logic block that are in an idle state in the form of a list. Through the free logic block list, the basic logic blocks in the current thread logic block that are in an idle state can be determined more quickly and directly, so that the basic logic blocks in the current thread logic block that are in an idle state can be allocated and used more quickly.

[0044] In some optional embodiments, the thread logic block includes a logic block sequence list. Among them, the logic block sequence list can display the basic logic blocks in the current thread logic block in a list form according to different states. The thread logic block usually includes a large number of basic logic blocks, and the basic logic blocks may be in use, idle or abnormal state. In a possible implementation, the basic logic blocks in the current thread logic block can be displayed in the form of a list in the order of being in use, idle and abnormal state through the logic block sequence list. This method can display the basic logic blocks in the current thread logic block in the form of a list based on the state of the basic logic blocks, so that the corresponding basic logic blocks can be quickly determined according to the state of the required basic logic blocks.

[0045] In some optional embodiments, the thread logic block includes a list of recently used logic blocks. The recently used logic block list includes a list obtained by sorting from high to low according to the number of times different basic logic blocks in the current thread logic block are used. In a possible implementation, the total number of basic logic blocks in the recently used logic block list can be preset. For example, the number of basic logic blocks in the recently used logic block list can be preset to 100, and then the basic logic blocks in the current thread logic block can be further sorted from high to low according to the number of times different basic logic blocks are used. If the ranking of the basic logic blocks at the end of the list exceeds 100, it can be considered to be eliminated, so that the basic logic blocks in the recently used logic block list can be updated. When it is necessary to search for the basic logic blocks in the current thread logic block, the basic logic blocks can be searched from the recently used logic block list first, and the required basic logic blocks can be hit with a greater probability, thereby improving the efficiency of hitting the basic logic blocks.

[0046] In some optional embodiments, the thread logic block includes input / output thread information. The input / output thread information is used to indicate the relevant information of the input / output thread exclusive to the current thread logic block. Any thread logic block corresponds to an exclusive input / output thread, and the exclusive input / output thread can interact with the worker thread to implement data input / output of the current thread logic block.

[0047] In some optional embodiments, the basic logic block includes a first hash value. The first hash value is used to indicate the location of the current basic logic block. For example, after receiving a data processing request, the data storage device can parse the first hash value and quickly locate the corresponding basic logic block according to the first hash value.

[0048] In some optional embodiments, the basic logic block includes a device identifier. The device identifier is used to indicate the data storage device to which the current basic logic block belongs. For example, after the data storage device locates the corresponding basic logic block according to the first hash value, the device identifier in the current basic logic block can be further checked. If the device identifier indicates that it does not belong to the current data storage device, it means that the current data processing request does not belong to the data storage device, and the data processing request can be further processed. If the device identifier indicates that it does not belong to the current data storage device, it means that the current data processing request does not belong to the data storage device, and the data processing request can be further discarded. This method can realize the screening of data processing requests.

[0049] In some optional embodiments, the basic logic block includes a first check value. The first check value is a check value obtained by processing the written data when writing data in the current basic logic block. For example, when writing data in the current basic logic block, a first check value can be obtained by processing the written data using a cyclic redundancy check (CRC). When it is necessary to read or process the data in the current basic logic block, a second check value can be obtained based on the currently stored data. For example, a second check value can be obtained based on the currently stored data using a CRC. If the first check value is the same as the second check value, it can be determined that the data written to the current basic logic block is consistent with the data stored in the current basic logic block, that is, it can be determined that the data stored in the current basic logic block has not changed during the storage process.

[0050] In one possible implementation, when writing data in a basic logic block, a first verification value can be obtained based on the written data processing; when reading data in the basic logic block, a second verification value can be obtained based on the read data processing; if the first verification value is the same as the second verification value, it is determined that the reading is successful; otherwise, it is determined that the reading has failed.

[0051] In some optional embodiments, the basic logic block includes a current read number. The current read number is used to indicate the number of the current basic logic block read by the client. If the current read number is greater than zero, it means that the current basic logic block is being read, and if the current read number is equal to zero, it means that the current basic logic block has not been read. In the process of deleting data in the basic logic block, it can be deleted according to the current read number in the basic logic block. For example, the data in the basic logic block whose current read number is equal to zero can be deleted directly without affecting the reading of the data in the current basic logic block. For example, the data in the basic logic block whose current read number is greater than zero cannot be deleted directly to avoid affecting the reading of the data in the current basic logic block.

[0052] In order to make the data processing method provided by the present disclosure clearer, the execution process of the scheme provided by the present disclosure is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] See also Figure 1 , Figure 1 1 is a flow chart of a data processing method shown in an embodiment provided by the present disclosure. The data processing method is applied to a data storage device. The data storage device manages the data in the solid-state hard disk by running a file system based on the solid-state hard disk. The present disclosure embodiment takes the data set used for model training stored in the solid-state hard disk as an example. The data stored in other achievable embodiments may include data for any other purpose. Figure 1 As shown, the process includes:

[0054] Step 101: Receive a data processing request sent by a first data party.

[0055] In an embodiment of the present disclosure, the first data party includes a data party that can interact with the data storage device. In a possible implementation, the first data party may include at least one of the following but is not limited to: a GPU, an AI chip, or a CPU. The first data party may be set in a device different from the data storage device; or, the first data party may be set in the same data storage device. In an embodiment of the present disclosure, there is no limitation on the specific location of the first data party.

[0056] When the first data party needs to interact with the solid-state hard disk in the data storage device, it can send a data processing request to the data storage device. For example, when the CPU needs to write data of a data set to the solid-state hard disk of the data storage device, it can send a data write request to the data storage device. For example, when the GPU needs to read data of a data set from the solid-state hard disk of the data storage device, it can send a data read request to the data storage device.

[0057] In this step, the data storage device may receive a data processing request sent by the first data party, for example, a data write request sent by the first data party, or a data read request sent by the first data party.

[0058] Step 102: parse the data processing request to obtain the block address of the basic logic block, the offset in the basic logic block and the length of the data to be processed.

[0059] Because in the file system implemented based on the solid-state drive, the solid-state drive is composed of a large number of basic logic blocks. Among them, different basic logic blocks have different addresses, that is, the block addresses of the basic logic blocks in the embodiment of the present disclosure. And because the basic logic block has a certain size, the specific position inside the basic logic block is measured by the offset, that is, the offset in the basic logic block in the embodiment of the present disclosure. The length of the data to be processed is the length of the data that needs to be processed corresponding to the data processing request. For example, when the data processing request includes a data read request, the length of the data to be processed may include the length of the data to be read. For another example, when the data processing request includes a data write request, the length of the data to be processed may include the length of the data to be written.

[0060] After receiving the data processing request sent by the first data party, the data storage device in this step can parse the data processing request to obtain the block address of the basic logic block corresponding to the data processing request, the offset in the corresponding basic logic block, and the length of the data to be processed. Exemplarily, this step can perform a hash operation on the file name corresponding to the data processing request, and the block address, offset, and length of the data to be processed of the corresponding basic logic block can be determined based on the hash value obtained by the operation.

[0061] Step 103: Determine a starting operation position of a data processing operation in the basic logic block according to the block address and the offset.

[0062] After parsing and obtaining the block address of the basic logic block, this step can determine the basic logic block corresponding to the data processing request from the solid-state hard disk according to the block address. After determining the basic logic block corresponding to the data processing request, the specific offset position in the basic logic block can be further determined according to the offset as the starting operation position of the data processing operation.

[0063] Exemplarily, the basic logic blocks included in the solid-state drive can be numbered in advance as the block addresses of the corresponding basic logic blocks. Thus, after parsing the block addresses of the basic logic blocks corresponding to the data processing request, the corresponding basic logic blocks can be determined according to the pre-set block addresses. After determining the basic logic blocks, the corresponding starting operation positions can be obtained by offsetting in the basic logic blocks. For example, in the case where the size of the basic logic block is 8k, if the offset parsed is 4k, the position of the offset of 4k can be offset in the basic logic block as the starting operation position of the data processing operation.

[0064] Step 104: In the basic logic block, a data processing operation is performed on the data of the length of the data to be processed from the starting operation position.

[0065] After determining the starting operation position in the basic logic block corresponding to the data processing request, this step can start from the starting operation position to perform data processing operations on the data of the length of the data to be processed. For example, this step can start from the starting operation position and write the data corresponding to the length of the data to be processed into the basic logic block. Alternatively, this step can start from the starting operation position and read the data corresponding to the length of the data to be processed from the basic logic block.

[0066] The data processing method provided by the embodiment of the present disclosure can directly store the data set required for model training in the solid-state drive, and enable the GPU or AI chip to read the data set stored in the solid-state drive based on the file system running on the data storage device. In this method, there is no need to store the data set in the memory, which reduces the consumption of memory and saves a lot of training costs.

[0067] In addition, since the memory space is usually at the GB level, and the data set used for model training is usually at the TB level, if the complete data set is stored in memory, multiple memories are usually required to achieve it, so the GPU or AI chip needs to access the data set in the memory across nodes when reading the data in the data set, which reduces the data reading efficiency. In the disclosed embodiment, the solid-state drive has a larger storage space than the memory, so a larger data set can be fully stored, and the GPU or AI chip reads the data in the data set more efficiently than storing the data set in the memory. Furthermore, the data set cached in the memory will be lost in the event of a restart or power failure. In the disclosed embodiment, the data set is stored in the solid-state drive, which can effectively avoid data loss in the event of a restart or power failure.

[0068] In some optional embodiments, the data processing request sent by the first data party is a data read request, and the length of the data to be processed may include the length of the data to be read. The length of the data to be read is the length of the data that needs to be read by the data read request. For example, when the first data party is a GPU or an AI chip, the first data party needs to read a large amount of data in the data set from the solid-state drive to complete the model training. In the case where the data processing request is a data read request, such as Figure 2 As shown, the specific implementation of step 104 may include the following steps:

[0069] Step 201: Determine the length of data that can be read in the basic logic block based on the offset and the total length of the basic logic block.

[0070] After determining the basic logic block corresponding to the data processing request, an offset is performed in the basic logic block based on the offset obtained by parsing, and the starting position of the read can be determined in the basic logic block. The data between the starting position and the end position of the basic logic block is the data that can be read in the basic logic block, and the length of the data that can be read is the readable data length in the embodiment of the present disclosure.

[0071] This step can determine the length of the data that can be read in the basic logic block, that is, determine the readable data length, based on the offset obtained by parsing and the total length of the basic logic block. For example, the readable data length can be obtained by subtracting the offset from the total length of the basic logic block.

[0072] Step 202, when the readable data length is less than or equal to the length of the data to be read, a read operation is performed on the data corresponding to the readable data length in the basic logic block; or, when the readable data length is greater than the length of the data to be read, a read operation is performed on the data corresponding to the length of the data to be read in the basic logic block.

[0073] The length of data to be read is the length of data to be read corresponding to the data read request. The length of readable data is the length of data that can be read in the basic logic block corresponding to the data read request. Since data of the length of data to be read may be stored across different basic logic blocks, the length of data to be read may be greater than the length of readable data in the basic logic block. This step can realize the reading of data in the basic logic block corresponding to the data read request by comparing the length of readable data with the length of data to be read.

[0074] When the length of readable data in the basic logic block corresponding to the data read request is less than or equal to the length of the data to be read that needs to be read by the data read request, the data corresponding to the length of the readable data can be read from the starting operation position (i.e., the starting reading position) in the basic logic block. For example, the data corresponding to the length of the readable data in the basic logic block can be returned to the first data party. For the data of the remaining length of the data to be read, the data can be read from another basic logic block through the next data read operation to finally complete the reading of the data corresponding to the complete length of the data to be read.

[0075] When the readable data length in the basic logic block corresponding to the data read request is greater than the length of the data to be read required by the data read request, a reading operation can be performed on the data corresponding to the length of the data to be read starting from the starting operation position (i.e., the starting reading position) in the basic logic block.

[0076] In the disclosed embodiment, by comparing the length of the data to be read with the length of the data that can be read in the basic logic block, the data in the basic logic block can be read. Thus, the data stored in the solid state drive can be read out, so that the first data party can read the data set in the solid state drive.

[0077] In some optional embodiments, the data processing request sent by the first data party is a data write request, the data write request may include the data to be written, and the length of the data to be processed may include the length of the data to be written. The length of the data to be written is the length of the data to be written by the data write request. For example, when the first data party is a CPU, the first data party needs to write the data in the data set to a solid state drive. In the case where the data processing request is a data read request, such as Figure 3 As shown, the specific implementation of step 104 may include the following steps:

[0078] Step 301: Determine the length of data that can be written into the basic logic block based on the offset and the total length of the basic logic block.

[0079] After determining the basic logic block corresponding to the data write request, the position after the offset in the corresponding basic logic block is the write start position of the writable data in the basic logic block. The length from the write start position to the end position of the basic logic block is the length of the writable data in the basic logic block, that is, the writable data length in the embodiment of the present disclosure.

[0080] This step can determine the length of writable data in the basic logic block corresponding to the data write request based on the offset and the total length of the basic logic block. For example, the length of writable data in the basic logic block can be obtained by subtracting the offset from the total length of the basic logic block corresponding to the data write request as the length of writable data.

[0081] Step 302, when the length of the writable data is less than or equal to the length of the data to be written, write the data corresponding to the length of the writable data in the data to be written into the basic logic block; or, when the length of the writable data is greater than the length of the data to be written, write the data corresponding to the length of the data to be written in the data to be written into the basic logic block.

[0082] The writable data length is the length of the writable data in the basic logic block corresponding to the data write request. The to-be-written data length is the length of the to-be-written data included in the data write request. Since the length of the to-be-written data included in the data write request may be larger than the size of the entire basic logic block, it may be necessary to use multiple basic logic blocks to write the complete to-be-written data. This step can be achieved by comparing the writable data length with the to-be-written data length to write the to-be-written data into the corresponding basic logic block.

[0083] In the case where the length of writable data in the basic logic block corresponding to the data write request is less than or equal to the length of the data to be written included in the data write request, the data corresponding to the length of writable data in the data to be written included in the data write request can be written into the basic logic block. The remaining data in the data to be written can be written into another basic logic block through the next data write operation, and finally the data to be written is completely written into the solid state drive.

[0084] If the length of data that can be written in the basic logic block corresponding to the data write request is greater than the length of the data to be written included in the data write request, the data corresponding to the length of the data to be written in the data to be written can be written into the basic logic block. That is, if the basic logic block corresponding to the data write request can store complete data to be written, the complete data to be written can be directly written into the basic logic block.

[0085] In the disclosed embodiment, by comparing the length of the data to be written with the length of the data that can be written in the basic logic block, the data to be written included in the data write request can be written into the corresponding basic logic block. Thus, the data in the data set can be written into the solid state drive, so that when the first data party needs to read the data set, the data in the data set can be directly read from the solid state drive.

[0086] Figure 4As shown, the present disclosure provides a data processing device, which can execute the data processing method of any embodiment of the present disclosure. The device is applied to a data storage device, the data storage device runs a file system based on a solid state drive, and the file system includes at least one basic logic block. The device includes: a request receiving module 401, a parsing module 402, a starting position determining module 403 and a processing module 404. Among them:

[0087] The request receiving module 401 is used to receive a data processing request sent by a first data party;

[0088] A parsing module 402 is used to parse the data processing request to obtain the block address of the basic logic block, the offset in the basic logic block and the length of the data to be processed;

[0089] A starting position determination module 403, configured to determine a starting operation position of a data processing operation in the basic logic block according to the block address and the offset;

[0090] The processing module 404 is used to perform a data processing operation on the data of the length of the data to be processed from the starting operation position in the basic logic block.

[0091] Optionally, the first data entity includes: a GPU, an AI chip or a CPU.

[0092] Optionally, in the case where the data processing request includes a data reading request, the length of the data to be processed includes the length of the data to be read;

[0093] The processing module 404 is used to perform a data processing operation on the data of the length of the data to be processed from the starting operation position in the basic logic block, including:

[0094] Determine the length of data that can be read in the basic logic block based on the offset and the total length of the basic logic block;

[0095] In the case where the readable data length is less than or equal to the length of the data to be read, a reading operation is performed on the data corresponding to the readable data length in the basic logic block; or,

[0096] In a case where the length of the readable data is greater than the length of the data to be read, a read operation is performed on the data corresponding to the length of the data to be read in the basic logic block.

[0097] Optionally, in the case where the data processing request includes a data writing request, the data writing request includes data to be written, and the length of the data to be processed includes the length of the data to be written of the data to be written;

[0098] The processing module 404 is used to perform a data processing operation on the data of the length of the data to be processed from the starting operation position in the basic logic block, including:

[0099] Determine the length of data that can be written in the basic logic block based on the offset and the total length of the basic logic block;

[0100] In the case where the length of the writable data is less than or equal to the length of the data to be written, writing the data corresponding to the length of the writable data in the data to be written into the basic logic block; or,

[0101] In a case where the length of the writable data is greater than the length of the data to be written, data in the data to be written corresponding to the length of the data to be written is written into the basic logic block.

[0102] Alternatively, if Figure 5 As shown, the device also includes:

[0103] The request discarding module 501 is used to determine the basic logic block corresponding to the block address as the target logic block, and discard the data processing request if the device identifier in the target logic block does not belong to the data storage device.

[0104] Alternatively, if Figure 6 As shown, the device also includes:

[0105] A first check value module 601 is used to obtain a first check value according to the written data when writing data in the basic logic block;

[0106] A second check value module 602, configured to obtain a second check value according to the read data when reading the data in the basic logic block;

[0107] The verification module 603 is used to determine that the reading is successful when the first verification value is the same as the second verification value.

[0108] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of at least one embodiment of the present disclosure. A person of ordinary skill in the art can understand and implement them without paying creative labor.

[0109] The present disclosure also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the data processing method of any embodiment of the present disclosure when executing the program.

[0110] Figure 7 A more specific hardware structure diagram of a computer device provided by an embodiment of the present disclosure is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other in communication within the device through the bus 1050.

[0111] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0112] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0113] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0114] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0115] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0116] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0117] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data processing method of any embodiment of the present disclosure can be implemented.

[0118] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present disclosure is not limited thereto.

[0119] In some optional embodiments, the embodiments of the present disclosure provide a computer program product, including computer readable codes, and when the computer readable codes are executed on a device, a processor in the device executes the data processing method provided in any of the above embodiments. The computer program product can be implemented in hardware, software, or a combination thereof.

[0120] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the inventions claimed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not claimed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0121] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0122] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A data processing method, characterized in that: Applied to a data storage device, the data storage device is used to store data sets required for deep learning model training; the data storage device runs a file system implemented based on a solid-state drive, the file system includes at least one disk logic block, each disk logic block includes at least one thread logic block, and each thread logic block includes at least one basic logic block; The disk logic block is obtained by dividing the solid state drive, each thread logic block corresponds to an exclusive input / output thread, and the method includes: receiving a data processing request sent by the first data party; Parsing the data processing request to obtain the block address of the basic logic block, the offset in the basic logic block and the length of the data to be processed; Determining a starting operation position of a data processing operation in the basic logic block according to the block address and the offset; In the basic logic block, a data processing operation is performed on the data of the length of the data to be processed from the starting operation position; The parsing of the data processing request to obtain the block address of the basic logic block, the offset in the basic logic block and the length of the data to be processed includes: A hash operation is performed on the file corresponding to the data processing request, and a block address of the basic logic block, an offset in the basic logic block, and a length of data to be processed are determined based on the hash value obtained by the operation.

2. The method according to claim 1, characterized in that The first data entity includes: a GPU, an AI chip or a CPU.

3. The method according to claim 1 or 2, characterized in that: In the case where the data processing request includes a data reading request, the length of the data to be processed includes the length of the data to be read; The step of performing a data processing operation on the data of the to-be-processed data length from the starting operation position in the basic logic block includes: Determine the length of data that can be read in the basic logic block based on the offset and the total length of the basic logic block; In the case where the readable data length is less than or equal to the length of the data to be read, a reading operation is performed on the data corresponding to the readable data length in the basic logic block; or, In a case where the length of the readable data is greater than the length of the data to be read, a read operation is performed on the data corresponding to the length of the data to be read in the basic logic block.

4. The method according to claim 1 or 2, characterized in that: In the case where the data processing request includes a data writing request, the data writing request includes data to be written, and the length of the data to be processed includes the length of the data to be written of the data to be written; The step of performing a data processing operation on the data of the to-be-processed data length from the starting operation position in the basic logic block includes: Determine the length of data that can be written in the basic logic block based on the offset and the total length of the basic logic block; When the length of the writable data is less than or equal to the length of the data to be written, writing the data corresponding to the length of the writable data in the data to be written into the basic logic block; or In a case where the length of the writable data is greater than the length of the data to be written, data in the data to be written corresponding to the length of the data to be written is written into the basic logic block.

5. The method according to claim 1, characterized in that: After parsing the data processing request to obtain the block address of the basic logic block, the method further includes: The basic logic block corresponding to the block address is determined as a target logic block, and when the device identifier in the target logic block does not belong to the data storage device, the data processing request is discarded.

6. The method according to claim 1, characterized in that The method further comprises: When writing data in the basic logic block, obtaining a first check value according to the written data processing; When reading the data in the basic logic block, obtaining a second check value according to the read data processing; When the first verification value is the same as the second verification value, it is determined that the reading is successful.

7. A data processing device, characterized in that: Applied to a data storage device, the data storage device is used to store data sets required for deep learning model training; the data storage device runs a file system implemented based on a solid-state drive, the file system includes at least one disk logic block, each disk logic block includes at least one thread logic block, and each thread logic block includes at least one basic logic block; The disk logic block is obtained by dividing the solid state hard disk, each thread logic block corresponds to an exclusive input / output thread, and the device includes: A request receiving module, used for receiving a data processing request sent by a first data party; A parsing module, used for parsing the data processing request to obtain the block address of the basic logic block, the offset in the basic logic block and the length of the data to be processed; a starting position determination module, configured to determine a starting operation position of a data processing operation in the basic logic block according to the block address and the offset; A processing module, configured to perform a data processing operation on the data of the length of the data to be processed from the starting operation position in the basic logic block; The parsing module is used to perform a hash operation on the file corresponding to the data processing request, and determine the block address of the basic logic block, the offset in the basic logic block and the length of the data to be processed based on the hash value obtained by the operation.

8. The device according to claim 7, characterized in that The first data entity includes: a GPU, an AI chip or a CPU.

9. The device according to claim 7 or 8, characterized in that In the case where the data processing request includes a data reading request, the length of the data to be processed includes the length of the data to be read; The processing module is used to perform a data processing operation on the data of the length of the data to be processed from the starting operation position in the basic logic block, including: Determine the length of data that can be read in the basic logic block based on the offset and the total length of the basic logic block; In the case where the readable data length is less than or equal to the length of the data to be read, a reading operation is performed on the data corresponding to the readable data length in the basic logic block; or, In a case where the length of the readable data is greater than the length of the data to be read, a read operation is performed on the data corresponding to the length of the data to be read in the basic logic block.

10. The device according to claim 7 or 8, characterized in that In the case where the data processing request includes a data writing request, the data writing request includes data to be written, and the length of the data to be processed includes the length of the data to be written of the data to be written; The processing module is used to perform a data processing operation on the data of the length of the data to be processed from the starting operation position in the basic logic block, including: Determine the length of data that can be written in the basic logic block based on the offset and the total length of the basic logic block; When the length of the writable data is less than or equal to the length of the data to be written, writing the data corresponding to the length of the writable data in the data to be written into the basic logic block; or In a case where the length of the writable data is greater than the length of the data to be written, data in the data to be written corresponding to the length of the data to be written is written into the basic logic block.

11. The device according to claim 7, characterized in that The device also includes: The request discarding module is used to determine the basic logic block corresponding to the block address as the target logic block, and discard the data processing request when the device identifier in the target logic block does not belong to the data storage device.

12. The device according to claim 7, characterized in that The device also includes: A first check value module, configured to obtain a first check value according to the written data when writing data in the basic logic block; A second check value module, configured to obtain a second check value according to the read data when reading the data in the basic logic block; The verification module is used to determine that the reading is successful when the first verification value is the same as the second verification value.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images