File processing method and system based on block storage and related device

Abstract

The invention provides a file processing method and system based on block storage and a related device, and relates to the field of block storage of files, and the method comprises the following steps: dividing each data block belonging to a file into a data packet, and storing the data packet into a storage device in a storage cluster. Subsequently, when the file is processed, the computing device sends a processing request, and the storage device performs near data processing on the file according to the processing request and returns a processing result to the computing device. By the adoption of the method, the near data processing problem and the file semantic problem can be efficiently solved, and compared with a traditional file processing method, bandwidth consumption is reduced.

Description

Technical Field

[0001] The present application relates to the field of block storage, and in particular to a file processing method, system and related devices based on block storage. Background Art

[0002] The traditional file storage method based on block storage is that before the computing side sends the file to the storage device through the network side, the computing side will divide the file into multiple data blocks. The network side receives the multiple data blocks sent by the computing side and disperses / randomly stores the multiple data blocks in multiple storage devices in the storage cluster.

[0003] This storage method breaks the semantic information of the file, resulting in the inability to efficiently implement near-data processing and file semantics issues. Summary of the invention

[0004] The present application provides a file processing method, system and related devices based on block storage. The file storage method of the present application is adopted to store multiple data blocks belonging to a file in a storage device, so that the semantic information of the file is not destroyed, which facilitates the subsequent efficient processing of near data processing problems and file-level semantic problems.

[0005] In a first aspect, the present application provides a file processing method based on block storage, which is described from the network device side. The method includes:

[0006] The network device receives a first write request sent by the computing device, wherein the first write request includes a plurality of data blocks, and each of the plurality of data blocks carries a block identifier;

[0007] The network device divides the data blocks belonging to the first file among the multiple data blocks into the first data packet according to the metadata; wherein the metadata includes an identifier of the file to which each data block among the multiple data blocks belongs;

[0008] The network device sends a second write request to a first storage device in the storage cluster, the second write request includes the first data packet, and the second write request is used to store the data block in the first data packet on the first storage device.

[0009] It can be seen that multiple data blocks belonging to one file are divided into one data packet, and then each data block in one data packet is stored on one storage device, that is, each data block belonging to one file is stored on one storage device.

[0010] By adopting the storage method of the present application, when a subsequent computing device needs to perform a certain processing operation on a certain file, the computing device can send a processing request to the corresponding storage device or network device, so that the storage device can process a certain file nearby according to the processing request, or further process the processing result of the storage device through the network device, and return the final processing result directly to the computing device. Compared with dispersing and storing multiple data blocks of a file in multiple storage devices, when the file needs to be processed, multiple data blocks belonging to the file are obtained from multiple storage devices, and then the file is sent to the computing device, and then the computing device processes the file. The solution of the present application saves network bandwidth consumption, realizes efficient near data processing, and realizes problems based on file semantic information. Traditional storage methods cannot solve the problem of near data processing, because in traditional storage methods, the file is dispersedly stored on multiple storage devices, and it is impossible to realize the problem of near data processing and problems based on file semantics through a certain storage device.

[0011] Based on the first aspect, in a possible implementation manner, before the network device divides the data blocks belonging to the first file among the multiple data blocks into the first data packet according to the metadata, the method further includes:

[0012] The network device obtains the metadata from a metadata server.

[0013] It can be understood that when the computing side stores data and metadata separately, the network device can obtain metadata from the metadata server.

[0014] Based on the first aspect, in a possible implementation manner, the first write request includes the metadata.

[0015] It can be understood that when the computing side stores data and metadata together, the metadata can be directly carried in the first write request.

[0016] Based on the first aspect, in a possible implementation manner, the method further includes:

[0017] The network device divides the data blocks belonging to the second file among the plurality of data blocks into a second data packet according to the metadata;

[0018] The network device determines, from the storage cluster, a second storage device for storing the second data packet;

[0019] The network device sends a third write request to the second storage device, the write request includes the second data packet, and the third write request is used to store the data block in the second data packet on the second storage device.

[0020] In the case where the multiple data blocks in the first write request include both data blocks belonging to the first file and data blocks belonging to the second file, the network device divides the data blocks belonging to different files into different data packets, so that the data blocks belonging to the same file are divided into one data packet, and the one data packet is stored on a storage device in the storage cluster. It should be noted that the first data packet may be stored on the first storage device and the second data packet may be stored on the second storage device; or both the first data packet and the second data packet may be stored on the first storage device or the second storage device. As long as the multiple data blocks (one data packet) belonging to one file are stored on one storage device in the storage cluster, it will be sufficient.

[0021] Based on the first aspect, in a possible implementation manner, the method further includes:

[0022] The network device records a first corresponding relationship, where the first corresponding relationship is used to indicate a corresponding relationship between an identifier of the first file and the first storage device.

[0023] Based on the first aspect, in a possible implementation manner, the method further includes:

[0024] The network device receives a first processing request sent by the computing device, where the first processing request includes an identifier of the first file and an identifier of a first operation;

[0025] The network device determines, according to the first corresponding relationship, the first storage device storing the first file;

[0026] The network device sends a second processing request to the first storage device, where the second processing request is used to instruct to perform the first operation on the first file;

[0027] The network device obtains a first processing result corresponding to the second processing request from the first storage device;

[0028] The network device returns the first processing result to the computing device.

[0029] Compared to reading the entire first file from multiple storage devices and then sending it to a computing device, which processes the first file, the method described in this application enables the first storage device to perform a first operation on the first file according to a second processing request, obtain a first processing result, and implement near data processing. The first storage device only needs to return the first processing result to the computing device. The method described in this application reduces bandwidth consumption and implements efficient computing of near data processing. At the same time, the method described in this application facilitates processing of file-level semantic issues.

[0030] Based on the first aspect, in a possible implementation manner, the first processing request also includes an identifier of the second file, and the method further includes:

[0031] The network device determines, according to the first corresponding relationship, a second storage device storing the second file;

[0032] The network device sends a third processing request to the second storage device, where the third processing request is used to instruct to perform the first operation on the second file;

[0033] The network device obtains, from the second storage device, a second processing result corresponding to the third processing request;

[0034] The network device summarizes the first processing result and the second processing result and returns them to the computing device.

[0035] In one implementation, the computing device needs to process multiple files at the same time. Taking the processing of two files at the same time as an example, the computing device sends a first processing request, which may include the identifier of the first file, the identifier of the second file, and the identifier of the first operation. The network device receives the first processing request and determines that the first storage device stores the first file and the second storage device stores the second file. Then the network device sends a second processing request to the first storage device and sends a third processing request to the second storage device. The second processing request includes the identifier of the first file and the identifier of the first operation, and the third processing request includes the identifier of the second file and the identifier of the first operation. Then the first storage device performs the first operation on the first file according to the second processing request, obtains the first processing result, and implements near data processing. Similarly, the second storage device performs the first operation on the second file according to the third processing request, obtains the second processing result, and implements near data processing. The network device obtains the first processing result from the first storage device and the second processing result from the second storage device, and then summarizes the first processing result and the second processing result and returns them to the computing device.

[0036] 1) In the present application, there is no need to read the first file and the second file to the computing device side respectively, and the computing device side performs calculations based on the first file and the second file. Instead, the first storage device and the second storage device directly perform near data processing, and only need to return the final result to the computing device, which reduces the bandwidth consumption and realizes efficient calculation of near data processing. At the same time, the method described in the present application is convenient for processing file-level semantic issues. 2) In addition, it should be noted that the present application realizes the processing and calculation of two files through one processing request, but in actual applications, the processing and calculation of a larger number of files can be realized through one processing request. 3) Compared with the computing device sending multiple read requests, reading multiple files to the computing device through multiple read requests, and the computing device calculating the multiple files, the method described in the present application realizes that multiple storage devices perform near data processing and calculation at the same time, and the results of multiple near data processing are summarized through the network device and returned to the computing device, which greatly reduces the bandwidth consumption, realizes efficient calculation, and improves the efficiency of file processing. Based on the first aspect, in a possible implementation, the first operation includes any one or more of read operation, character statistics, character screening, and character sorting.

[0037] It can be understood that the first operation includes a read operation, and the read operation is a type of processing operation.

[0038] In a second aspect, the present application provides a file processing method based on block storage, described from the first storage device side, the method comprising:

[0039] A first storage device in the storage cluster receives a second write request sent by a network device, the second write request includes a first data packet, the first data packet includes a data block belonging to a first file, and the second write request is used to store the data block in the first data packet on the first storage device;

[0040] The data block in the first data packet in the second write request is stored on the first storage device.

[0041] It can be seen that the data blocks included in the first data packet all belong to the first file, and the first storage device stores the data blocks belonging to the first file on the first storage device, so as to facilitate the subsequent near data processing calculations and file-level semantic problem calculations of the first file by the first storage device.

[0042] Based on the second aspect, in a possible implementation, the method also includes: the first storage device records a second correspondence relationship, and the second correspondence relationship is used to indicate the correspondence between the block identifier of each data block in the first data packet and the storage position of each data block in the first data packet in the first storage device.

[0043] Based on the second aspect, in a possible implementation, the method further includes:

[0044] The first storage device receives a second processing request sent by the network device, where the second processing request is used to instruct to perform the first operation on the first file;

[0045] The first storage device performs the first operation on the first file according to the second processing request to obtain the first processing result;

[0046] The first storage device sends the first processing result to the network device.

[0047] It can be seen that the embodiment of the present application implements near data processing calculations of the first storage device.

[0048] Based on the second aspect, in a possible implementation manner, the second processing request includes an identifier of the first file and an identifier of the first operation;

[0049] The first storage device determines, according to the identifier of the first file, a block identifier of a data block included in the first file;

[0050] The first storage device obtains the data blocks of the first file according to the block identifiers of the data blocks included in the first file and the second corresponding relationship, and performs the first operation on the data blocks of the first file to obtain the first processing result.

[0051] Based on the second aspect, in a possible implementation, the network device records a third correspondence relationship, where the third correspondence relationship is used to indicate a correspondence relationship between an identifier of the first file and a block identifier of each data block in the first data packet;

[0052] The first storage device determines, according to the identifier of the first file, a block identifier of a data block included in the first file, including:

[0053] The first storage device sends an acquisition request to the network device, where the acquisition request includes an identifier of the first file;

[0054] The first storage device receives the block identifier of the data block included in the first file sent by the network device based on the acquisition request, and the block identifier of the data block included in the first file is determined by the network device according to the third corresponding relationship.

[0055] Based on the second aspect, in a possible implementation manner, the first data packet carries an identifier of the first file,

[0056] The first storage device records a third corresponding relationship according to the identifier of the first file, where the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet;

[0057] The first storage device determines, according to the identifier of the first file, a block identifier of a data block included in the first file, including:

[0058] The first storage device determines the block identifier of the data block included in the first file according to the locally recorded third corresponding relationship and the identifier of the first file.

[0059] Based on the second aspect, in a possible implementation, the second processing request includes a block identifier of each data block in multiple data blocks of the first file and an identifier of the first operation; the block identifier of each data block is determined by the network device according to a recorded third corresponding relationship; the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet;

[0060] The first storage device obtains multiple data blocks of the first file according to the second corresponding relationship, and performs the first operation on the data blocks of the first file to obtain the first processing result.

[0061] Based on the second aspect, in a possible implementation, the first operation includes any one or more of a read operation, character statistics, character screening, and character sorting.

[0062] It can be understood that the first operation includes a read operation, and the read operation is a type of processing operation.

[0063] In a third aspect, the present application provides a file processing system based on block storage, the system comprising a network device and a storage cluster, the storage cluster comprising at least a first storage device, wherein:

[0064] The network device is used to: receive a first write request sent by a computing device, wherein the first write request includes a plurality of data blocks, and each of the plurality of data blocks carries a block identifier; divide the data blocks belonging to a first file among the plurality of data blocks into a first data packet according to metadata; wherein the metadata includes an identifier of the file to which each of the plurality of data blocks belongs; and send a second write request to a first storage device in a storage cluster, wherein the second write request includes the first data packet, and the second write request is used to store the data blocks in the first data packet on the first storage device;

[0065] The first storage device is used to: receive the second write request sent by the network device, and store the data block in the first data packet in the second write request on the first storage device.

[0066] Based on the third aspect, in a possible implementation, the network device is also used to: divide the data blocks belonging to the second file among the multiple data blocks into a second data packet according to the metadata; determine a second storage device for storing the second data packet from the storage cluster; send a third write request to the second storage device, the write request including the second data packet, and the third write request is used to store the data blocks in the second data packet on the second storage device; the second storage device is also used to: receive the third write request sent by the network device, and store the data blocks in the second data packet in the third write request on the second storage device.

[0067] Based on the third aspect, in a possible implementation, the network device is further used to record a first corresponding relationship, where the first corresponding relationship is used to indicate a corresponding relationship between an identifier of the first file and the first storage device.

[0068] Based on the third aspect, in a possible implementation, the network device is also used to: receive a first processing request sent by the computing device, the first processing request including an identifier of the first file and an identifier of a first operation; determine the first storage device storing the first file according to the first correspondence; send a second processing request to the first storage device, the second processing request being used to instruct execution of the first operation on the first file; the first storage device is also used to: execute the first operation on the first file according to the second processing request to obtain the first processing result; send the first processing result to the network device; the network device is also used to: return the first processing result to the computing device.

[0069] Based on the third aspect, in a possible implementation, the second processing request includes an identifier of the first file and an identifier of the first operation; the first storage device is also used to: determine a block identifier of a data block contained in the first file based on the identifier of the first file; obtain the data block of the first file based on the block identifier of the data block contained in the first file and a second correspondence, and perform the first operation on the data block of the first file to obtain the first processing result; the second correspondence is used to indicate a correspondence between the block identifier of each data block in the first data packet and a storage position of each data block in the first data packet in the first storage device.

[0070] Based on the third aspect, in a possible implementation method, the network device is used to record a third correspondence relationship, and the third correspondence relationship is used to indicate the correspondence between the identifier of the first file and the block identifier of each data block in the first data packet; the first storage device is used to send an acquisition request to the network device, and the acquisition request includes the identifier of the first file; the network device is used to determine the block identifier of the data block contained in the first file according to the third correspondence relationship in response to the acquisition request, and send the block identifier of the data block contained in the first file to the first storage device.

[0071] Based on the third aspect, in a possible implementation, the first data packet carries an identifier of the first file, and the first storage device records a third correspondence relationship, wherein the third correspondence relationship is used to indicate the correspondence between the identifier of the first file and the block identifier of each data block in the first data packet; the first storage device is used to determine the block identifiers of the data blocks contained in the first file based on the locally recorded third correspondence relationship and the identifier of the first file.

[0072] Based on the third aspect, in a possible implementation, the second processing request includes a block identifier of each data block in multiple data blocks of the first file and an identifier of the first operation; the block identifier of each data block is determined by the network device based on a recorded third correspondence; the third correspondence is used to indicate the correspondence between the identifier of the first file and the block identifier of each data block in the first data packet; the first storage device is used to: obtain multiple data blocks of the first file according to the second correspondence, and perform the first operation on the data blocks of the first file to obtain the first processing result; the second correspondence is used to indicate the correspondence between the block identifier of each data block in the first data packet and the storage position of each data block in the first data packet in the first storage device.

[0073] Based on the third aspect, in a possible implementation method, the first processing request also includes an identifier of the second file, and the network device is also used to: determine a second storage device that stores the second file based on the first correspondence; send a third processing request to the second storage device, and the third processing request is used to instruct to perform the first operation on the second file; obtain a second processing result corresponding to the third processing request from the second storage device; and summarize the first processing result and the second processing result and return them to the computing device.

[0074] Based on the third aspect, in a possible implementation, the first operation includes any one or more of a read operation, character statistics, character screening, and character sorting.

[0075] In a fourth aspect, the present application provides a network device, the network device comprising:

[0076] A receiving module, configured to receive a first write request sent by a computing device, wherein the first write request includes a plurality of data blocks, and each of the plurality of data blocks carries a block identifier;

[0077] A partitioning module, configured to partition data blocks belonging to a first file among the plurality of data blocks into a first data packet according to metadata; wherein the metadata includes an identifier of a file to which each data block among the plurality of data blocks belongs;

[0078] The sending module is used to send a second write request to a first storage device in the storage cluster, wherein the second write request includes the first data packet, and the second write request is used to store the data block in the first data packet on the first storage device.

[0079] Based on the fourth aspect, in a possible implementation manner, the receiving module is further used to: obtain the metadata from a metadata server.

[0080] Based on the fourth aspect, in a possible implementation, the first write request includes the metadata.

[0081] Based on the fourth aspect, in a possible implementation,

[0082] The division module is further used to divide the data blocks belonging to the second file among the plurality of data blocks into the second data packet according to the metadata;

[0083] A determination module, configured to determine a second storage device for storing the second data packet from the storage cluster;

[0084] The sending module is further used to send a third write request to the second storage device, wherein the write request includes the second data packet, and the third write request is used to store the data block in the second data packet on the second storage device.

[0085] Based on the fourth aspect, in a possible implementation, the network device further includes:

[0086] The recording module is used to record a first corresponding relationship, where the first corresponding relationship is used to indicate a corresponding relationship between an identifier of the first file and the first storage device.

[0087] Based on the fourth aspect, in a possible implementation,

[0088] The receiving module is further configured to receive a first processing request sent by the computing device, wherein the first processing request includes an identifier of the first file and an identifier of a first operation;

[0089] The determining module is further configured to determine, according to the first corresponding relationship, the first storage device storing the first file;

[0090] The sending module is further used to send the second processing request to the first storage device;

[0091] The receiving module is further used to obtain a first processing result corresponding to the second processing request from the first storage device;

[0092] The sending module is further used to return the first processing result to the computing device.

[0093] Based on the fourth aspect, in a possible implementation, the first processing request also includes an identifier of the second file.

[0094] The determining module is further configured to determine, according to the first corresponding relationship, a second storage device storing the second file;

[0095] The sending module is further used to send a third processing request to the second storage device, where the third processing request is used to instruct to perform the first operation on the second file;

[0096] The receiving module is further used to obtain, from the second storage device, a second processing result corresponding to the third processing request;

[0097] The sending module is further used to summarize the first processing result and the second processing result and return them to the computing device.

[0098] Based on the fourth aspect, in a possible implementation, the first operation includes any one or more of a read operation, character statistics, character screening, and character sorting.

[0099] The functional modules of the fourth aspect are used to implement the method described in the first aspect and any possible implementation method of the first aspect.

[0100] In a fifth aspect, the present application provides a storage cluster, the storage cluster comprising at least a first storage device, the first storage device comprising:

[0101] a receiving module, configured to receive a second write request sent by a network device, wherein the second write request includes a first data packet, the first data packet includes a data block belonging to a first file, and the second write request is used to store the data block in the first data packet on the first storage device;

[0102] A storage module is used to store the data block in the first data packet in the second write request on the first storage device.

[0103] Based on the fifth aspect, in a possible implementation, the first storage device also includes: a recording module, used to record a second correspondence relationship, wherein the second correspondence relationship is used to indicate the correspondence between the block identifier of each data block in the first data packet and the storage position of each data block in the first data packet in the first storage device.

[0104] Based on the fifth aspect, in a possible implementation,

[0105] The receiving module is further used to receive a second processing request sent by the network device, where the second processing request is used to instruct to perform the first operation on the first file;

[0106] an execution module, configured to execute the first operation on the first file according to the second processing request to obtain the first processing result;

[0107] A sending module is used to send the first processing result to the network device.

[0108] Based on the fifth aspect, in a possible implementation manner, the second processing request includes an identifier of the first file and an identifier of the first operation;

[0109] The execution module is used for:

[0110] Determine a block identifier of a data block included in the first file according to the identifier of the first file;

[0111] The data blocks of the first file are acquired according to the block identifiers of the data blocks included in the first file and the second corresponding relationship, and the first operation is performed on the data blocks of the first file to obtain the first processing result.

[0112] Based on the fifth aspect, in a possible implementation manner, the network device records a third correspondence relationship, where the third correspondence relationship is used to indicate a correspondence relationship between an identifier of the first file and a block identifier of each data block in the first data packet;

[0113] The sending module is used to send an acquisition request to the network device, where the acquisition request includes an identifier of the first file;

[0114] The receiving module is used to receive the block identifier of the data block included in the first file sent by the network device based on the acquisition request, and the block identifier of the data block included in the first file is determined by the network device according to the third corresponding relationship.

[0115] Based on the fifth aspect, in a possible implementation manner, the first data packet carries an identifier of the first file,

[0116] The recording module is used to record a third corresponding relationship according to the identifier of the first file, where the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet;

[0117] The execution module is used for: the first storage device to determine the block identifier of the data block included in the first file according to the third corresponding relationship recorded locally and the identifier of the first file.

[0118] Based on the fifth aspect, in a possible implementation, the second processing request includes a block identifier of each data block in multiple data blocks of the first file and an identifier of the first operation; the block identifier of each data block is determined by the network device according to a recorded third correspondence; the third correspondence is used to indicate a correspondence between the identifier of the first file and the block identifier of each data block in the first data packet;

[0119] The execution module is used to obtain multiple data blocks of the first file according to the second corresponding relationship, and perform the first operation on the data blocks of the first file to obtain the first processing result.

[0120] Based on the fifth aspect, in a possible implementation, the first operation includes any one or more of a read operation, character statistics, character screening, and character sorting.

[0121] The functional modules of the fifth aspect are used to implement the method described in the second aspect and any possible implementation method of the second aspect.

[0122] In a sixth aspect, the present application provides a network device, comprising a memory and a processor, wherein the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the method described in the first aspect and any possible implementation method of the first aspect.

[0123] In the seventh aspect, the present application provides a storage cluster, comprising at least a first storage device, each of the multiple storage devices comprising a memory and a processor, the memory of each storage device being used to store instructions, and the processor of each storage device being used to execute the instructions stored in the memory, so as to implement the method described in the above-mentioned second aspect and any possible implementation method of the second aspect.

[0124] In an eighth aspect, the present application provides a computer storage medium comprising program instructions, which, when executed by a network device, enables the network device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0125] In a ninth aspect, the present application provides a computer storage medium, comprising program instructions, which, when executed by a storage cluster, enables the storage cluster to execute the method described in the second aspect and any possible implementation of the second aspect.

[0126] In a tenth aspect, the present application provides a computer program product, including program instructions, which, when executed by a computer, enable the computer to execute the method described in the first aspect and any possible implementation of the first aspect. The computer program product may be a software installation package, and when the method provided by any possible design of the first aspect is needed, the computer program product may be downloaded and executed on a computer to implement the method described in the first aspect and any possible implementation of the first aspect.

[0127] In an eleventh aspect, the present application provides a computer program product, including program instructions, which, when executed by a computer, enable the computer to execute the method described in the second aspect and any possible implementation of the second aspect. The computer program product may be a software installation package, and when the method provided by any possible design of the second aspect is needed, the computer program product may be downloaded and executed on a computer to implement the method described in the second aspect and any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0128] Figure 1 A schematic diagram of a system architecture provided for this application;

[0129] Figure 2 A flowchart of a file processing method based on block storage provided in this application;

[0130] Figure 3 An example diagram provided for this application;

[0131] Figure 4 A flowchart of a file processing method based on block storage provided in this application;

[0132] Figure 5 A flowchart of a file processing method based on block storage is also provided for this application;

[0133] Figure 6 A schematic diagram of a scenario provided for this application;

[0134] Figure 7 A schematic diagram of a scenario provided for this application;

[0135] Figure 8 A schematic diagram of the structure of a network device provided for this application;

[0136] Fig. 9 A schematic diagram of the device structure of a storage device provided in this application;

[0137] Fig.10 A schematic diagram of the structure of a hardware device provided for this application;

[0138] Fig.11 A schematic diagram of the structure of a storage cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0139] This application provides a system, see Figure 1 , Figure 1 A schematic diagram of the architecture of a system 100 provided in the present application, wherein the system 100 involves a computing side, a network side, and a storage side. The system 100 includes at least one computing device 110, a network device 120, and a storage cluster 130, wherein the storage cluster 130 includes multiple storage devices 131.

[0140] The at least one computing device 110, the network device 120, and the multiple storage devices 131 may be located in the same data center, or in multiple data centers connected to each other. In the case where the at least one computing device 110, the network device 120, and the multiple storage devices 131 are located in multiple data centers connected to each other, any one or more of the at least one computing device 110, the network device 120, and the multiple storage devices 131 may be located in any one of the multiple data centers.

[0141] The computing device 110 may be a mobile terminal, a desktop computer, a server or other electronic device used for computing. In application scenarios such as big data analysis, high performance computing (HPC), and artificial intelligence, the computing device 110 is used to perform large-scale computing. Optionally, a network card may be provided on the computing device 110, and the computing device 110 sends the data obtained by computing to the network device 120 through the network card.

[0142] The network device 120 may be, for example, a router, a switch, or other device. The network device 120 is used to receive data or a request sent by at least one computing device 110, process the data or the request, and send the processed data or the request to one or more storage devices 131 in the storage cluster 130; the network device 120 is also used to receive data returned by the storage device 131 in the storage cluster 130, and send the data to the computing device 110, or process the data and send the processed data to the computing device 110.

[0143] The storage device 131 is used to receive data or requests sent by the network device 120 and process the data or requests.

[0144] In the present application, the network device 120 has computing and request processing functions in addition to the routing and forwarding functions. Optionally, the network device 120 may be provided with a data processing unit (DPU), which is used to compute data and process and distribute requests or instructions.

[0145] In the present application, the storage device 131 has a data processing function. Optionally, a DPU may be provided on the storage device 131, and the DPU is used to perform some near data processing operations, including but not limited to compression, decompression, statistics, screening, etc. For the specific functions of the network device 120 and the storage device 131, reference may be made to the specific description of the method embodiment below, which will not be described in detail here.

[0146] Figure 1 The system architecture shown can be applied to a storage area network (SAN) scenario. In a SAN storage scenario, a file system is provided on the computing device 110. When the computing device 110 sends a file to be stored to the storage device 131, where the file to be stored may include one file or multiple files, the computing device 110 divides each file in the file to be stored into multiple data blocks, and then sends a write request to the storage device 131, where the write request includes the multiple data blocks after the division, where each data block carries a block identifier. The file system maintains a mapping relationship between each file and the multiple data blocks after the file is divided.

[0147] Based on the above system architecture, this application provides a file processing method based on block storage, see Figure 2 , Figure 2 A flowchart of a file processing method based on block storage provided in the present application, the method includes but is not limited to the description of the following content.

[0148] S101. A computing device sends a first write request to a network device. Correspondingly, the network device receives the first write request sent by the computing device. The first write request includes multiple data blocks, and each of the multiple data blocks carries a block identifier.

[0149] The computing device sends a first write request to the network device, and the network device receives the first write request sent by the computing device. The first write request includes multiple data blocks, and each data block carries a block identifier. It can be understood that when the computing device sends one or more files to the network device, the computing device has broken up each of the one or more files into multiple data blocks. Therefore, the first write request includes multiple data blocks, and these multiple data blocks correspond to one or more files.

[0150] It can be understood that the multiple data blocks are obtained by dividing one or more files by a computing device.

[0151] For a system composed of multiple computing devices, network devices, and a storage cluster, the network device may receive multiple write requests sent by multiple computing devices. The embodiment of the present application is described by taking the first write request as an example.

[0152] S102: The network device obtains metadata.

[0153] The metadata includes an identifier of a file to which each of the multiple data blocks included in the first request belongs, wherein the identifier of the file may be a file name, or other characters used to represent the file, which is not limited in this application.

[0154] In one implementation, the computing device stores the file (data) and metadata together. When the computing device sends a first write request to the network device, the metadata may be carried in the first write request, that is, the first write request includes multiple data blocks and metadata.

[0155] In another implementation, the computing device stores the file (data) and metadata separately, and the network device can obtain the metadata from the metadata server, where the metadata server refers to a server for storing metadata. Here, the metadata server is a logical concept. In actual applications, the metadata server and the computing device may be the same physical device or different physical devices. Regardless of whether the metadata server and the computing device are the same physical device, the metadata is obtained separately and is not carried in the first write request.

[0156] When the network device obtains metadata from the metadata server, the network device can obtain all metadata from the metadata server in advance, and then, after receiving the first write request, filter out the metadata of the file to which the multiple data blocks included in the first write request belong; the network device can also obtain the metadata of the file to which the multiple data blocks included in the first write request belong from the metadata server according to demand after receiving the first write request.

[0157] S103: The network device divides the data blocks belonging to the first file from the multiple data blocks into a first data packet according to the metadata, and the first data packet carries an identifier of the first file.

[0158] The network device divides the multiple data blocks included in the first write request according to the metadata, and divides the data blocks belonging to the first file into a first data packet, and the first data packet carries a packet identifier, which is an identifier of the first file. In this way, at least one data block belonging to the first file is all divided into the first data packet, that is, all the data blocks included in the first data packet belong to the first file.

[0159] Optionally, the multiple data blocks may also include data blocks belonging to a second file, and the network device divides the data blocks belonging to the second file from the multiple data blocks into a second data packet according to the metadata, and the second data packet carries a packet identifier, which is an identifier of the second file. Through this step, the data blocks belonging to a file from the multiple data blocks are divided into a data packet, and each data packet carries a packet identifier, which is an identifier of the corresponding file.

[0160] S104: The network device sends a second write request to a first storage device in the storage cluster, where the second write request includes the first data packet, and the second write request is used to store the data block in the first data packet on the first storage device.

[0161] In the present application, a data packet is sent to a storage device in a storage cluster, that is, at least one data block corresponding to a file is sent and stored in a storage device, so as to facilitate the subsequent processing of file-level semantic problems. File-level semantic problems refer to problems that need to be processed based on the semantic information of the entire file. The entire file must be obtained to process file-level semantic problems. Only based on part of the file, the semantic information of the file cannot be obtained, and file-level semantic problems cannot be processed.

[0162] The network device generates a second write request, wherein the second write request includes a first data packet, the first data packet carries an identifier of the first file, and the second write request is used to store the data block in the first data packet on the first storage device. The network device sends the second write request to the first storage device.

[0163] Optionally, in the case where multiple data blocks include data blocks belonging to the second file, the network device divides the data blocks belonging to the second file into a second data packet. The network device generates a third write request, the third write request includes a second data packet, the second data packet carries an identifier of the second file, the third write request is used to store the data blocks in the second data packet on a second storage device, and the network device sends the third write request to the second storage device. It should be noted that the second storage device and the first storage device here can be the same storage device or different storage devices, that is, the network device can send the third write request to the first storage device or the second storage device, and the present application only needs to ensure that a data packet consisting of data blocks belonging to one file is sent to one storage device.

[0164] Optionally, the network device may first determine the load status of each storage device in the storage cluster, the load status of the storage device including a high load status and a low load status, and then determine to which storage device the second write request and / or the third write request is sent according to the load status of each storage device. When the storage device is in a low load status, the second write request and / or the third write request is preferentially sent to the storage device in the low load status.

[0165] Among them, the method for the network device to determine the load status of the storage device can be that a monitoring module is set in the network device, and the monitoring module is used to monitor the load status of each storage device. For example, the monitoring module can be used to count the number of requests processed by each storage device within a certain period of time. If the number of requests processed within a certain period of time exceeds a set threshold, it is determined that the storage device is in a high-load state. If the number of requests processed within a certain period of time does not exceed the set threshold, it is determined that the storage device is in a low-load state; or, the number of requests processed by each storage device within a certain period of time is arranged from small to large, and the first m storage devices with a small number of processed requests are selected, and the first m storage devices are storage devices in a low-load state.

[0166] The network device records a first correspondence, which is used to indicate the correspondence between the identifier of the first file and the first storage device storing the first data packet. The first correspondence can be understood as the correspondence between the first file (or the identifier of the first file) and the first storage device storing the first file. Subsequently, the network device can determine which storage device stores the first file based on the first correspondence.

[0167] The network device also records a third correspondence, which is used to indicate the correspondence between the identifier of the first file and the block identifier of at least one data block included in the first data packet. The third correspondence can be understood as the correspondence between the first file and at least one data block belonging to the first file. Subsequently, the network device can determine the block identifier of at least one data block corresponding to the identifier of the first file according to the third correspondence.

[0168] For example, see Figure 3 , Figure 3 An example diagram provided for the present application. A network device receives a first write request sent by a computing device, the first write request includes data blocks block1, block2, block3, block4, block5, and block6, each of which carries a block identifier. The network device divides the data blocks belonging to the first file into a first data packet according to metadata, obtains the first data packet, and names the packet identifier of the first data packet as file1, file1 is the identifier of the first file, specifically, file1 can be the file name of the first file; divides the data blocks belonging to the second file into a second data packet, obtains the second data packet, and names the packet identifier of the second data packet as file2, file2 is the identifier of the second file, specifically, file2 can be the file name of the second file.

[0169] The network device generates a second write request, the second write request includes a first data packet, the first data packet carries a packet identifier file1, and the network device sends the second write request to storage device a. The network device also generates a third write request, the third write request includes a second data packet, the second data packet carries a packet identifier file2, and the network device sends the third write request to storage device c. In this example, the block identifier, the file identifier, and the name of the storage device are only used for example. In actual applications, the block identifier, the file identifier, and the name of the storage device can all be in other forms.

[0170] S105: The first storage device stores each data block in the first data packet in the second write request on the first storage device.

[0171] The first storage device receives a second write request sent by the network device, the second write request includes a first data packet, and the first data packet carries an identifier of the first file. The first storage device performs a write operation on at least one data block in the first data packet according to the second write request, and writes the at least one data block into the first storage device.

[0172] In one implementation, the first storage device records the second correspondence and the third correspondence, wherein the third correspondence is used to indicate the correspondence between the identifier of the first file and the block identifier of each data block in the first data packet, and the second correspondence is used to indicate the correspondence between the block identifier of each data block in the first data packet and the storage position of each data block in the first data packet in the first storage device. Subsequently, the first storage device can determine the block identifier of at least one data block corresponding to the identifier of the file based on the identifier of the file and the third correspondence here, and then determine the storage position of at least one data block based on the block identifier of at least one data block and the second correspondence here, so that the first storage device can read the file corresponding to the identifier of the file from the storage medium based on the storage position.

[0173] In another implementation, the first storage device only records the second correspondence, and the second correspondence is used to indicate the correspondence between the block identifier of each data block in the first data packet and the storage position of each data block in the first data packet in the first storage device. Subsequently, the first storage device can determine the storage position of at least one data block according to the block identifier of at least one data block and the second correspondence here, so that the first storage device reads the file corresponding to the file identifier according to the storage position.

[0174] Optionally, when the network device sends the third write request to the second storage device in the storage cluster, the second storage device receives the third write request and stores each data block in the second data packet in the third write request on the second storage device according to the third write request.

[0175] It should be noted that, in this application, the first data packet, the second data packet, the first storage device, and the second storage device are used as examples for introduction and description. When the multiple data blocks received by the network device also include data blocks belonging to the third file...the nth file, the network device divides the data blocks belonging to the third file into the third data packet and generates a fourth write request,...divides the data blocks belonging to the nth file into the nth data packet and generates an n+1th write request, and then sends the fourth write request...the n+1th write request to a storage device in the storage cluster respectively. After receiving the write request, the corresponding storage device stores each data block in the data packet included in the write request on its own storage device.

[0176] It can be seen that the network device divides the multiple data blocks in the first write request sent by the computing device into one or more data packets according to the metadata, so that the data blocks belonging to one file are divided into one data packet. Therefore, each data packet obtained corresponds to a file, and different data packets correspond to different files. The network device sends a data packet to a storage device, thereby storing multiple data blocks belonging to one file on one storage device. Compared with the network device breaking up and storing multiple data blocks belonging to one file on multiple storage devices, the block storage method provided by the present application solution facilitates the realization of near data processing problems and file-level semantic problems.

[0177] exist Figure 2 Based on the file processing method described above, the present application also provides a file processing method based on block storage, see Figure 4 , Figure 4 A flowchart of a file processing method based on block storage provided in the present application, the method includes but is not limited to the description of the following content.

[0178] S201. The computing device sends a first processing request to the network device. Correspondingly, the network device receives the first processing request sent by the computing device. The first processing request includes an identifier of a first file and an identifier of a first operation.

[0179] The computing device sends a first processing request to the network device, and the network device receives the first processing request sent by the computing device, wherein the first processing request includes an identifier of the first file and an identifier of the first operation, and the first processing request is used to instruct to perform the first operation on the first file. The identifier of the first file may be the file name of the first file, for example, the identifier of the first file may be file1, where file1 is the file name of the first file, and the identifier of the first file may also be other identifier characters of the first file, which is not limited in this application.

[0180] The first operation may include any one or more of a read operation, character counting, character filtering, and character sorting. For example, the first operation may be a read operation, and the first processing request is used to instruct to perform a read operation on the first file; for another example, the first operation may be to count the number of "character B", and the first processing request is used to instruct to count the number of "character B" in the first file; for another example, the first operation may be to filter out the "target character", and the first processing request is used to instruct to filter out the "target character" from the first file; for another example, the first operation may be to sort the contents of the target area in a certain order, and the first processing request is used to instruct to sort the contents of the target area in the first file in a certain order; and so on.

[0181] The identifier of the first operation may be any character used to represent the first operation. For example, the identifier of the read operation may be "R", the identifier of the character counting operation may be "T", the identifier of the character screening operation may be "S1", and the identifier of the character sorting operation may be "S2". Here, the identifiers of each operation are only used for example and do not constitute a limitation on the present application.

[0182] S202: The network device determines that a storage device storing a first file is a first storage device.

[0183] The network device records the correspondence between the identifier of the first file and the storage device storing the first file (first correspondence). Therefore, the network device determines that the storage device storing the first file is the first storage device based on the first correspondence and the identifier of the first file carried in the first processing request.

[0184] It can be known from the above method embodiment that the network device also records the correspondence relationship (third correspondence relationship) between the identifier of the first file and the block identifier of at least one data block belonging to the first file.

[0185] S203: The network device sends the second processing request to the first storage device.

[0186] The second processing request is used to instruct to perform a first operation on the first file.

[0187] In one implementation, the second processing request includes an identifier of the first file and an identifier of the first operation, and the network device sends the second processing request to the first storage device.

[0188] In another implementation, the second processing request includes a block identifier of each of the multiple data blocks belonging to the first file and an identifier of the first operation, wherein the block identifiers of the multiple data blocks belonging to the first file are determined by the network device according to a third corresponding relationship recorded locally, wherein the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the block identifier of each data block belonging to the first file. The network device sends the second processing request to the first storage device.

[0189] S204: The first storage device performs a first operation on the first file according to the second processing request to obtain a first processing result.

[0190] First, the first storage device obtains the first file according to the second processing request.

[0191] In the case where the second processing request includes the identifier of the first file, in one implementation, the first storage device records the second correspondence and the third correspondence, wherein the third correspondence is used to indicate the correspondence between the identifier of the first file and the block identifier of the data block belonging to the first file, and the second correspondence is used to indicate the correspondence between the block identifier of each data block and the storage position of each data block in the first storage device. The first storage device obtains the storage position of each data block corresponding to the first file according to the identifier of the first file carried in the second processing request, the locally recorded third correspondence, and the second correspondence, thereby reading the first file.

[0192] In another implementation, the first storage device only records the second correspondence, that is, only records the correspondence between the block identifier of each data block and the storage position of each data block in the first storage device. In this case, the first storage device can send a request for acquisition to the network device, the request for acquisition includes the identifier of the first file, and the request for acquisition is used to instruct the acquisition of the block identifiers of each data block corresponding to the identifier of the first file; after receiving the acquisition request, the network device queries the block identifier of at least one data block corresponding to the identifier of the first file according to the third correspondence recorded in the network device, and sends the block identifier of at least one data block corresponding to the identifier of the first file to the first storage device, so that the first storage device obtains the block identifier of at least one data block corresponding to the identifier of the first file. The first storage device then obtains the storage position of each data block corresponding to the first file according to the second correspondence recorded in the first storage device, thereby reading the first file.

[0193] When the second processing request includes the block identifier of each data block in the multiple data blocks of the first file, the first storage device directly obtains the storage location of each data block corresponding to the first file according to the second corresponding relationship recorded locally, thereby reading the first file.

[0194] Then, after obtaining the first file, the first storage device performs a first operation on the first file to obtain a first processing result.

[0195] For example, the first processing request includes the identifier of the first file file 1 and the identifier of the first operation. If the identifier of the first operation is "R", where "R" represents a read operation, the first storage device performs a read operation on the first file; if the identifier of the first operation is "T, character B", where "T" represents a character counting operation, the first storage device counts the number of occurrences of "character B" in the first file; if the identifier of the first operation is "S1, target character", where "S1" represents a character filtering operation, the first storage device filters the "target character" in the first file; if the identifier of the first operation is "S2, target area", where "S2" represents a character sorting operation, the first storage device sorts the content of the "target area" in the first file. The identifier of the first operation in this example is only an example. In actual applications, the identifier of the first operation can be represented by other forms, and the identifier of the first operation can include more or less identification information. This example does not constitute a limitation on the present application.

[0196] S205. The first storage device sends the first processing result to the network device. Correspondingly, the network device receives the first processing result sent by the first storage device.

[0197] S206: The network device sends the first processing result to the computing device.

[0198] The first storage device sends the first processing result to the network device. The network device receives the first processing result and sends the first processing result to the computing device.

[0199] It can be seen that this embodiment provides a file processing method. When the computing device needs to process a certain file, taking the processing of the first file as an example, the computing device sends a first processing request, and the first processing request includes an identifier of the first file and an identifier of the first operation. The network device determines that the storage device storing the first file is the first storage device based on the second correspondence recorded, and sends a second processing request to the first storage device. The second processing request may include an identifier of the first file and an identifier of the first operation, or the second processing request includes a block identifier of each data block in multiple data blocks of the first file and an identifier of the first operation. Then, the first storage device performs the first operation on the first file according to the second processing request, obtains a first processing result, and realizes near data processing. The first storage device only needs to return the first processing result to the computing device.

[0200] Compared to reading the entire first file from multiple storage devices and then sending it to a computing device, which then processes the first file, the method described in the present application reduces bandwidth consumption and achieves efficient computing for near-data processing. At the same time, the method described in the present application facilitates processing file-level semantic issues.

[0201] In the above Figure 2 Based on the method embodiment, the present application also provides a file processing method based on block storage. Figure 5 , Figure 6 and Figure 7 , Figure 5 The present application also provides a flowchart of a file processing method based on block storage. Figure 6 and Figure 7 They are schematic diagrams of a scenario provided by this application, and the following is combined with Figure 5 , Figure 6 and Figure 7 The present invention provides a block storage-based file processing method, which includes but is not limited to the following description.

[0202] S301. The computing device sends a first processing request to the network device. Correspondingly, the network device receives the first processing request sent by the computing device. The first processing request includes an identifier of a first file, an identifier of a second file, and an identifier of a first operation.

[0203] This step and Figure 4 Step S201 in the method embodiment is similar, and can be referred to Figure 4 For the sake of brevity, the description of the content of step S201 in the method embodiment will not be repeated here.

[0204] The difference between this step and step S201 is that, in this embodiment, the first processing request includes not only the identifier of the first file and the identifier of the first operation, but also the identifier of the second file. That is, this embodiment processes multiple files; Figure 4 The method embodiment processes a file.

[0205] This embodiment is introduced by taking the example that the first processing request includes identifiers of two files. In actual applications, the first processing request may also include identifiers of a larger number of files, which is not limited in this application.

[0206] S302: The network device determines, according to the first corresponding relationship, that a storage device storing the first file is the first storage device, and determines that a storage device storing the second file is the second storage device.

[0207] The network device records the correspondence between the identifier of the first file and the storage device storing the first file, and the correspondence between the identifier of the second file and the storage device storing the second file (first correspondence). Therefore, the network device determines that the storage device storing the first file is the first storage device, and determines that the storage device storing the second file is the second storage device based on the first correspondence.

[0208] Optionally, in one implementation, the first storage device and the second storage device are different devices in the storage cluster, that is, the first file and the second file are stored on different storage devices. In another implementation, the first storage device and the second storage device may be the same storage device in the storage cluster, in which case, the first file and the second file are stored on the same storage device.

[0209] By the above Figure 2 It can be known from the method embodiment that the network device also records the correspondence between the identifier of the first file and the block identifier of at least one data block belonging to the first file, and the correspondence between the identifier of the second file and the block identifier of at least one data block belonging to the second file (third correspondence).

[0210] S303: The network device generates a second processing request and a third processing request.

[0211] The second processing request includes the identifier of the first file and the identifier of the first operation, and the second processing request is used to instruct to perform the first operation on the first file. The third processing request includes the identifier of the second file and the identifier of the first operation, and the third processing request is used to instruct to perform the first operation on the second file. For example, if the first processing request includes the identifier of the first file file1, the identifier of the second file file2, and the identifier R of the first operation, the network device generates a second processing request and a third processing request, the second processing request includes the identifier of the first file file1 and the identifier R of the first operation, and the third processing request includes the identifier of the second file file2 and the identifier R of the first operation.

[0212] S304a: The network device sends the second processing request to the first storage device. Correspondingly, the first storage device receives the second processing request sent by the network device.

[0213] S304b: The network device sends the third processing request to the second storage device. Correspondingly, the second storage device receives the third processing request sent by the network device.

[0214] For example, in Figure 6 In the case where the network device determines that the first file is stored in storage device a and that the second file is stored in storage device c, the network device sends the generated second processing request to storage device a and sends the generated third processing request to storage device c.

[0215] S305a: The first storage device performs a first operation on the first file according to the second processing request to obtain a first processing result.

[0216] S305b: The second storage device performs the first operation on the second file according to the third processing request to obtain a second processing result.

[0217] Step S305a, step S305b and reference Figure 4 Step S204 in the method embodiment is similar, and the specific content can be referred to Figure 4 For the sake of brevity, the description of the content of step S204 in the method embodiment will not be repeated here.

[0218] For example, in Figure 6 In the figure, storage device a obtains the data blocks corresponding to the first file according to the identifier of the first file, and the data blocks corresponding to the first file include block1, block2 and block3, and then performs the first operation on the first file to obtain the first processing result (the first processing result is not shown in the figure). Storage device c obtains the data blocks corresponding to the second file according to the identifier of the second file, and the data blocks corresponding to the second file include block4, block5 and block6, and then performs the first operation on the second file to obtain the second processing result (the second processing result is not shown in the figure).

[0219] S306a. The first storage device sends the first processing result to the network device. Correspondingly, the network device receives the first processing result sent by the first storage device.

[0220] S306b. The second storage device sends the second processing result to the network device. Correspondingly, the network device receives the second processing result sent by the second storage device.

[0221] like Figure 7 As shown, after the storage device a obtains the first processing result, it sends the first processing result to the network device, and after the storage device c obtains the second processing result, it sends the second processing result to the network device.

[0222] S307: The network device aggregates the first processing result and the second processing result to obtain a final result.

[0223] like Figure 7 As shown, the network device aggregates the first processing result and the second processing result to obtain a final result. The aggregate processing may be to sum the first processing result and the second processing result to obtain the final result, and the network device sends the final result to the computing device; or it may be to merge the first processing result and the second processing result, and then send it to the computing device through a response message; the aggregate processing may also be other processing methods, and the network device may determine the corresponding aggregate processing method according to the identifier of the first operation, which is not limited in this application.

[0224] For example, the first processing request includes the identifier file1 of the first file, the identifier file2 of the second file, and the identifier "R" of the first operation. The network device generates a second processing request and a third processing request according to the first processing request. The second processing request includes the identifier file1 of the first file and the identifier "R" of the first operation, and the third processing request includes the identifier file2 of the second file and the identifier "R" of the first operation. The network device sends the second processing request to the first storage device and sends the third processing request to the second storage device. The first storage device performs a read operation on the first file according to the second processing request, and sends the read first file (first processing result) to the network device. The second storage device performs a read operation on the second file according to the third processing request, and sends the read second file (second processing result) to the network device. The network device sends the first file (first processing result) and the second file (second processing result) together to the computing device via a response message.

[0225] For another example, the first processing request includes the identifier file1 of the first file, the identifier file2 of the second file, and the identifier "T, character B" of the first operation. The network device generates a second processing request and a third processing request according to the first processing request. The second processing request includes the identifier file1 of the first file and the identifier "T, character B" of the first operation, and the third processing request includes the identifier file2 of the second file and the identifier "T, character B" of the first operation. The network device sends the second processing request to the first storage device and sends the third processing request to the second storage device. The first storage device generates a second processing request and a third processing request according to the third processing request. The first processing device counts the number of occurrences of "character B" in the first file to obtain statistical result 1 (first processing result), and sends statistical result 1 (first processing result) to the network device; the second storage device counts the number of occurrences of "character B" in the second file according to the third processing request to obtain statistical result 2 (second processing result), and sends statistical result 2 (second processing result) to the network device; the network device summarizes statistical result 1 (first processing result) and statistical result 2 (second processing result), that is, adds statistical result 1 to statistical result 2 to obtain a final result, and sends the final result to the computing device through a response message.

[0226] For another example, the first processing request includes the identifier file1 of the first file, the identifier file2 of the second file, and the identifier "S1, target character" of the first operation. The network device generates a second processing request and a third processing request according to the first processing request. The second processing request includes the identifier file1 of the first file and the identifier "S1, target character" of the first operation, and the third processing request includes the identifier file2 of the second file and the identifier "S1, target character" of the first operation; the network device sends the second processing request to the first storage device, and sends the third processing request to the second storage device; the first storage device filters out the "target character" from the first file according to the second processing request, obtains filtering result 1 (first processing result), and sends the filtering result 1 (first processing result) to the network device; the second storage device filters out the "target character" from the second file according to the third processing request, obtains filtering result 2 (second processing result), and sends the filtering result 2 (second processing result) to the network device; the network device merges the filtering result 1 (first processing result) and the filtering result 2 (second processing result) together to obtain a final result, and sends the final result to the computing device via a response message.

[0227] S308: The network device sends the final result to the computing device.

[0228] The network device sends the final result to the computing device, and the computing device receives the final result.

[0229] It can be seen that the computing device needs to process multiple files at the same time. Taking the processing of two files at the same time as an example, the computing device sends a first processing request, which may include the identifier of the first file, the identifier of the second file, and the identifier of the first operation. The network device receives the first processing request and determines that the first storage device stores the first file and the second storage device stores the second file. Then the network device sends a second processing request to the first storage device and sends a third processing request to the second storage device. The second processing request includes the identifier of the first file and the identifier of the first operation, and the third processing request includes the identifier of the second file and the identifier of the first operation. Then the first storage device performs the first operation on the first file according to the second processing request, obtains the first processing result, and realizes the near data processing. Similarly, the second storage device performs the first operation on the second file according to the third processing request, obtains the second processing result, and realizes the near data processing. The network device obtains the first processing result from the first storage device and the second processing result from the second storage device, and then summarizes the first processing result and the second processing result and returns them to the computing device.

[0230] In the present application, there is no need to read the first file and the second file separately to the computing device side, and the computing device side performs calculations based on the first file and the second file. Instead, the first storage device and the second storage device directly perform near data processing, and only the final result needs to be returned to the computing device, which reduces bandwidth consumption and realizes efficient calculation of near data processing. At the same time, the method described in the present application facilitates the processing of file-level semantic problems.

[0231] In addition, it should be noted that the present application realizes the processing and calculation of two files through one processing request, but in actual applications, the processing and calculation of a larger number of files can be realized through one processing request. Compared with the computing device sending multiple read requests, reading multiple files to the computing device through multiple read requests, and the computing device calculating the multiple files, the method described in the present application realizes that multiple storage devices perform near data processing and calculation at the same time, and summarizes the results of multiple near data processing through network devices and returns them to the computing device, which greatly reduces the consumption of bandwidth, realizes efficient calculation, and improves the efficiency of file processing.

[0232] exist Figure 2 Based on the method embodiment, the present application also provides a file processing method based on block storage, which is summarized below.

[0233] The computing device sends a first processing request to the network device, wherein the first processing request includes a block identifier of a first data block and an identifier of a first operation, wherein the first data block is a data block in a first file, and the first processing request may be a read request, i.e., the first operation is a read operation, i.e., the first processing request is used to read the content of the first data block.

[0234] The network device receives the first processing request sent by the computing device, determines that the first data block belongs to the first file based on the third correspondence recorded locally and the block identifier of the first data block, and determines the first storage device that stores the first file based on the first correspondence recorded locally and the identifier of the first file, that is, the first data block is stored on the first storage device.

[0235] The network device sends a second processing request to the first storage device, the second processing request is used to read the content of the first file. After receiving the second processing request, the first storage device performs a first operation on the first file according to the second processing request, reads multiple data blocks contained in the first file, and returns the read multiple data blocks of the first file to the network device. The network device selects a first data block from the multiple data blocks of the first file according to the block identifier of the first data block, returns the first data block to the computing device, and the computing device obtains the first data block.

[0236] Optionally, in one implementation, the second processing request includes an identifier of the first file and an identifier of the first operation. In another implementation, the second processing request includes a block identifier of each data block in multiple data blocks of the first file and an identifier of the first operation.

[0237] The way in which the first storage device obtains the first file according to the second processing request may be based on the second correspondence and the third correspondence recorded locally, or may be by obtaining the block identifiers of multiple data blocks of the first file from the network device. The first storage device obtains the first file based on the block identifiers of the multiple data blocks of the first file and the second correspondence recorded locally.

[0238] Optionally, in this embodiment, the second processing request may also only include the block identifier of the first data block and the identifier of the first operation. After receiving the second processing request, the first storage device directly reads the first data block according to the second corresponding relationship recorded locally and the block identifier of the first data block, and sends the first data block to the network device. The network device then returns the first data block to the computing device, and the computing device obtains the first data block.

[0239] It should be noted that this embodiment is only an exemplary implementation and should not constitute a limitation on other embodiments of the present application.

[0240] The method embodiments are introduced above. Based on the above method embodiments, the present application also provides device embodiments corresponding to the method embodiments.

[0241] See also Figure 8 , Figure 8 A schematic diagram of the structure of a network device 600 provided in the present application, the network device 600 includes:

[0242] A receiving module 610 is configured to receive a first write request sent by a computing device, wherein the first write request includes a plurality of data blocks, and each of the plurality of data blocks carries a block identifier;

[0243] A division module 620 is used to divide the data blocks belonging to the first file from among the multiple data blocks into the first data packet according to the metadata; wherein the metadata includes an identifier of the file to which each of the multiple data blocks belongs;

[0244] The sending module 630 is used to send a second write request to the first storage device in the storage cluster, where the second write request includes the first data packet and is used to store the data block in the first data packet on the first storage device.

[0245] In a possible implementation, the receiving module 610 is further configured to: obtain metadata from a metadata server.

[0246] In a possible implementation, the first write request includes metadata.

[0247] In a possible implementation,

[0248] The division module 620 is further used to divide the data blocks belonging to the second file from the plurality of data blocks into the second data packet according to the metadata;

[0249] A determination module 640 is configured to determine a second storage device for storing a second data packet from the storage cluster;

[0250] The sending module 630 is further used to send a third write request to the second storage device, where the write request includes the second data packet, and the third write request is used to store the data block in the second data packet on the second storage device.

[0251] In a possible implementation, the network device 600 further includes:

[0252] The recording module 650 is used for a first corresponding relationship, where the first corresponding relationship is used to indicate a corresponding relationship between an identifier of a first file and a first storage device.

[0253] In a possible implementation,

[0254] The receiving module 610 is further configured to receive a first processing request sent by a computing device, wherein the first processing request includes an identifier of a first file and an identifier of a first operation;

[0255] The determination module 640 is further configured to determine, according to the first corresponding relationship, a first storage device storing the first file;

[0256] The sending module 630 is further configured to send a second processing request to the first storage device;

[0257] The receiving module 610 is further configured to obtain, from the first storage device, a first processing result corresponding to the second processing request;

[0258] The sending module 630 is further configured to return the first processing result to the computing device.

[0259] In a possible implementation, the first processing request also includes an identifier of the second file.

[0260] The determination module 640 is further configured to determine, according to the first corresponding relationship, a second storage device storing the second file;

[0261] The sending module 630 is further used to send a third processing request to the second storage device, where the third processing request is used to instruct to perform the first operation on the second file;

[0262] The receiving module 610 is further configured to obtain, from the second storage device, a second processing result corresponding to the third processing request;

[0263] The sending module 630 is further used to summarize the first processing result and the second processing result and return them to the computing device.

[0264] In a possible implementation, the first operation includes any one or more of a read operation, character statistics, character screening, and character sorting.

[0265] The various functional modules in the network device 600 are used to implement the method described on the network device side in any of the above method embodiments. For details, please refer to the description of the above method embodiments. For the sake of brevity of the specification, they will not be repeated here.

[0266] Each functional module in the network device 600 can be implemented by software or hardware. The division of each functional module is only an example. In actual applications, the network device can be divided into more or fewer functional modules, which is not limited in this application.

[0267] In some possible embodiments, the above network device may be implemented as a virtualized device.

[0268] For example, a virtualized device may be a virtual machine (VM) running a program for sending a message, and the virtual machine is deployed on a hardware device (e.g., a physical host or a physical server). A virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. A virtual machine may be configured as a network device. For example, a network device may be implemented based on a general physical server combined with network functions virtualization (NFV) technology.

[0269] For example, a virtualized device may be a container, which is an entity used to provide an isolated virtualized environment. For example, the container may be a docker container. A container may be configured as a network device. For example, a network device may be created through a corresponding image. For example, a container instance proxy-container1 may be created for a proxy-container through an image of a proxy-container (a container that provides proxy services), and the container instance proxy-container1 may be provided as a network device. When implemented using container technology, a network device may run using the kernel of a physical machine, and multiple network devices may share the operating system of a physical machine. Different network devices may be isolated through container technology. Containerized network devices may run in a virtualized environment, such as a virtual machine, or they may run directly in a physical machine.

[0270] For example, a virtualized device can be a Pod, which is the basic unit for Kubernetes (Kubernetes is a container orchestration engine open sourced by Google, referred to as K8s in English) to deploy, manage, and orchestrate containerized applications. A Pod may include one or more containers. Each container in the same Pod is usually deployed on the same host, so each container in the same Pod can communicate through the host and can share the storage resources and network resources of the host. A Pod can be configured as a network device. For example, specifically, a container as a service (CaaS, a container-based PaaS service) can be instructed to create a Pod and provide the Pod as a network device.

[0271] Of course, the network device can also be other virtualized devices, which are not listed here one by one.

[0272] In some possible embodiments, the above-mentioned network device can also be implemented by a general-purpose processor. For example, the general-purpose processor can be in the form of a chip. Specifically, the general-purpose processor that implements the network device includes a processing circuit and an input interface and an output interface that are internally connected and communicated with the processing circuit. The processing circuit is used to perform the message generation step in the above-mentioned various method embodiments through the input interface, the processing circuit is used to perform the receiving step in the above-mentioned various method embodiments through the input interface, and the processing circuit is used to perform the sending step in the above-mentioned various method embodiments through the output interface. Optionally, the general-purpose processor may also include a storage medium, and the processing circuit is used to perform the storage step in the above-mentioned various method embodiments through the storage medium. The storage medium may store instructions executed by the processing circuit, and the processing circuit is used to execute the instructions stored in the storage medium to execute the above-mentioned various method embodiments.

[0273] The present application also provides a schematic diagram of the device structure of a storage device, see Fig. 9 , Fig. 9 A schematic diagram of the device structure of a storage device 700 provided in the present application, the storage device 700 can be configured as the first storage device in the above method embodiment, and can also be configured as the second storage device in the storage cluster, the storage device 700 includes:

[0274] The receiving module 710 is used to receive a second write request sent by the network device, the second write request includes a first data packet, the first data packet includes a data block belonging to the first file, and the second write request is used to store the data block in the first data packet on the first storage device;

[0275] The storage module 720 is configured to store the data block in the first data packet in the second write request on the first storage device.

[0276] In a possible implementation, the first storage device also includes: a recording module 730, which is used to record a second correspondence relationship, where the second correspondence relationship is used to indicate the correspondence between the block identifier of each data block in the first data packet and the storage position of each data block in the first data packet in the first storage device.

[0277] In a possible implementation,

[0278] The receiving module 710 is further used to receive a second processing request sent by the network device, where the second processing request is used to instruct to perform a first operation on the first file;

[0279] An execution module 740 is used to execute a first operation on the first file to obtain a first processing result according to the second processing request;

[0280] The sending module 750 is used to send the first processing result to the network device.

[0281] In a possible implementation, the second processing request includes an identifier of the first file and an identifier of the first operation;

[0282] The execution module 740 is used to:

[0283] Determine a block identifier of a data block included in the first file according to the identifier of the first file;

[0284] The data blocks of the first file are acquired according to the block identifiers of the data blocks included in the first file and the second corresponding relationship, and a first operation is performed on the data blocks of the first file to obtain a first processing result.

[0285] In a possible implementation, the network device records a third correspondence relationship, where the third correspondence relationship is used to indicate a correspondence relationship between an identifier of the first file and a block identifier of each data block in the first data packet;

[0286] The sending module 750 is used to send an acquisition request to the network device, where the acquisition request includes an identifier of the first file;

[0287] The receiving module 710 is used to receive the block identifier of the data block included in the first file sent by the network device based on the acquisition request, and the block identifier of the data block included in the first file is determined by the network device according to the third corresponding relationship.

[0288] In a possible implementation, the first data packet carries an identifier of the first file.

[0289] The recording module 730 is used to record a third corresponding relationship according to the identifier of the first file, where the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet;

[0290] The execution module 740 is used for: the first storage device to determine the block identifier of the data block included in the first file according to the locally recorded third corresponding relationship and the identifier of the first file.

[0291] In a possible implementation, the second processing request includes a block identifier of each data block in the plurality of data blocks of the first file and an identifier of the first operation; the block identifier of each data block is determined by the network device according to the recorded third corresponding relationship; the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet;

[0292] The execution module 740 is used to obtain multiple data blocks of the first file according to the second corresponding relationship, and perform a first operation on the data blocks of the first file to obtain a first processing result.

[0293] In a possible implementation, the first operation includes any one or more of a read operation, character statistics, character screening, and character sorting.

[0294] The various functional modules in the storage device 700 are used to implement the method described on the first storage device side or the second storage device side in any of the above method embodiments. For details, please refer to the description of the above method embodiments. For the sake of brevity of the specification, they will not be repeated here.

[0295] Each functional module in the storage device 700 can be implemented by software or hardware. The division of each functional module is only an example. In actual applications, the network device can be divided into more or fewer functional modules, which is not limited in this application.

[0296] This application also provides a hardware device, see Fig.10 , Fig.10 This is a schematic diagram of the structure of a hardware device 800 provided in the present application. The hardware device 800 can be configured as a network device in a method embodiment, or can be configured as a storage device in a method embodiment, such as a first storage device or a second storage device. The hardware device 800 includes:

[0297] Bus 802, processor 804, memory 806 and communication interface 808. Processor 804, memory 806 and communication interface 808 communicate with each other via bus 802. It should be understood that the present application does not limit the number of processors and memories in hardware device 800.

[0298] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The bus 802 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the hardware device 800 (eg, the memory 806, the processor 804, and the communication interface 808).

[0299] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0300] The memory 806 may include a volatile memory, such as a random access memory (RAM). The processor 804 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0301] The memory 806 stores executable code. When the hardware device 800 is configured as a network device or a network device 600 in the method embodiment, the processor 804 executes the executable code to implement the functions of the aforementioned receiving module 610, the dividing module 620, the sending module 630, the determining module 640, and the recording module 650, thereby implementing the method described on the network side in the file processing method based on block storage. That is, the memory 806 stores instructions for executing the method described on the network side in the file processing method based on block storage.

[0302] When the hardware device 800 is configured as the first storage device or the storage device 700 in the method embodiment, the processor 804 executes the executable code to implement the functions of the aforementioned receiving module 710, storage module 720, recording module 730, execution module 740 and sending module 750, thereby implementing the method described on the first storage device side in the file processing method based on block storage. That is, the memory 806 stores instructions for executing the method described on the first storage device side in the file processing method based on block storage.

[0303] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the hardware device 800 and other devices or a communication network.

[0304] When the hardware device 800 is configured as the first storage device or the storage device 700, the embodiment of the present application further provides a storage cluster, such as Fig.11 As shown, the storage cluster includes at least one hardware device 800. The memory 806 in one or more hardware devices 800 in the storage cluster may store the same instructions for executing the method described on the first storage device side in the block storage-based file processing method.

[0305] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the method described on the first storage device side in the file processing method based on block storage, or executes the method described on the network device side in the file processing method based on block storage.

[0306] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the method described on the first storage device side in the file processing method based on block storage, or to execute the method described on the network device side in the file processing method based on block storage.

[0307] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file processing method based on block storage, characterized in that: The method comprises: The network device receives a first write request sent by the computing device, wherein the first write request includes a plurality of data blocks, and each of the plurality of data blocks carries a block identifier; The network device divides the data blocks belonging to the first file among the multiple data blocks into the first data packet according to the metadata; wherein the metadata includes an identifier of the file to which each data block among the multiple data blocks belongs; The network device sends a second write request to a first storage device in the storage cluster, the second write request includes the first data packet, and the second write request is used to store the data block in the first data packet on the first storage device.

2. The method according to claim 1, characterized in that Before the network device divides the data blocks belonging to the first file from the plurality of data blocks into the first data packet according to the metadata, the method further includes: The network device obtains the metadata from a metadata server.

3. The method according to claim 1, characterized in that The first write request includes the metadata.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The network device divides the data blocks belonging to the second file among the plurality of data blocks into a second data packet according to the metadata; The network device determines, from the storage cluster, a second storage device for storing the second data packet; The network device sends a third write request to the second storage device, wherein the third write request includes the second data packet, and the third write request is used to store the data block in the second data packet on the second storage device.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The network device records a first corresponding relationship, where the first corresponding relationship is used to indicate a corresponding relationship between an identifier of the first file and the first storage device.

6. The method according to claim 5, characterized in that The method further comprises: The first storage device receives the second write request sent by the network device, and stores the data block in the first data packet in the second write request on the first storage device; The first storage device records a second correspondence relationship, where the second correspondence relationship is used to indicate a correspondence relationship between a block identifier of each data block in the first data packet and a storage position of each data block in the first data packet in the first storage device.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: The network device receives a first processing request sent by the computing device, where the first processing request includes an identifier of the first file and an identifier of a first operation; The network device determines, according to the first corresponding relationship, the first storage device storing the first file; The network device sends a second processing request to the first storage device, where the second processing request is used to instruct to perform the first operation on the first file; The network device obtains a first processing result corresponding to the second processing request from the first storage device; The network device returns the first processing result to the computing device.

8. The method according to claim 7, characterized in that The second processing request includes an identifier of the first file and an identifier of the first operation; The first storage device determines, according to the identifier of the first file, a block identifier of a data block included in the first file; The first storage device obtains the data blocks of the first file according to the block identifiers of the data blocks included in the first file and the second corresponding relationship, and performs the first operation on the data blocks of the first file to obtain the first processing result.

9. The method according to claim 8, characterized in that The network device records a third correspondence relationship, where the third correspondence relationship is used to indicate a correspondence relationship between the identifier of the first file and the block identifier of each data block in the first data packet; The first storage device determines, according to the identifier of the first file, a block identifier of a data block included in the first file, including: The first storage device sends an acquisition request to the network device, where the acquisition request includes an identifier of the first file; In response to the acquisition request, the network device determines the block identifiers of the data blocks included in the first file according to the third corresponding relationship, and sends the block identifiers of the data blocks included in the first file to the first storage device.

10. The method according to claim 8, characterized in that The first data packet carries an identifier of the first file, The first storage device records a third corresponding relationship according to the identifier of the first file, where the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet; The first storage device determines, according to the identifier of the first file, a block identifier of a data block included in the first file, including: The first storage device determines the block identifier of the data block included in the first file according to the locally recorded third corresponding relationship and the identifier of the first file.

11. The method according to claim 7, characterized in that The second processing request includes a block identifier of each data block in the plurality of data blocks of the first file and an identifier of the first operation; the block identifier of each data block is determined by the network device according to the recorded third corresponding relationship; The third corresponding relationship is used to indicate the corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet; The first storage device obtains multiple data blocks of the first file according to the second corresponding relationship, and performs the first operation on the data blocks of the first file to obtain the first processing result.

12. The method according to claim 7, characterized in that The first processing request also includes an identifier of the second file, and the method further includes: The network device determines, according to the first corresponding relationship, a second storage device storing the second file; The network device sends a third processing request to the second storage device, where the third processing request is used to instruct to perform the first operation on the second file; The network device obtains, from the second storage device, a second processing result corresponding to the third processing request; The network device summarizes the first processing result and the second processing result and returns them to the computing device.

13. The method according to any one of claims 6 to 12, characterized in that: The first operation includes any one or more of a read operation, character statistics, character screening, and character sorting.

14. A file processing system based on block storage, characterized in that: The system includes a network device and a storage cluster, wherein the storage cluster includes at least a first storage device, wherein: The network device is used to: receive a first write request sent by a computing device, wherein the first write request includes a plurality of data blocks, and each of the plurality of data blocks carries a block identifier; divide the data blocks belonging to a first file among the plurality of data blocks into a first data packet according to metadata; wherein the metadata includes an identifier of the file to which each of the plurality of data blocks belongs; and send a second write request to a first storage device in a storage cluster, wherein the second write request includes the first data packet, and the second write request is used to store the data blocks in the first data packet on the first storage device; The first storage device is used to: receive the second write request sent by the network device, and store the data block in the first data packet in the second write request on the first storage device.

15. The system according to claim 14, characterized in that The network device is further used to: divide the data blocks belonging to the second file among the plurality of data blocks into a second data packet according to the metadata; Determining a second storage device for storing the second data packet from the storage cluster; Sending a third write request to the second storage device, the write request including the second data packet, the third write request being used to store the data block in the second data packet on the second storage device; The second storage device is further used to: receive the third write request sent by the network device, and store the data block in the second data packet in the third write request on the second storage device.

16. The system according to claim 14 or 15, characterized in that The network device is further used to record a first corresponding relationship, where the first corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the first storage device.

17. The system according to claim 16, characterized in that The network device is further used to: receive a first processing request sent by the computing device, the first processing request including an identifier of the first file and an identifier of a first operation; determine the first storage device storing the first file according to the first corresponding relationship; and send a second processing request to the first storage device, the second processing request being used to instruct to perform the first operation on the first file; The first storage device is further used for: performing the first operation on the first file to obtain the first processing result according to the second processing request; and sending the first processing result to the network device; The network device is further configured to return the first processing result to the computing device.

18. The system according to claim 17, characterized in that The second processing request includes an identifier of the first file and an identifier of the first operation; The first storage device is further used to: determine the block identifier of the data block included in the first file according to the identifier of the first file; obtain the data block of the first file according to the block identifier of the data block included in the first file and the second corresponding relationship, and perform the first operation on the data block of the first file to obtain the first processing result; The second corresponding relationship is used to indicate the corresponding relationship between the block identifier of each data block in the first data packet and the storage position of each data block in the first data packet in the first storage device.

19. The system according to claim 18, characterized in that The network device is used to record a third corresponding relationship, where the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet; The first storage device is used to send an acquisition request to the network device, where the acquisition request includes an identifier of the first file; The network device is used to, in response to the acquisition request, determine the block identifiers of the data blocks included in the first file according to the third corresponding relationship, and send the block identifiers of the data blocks included in the first file to the first storage device.

20. The system according to claim 18, characterized in that The first data packet carries an identifier of the first file, and the first storage device records a third corresponding relationship, where the third corresponding relationship is used to indicate a corresponding relationship between the identifier of the first file and a block identifier of each data block in the first data packet; The first storage device is used to determine the block identifier of the data block included in the first file according to the third corresponding relationship recorded locally and the identifier of the first file.

21. The system according to claim 17, characterized in that The second processing request includes a block identifier of each data block in the plurality of data blocks of the first file and an identifier of the first operation; the block identifier of each data block is determined by the network device according to the recorded third corresponding relationship; The third corresponding relationship is used to indicate the corresponding relationship between the identifier of the first file and the block identifier of each data block in the first data packet; The first storage device is used to: obtain multiple data blocks of the first file according to the second corresponding relationship, and perform the first operation on the data blocks of the first file to obtain the first processing result; The second corresponding relationship is used to indicate the corresponding relationship between the block identifier of each data block in the first data packet and the storage position of each data block in the first data packet in the first storage device.

22. The system according to claim 17, characterized in that The first processing request also includes an identifier of the second file, and the network device is further used to: determine a second storage device storing the second file according to the first corresponding relationship; send a third processing request to the second storage device, wherein the third processing request is used to instruct to perform the first operation on the second file; Acquire, from the second storage device, a second processing result corresponding to the third processing request; The first processing result and the second processing result are summarized and returned to the computing device.

23. A computer storage medium, characterized in that The method comprises program instructions, and when the program instructions are executed by a device with computing functions, the device with computing functions executes the method according to any one of claims 1 to 13.

Images