Data operation method, device and communication equipment

By using the data operation method of communication devices to obtain and cache data blocks in Ceph object storage system, the problem of slow download and reading speed caused by large concurrent reused data in Ceph is solved, and efficient data access and modification are achieved.

CN114327282BActive Publication Date: 2024-11-26SHANGHAI SENSETIME TECH DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111651910.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-30
Publication Date
2024-11-26
Estimated Expiration
2041-12-30

AI Technical Summary

Technical Problem

In Ceph object storage system, when large concurrent reuse data is used, it leads to the problem of repeated downloads or slow data reading speeds, especially in artificial intelligence applications that cannot quickly read the same batch of data.

Method used

By implementing the data operation method in the communication device, obtain the target directory of the target file in Ceph, determine the data slices and data blocks to be downloaded/read, and determine whether they have been cached. If they have been cached, they will be returned directly, otherwise cached , improve the download and read speed of data. In addition, efficient data modification and reading is achieved through preset list cache and synchronized directory information.

Benefits of technology

It effectively solves the problem of repeatedly downloading/reading data, improves the download and reading speed of data, improves the efficiency of data modification, avoids repeated uploads of the same data, and improves the efficiency of data use.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114327282B_ABST
    Figure CN114327282B_ABST
Patent Text Reader

Abstract

The present application discloses a data operation method, device and communication equipment. The method includes: determining a target file to be operated, and obtaining a target directory of the target file in Ceph, wherein the target file includes a data piece to be downloaded / read; determining the target data piece corresponding to the data piece to be downloaded / read in Ceph and the target data block where the target data piece is located according to the target directory; determining whether the target data piece or target data block has been cached; and if so, returning the target data piece or target data block to download / read the target data piece or target data block. The above scheme can improve the speed of downloading / reading data and solve the problem of repeated downloading / reading.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data operation method, device and communication equipment. Background Art

[0002] As a new type of object storage system, Ceph has attracted more and more attention from users. How to use Ceph more conveniently has always been the focus of attention. In the case of large concurrent reuse of data, there will be problems of repeated downloads. When artificial intelligence needs to read the same batch of data multiple times, it cannot read the data quickly, so realizing fast data reading is an urgent problem to be solved. Summary of the invention

[0003] The present application at least provides a data operation method, apparatus and communication equipment.

[0004] A first aspect of the present application provides a data operation method, which is applied to a communication device, wherein the communication device mounts a distributed storage system Ceph, and the distributed storage system Ceph includes at least one data block, each data block includes at least one data slice, and the method includes: determining a target file to be operated, and obtaining a target directory of the target file in Ceph, wherein the target file includes a data slice to be downloaded / read; determining, based on the target directory, a target data slice corresponding to the data slice to be downloaded / read in Ceph and a target data block where the target data slice is located; determining whether the target data slice or target data block has been cached; and if so, returning the target data slice or target data block to download / read the target data slice or target data block.

[0005] Therefore, by obtaining the target directory of the target file in Ceph, the target data piece and target data block corresponding to the data piece to be downloaded / read of the target file in Ceph are determined based on the target directory, and then when the target data piece or target data block has been cached, the target data piece or target data block is returned to realize downloading / reading the target data piece or target data block. In this way, the speed of downloading / reading data is improved and the problem of repeated downloading / reading is solved.

[0006] In some embodiments, the method further includes: determining a preset number of data blocks after the target data block in Ceph; determining whether the preset number of data blocks have been cached; and if not, caching the preset number of data blocks. By caching the data blocks after the target data block, preloading is achieved, thereby improving the speed of sequential downloading / reading of files.

[0007] In some embodiments, obtaining a target directory of a target file in Ceph includes: obtaining a directory tree structure of Ceph through a cache service, wherein the target tree structure includes a target node for representing the target directory; wherein the target node includes detailed information and an identifier of the target file, and the identifier of the target file is used to indicate a target data block of the target file in Ceph. Obtaining the directory tree structure of Ceph through the cache service improves the access speed of Ceph.

[0008] In some embodiments, obtaining the directory tree structure of Ceph is performed in response to the input file structure identifier matching the target tree structure. By performing the cache service in response to the input file structure identifier matching the target tree structure to obtain the directory tree structure of Ceph, the reading speed of the file directory is greatly improved without scanning all files under the directory.

[0009] In some embodiments, the identity of the target file is represented by a hash value.

[0010] In some embodiments, the method further includes: obtaining a preset list, the preset list includes at least one cached data piece to be uploaded / written, at least one data piece to be uploaded / written is a data piece to be uploaded / written to a target file; and uploading / writing the at least one cached data piece to be uploaded / written to a corresponding data piece in a target data block of the target file in Ceph. By using the preset list, the cached data pieces are uploaded / written to the corresponding data pieces in the target data block where the target file is located, thereby achieving efficient data modification without modifying the entire target file before uploading it, and avoiding repeated uploading of the same data, thereby improving data usage efficiency.

[0011] In some embodiments, the method further includes: in response to at least one data piece to be uploaded / written being uploaded / written to a corresponding data piece in the target data block, updating the target directory and synchronizing it to the cache service. By updating the target directory and synchronizing it to the cache service, the reading speed of the file directory is greatly improved without scanning all files in the directory.

[0012] In some embodiments, the method further includes: determining a file to be uploaded / written, the file to be uploaded / written includes at least one data piece to be uploaded / written and directory information; determining the data block of the file to be uploaded / written in Ceph based on the directory information; obtaining a preset list, the preset list includes at least one cached data piece to be uploaded / written; and uploading / writing the cached at least one data piece to be uploaded / written to the corresponding data piece in the data block of the file to be uploaded / written in Ceph. Through the preset list, the cached data pieces are uploaded / written to the corresponding data pieces in the target data block where the file is located, so as to achieve efficient data modification without modifying the entire file before uploading it, and at the same time, avoid uploading the same data repeatedly, so as to improve the efficiency of data use.

[0013] In some embodiments, the method further includes: synchronizing the directory information to the cache service in response to at least one data piece to be uploaded / written being uploaded / written to a corresponding data piece in a data block of a file to be uploaded / written in Ceph. By synchronizing the directory information to the cache service, the reading speed of the file directory is greatly improved without scanning all files under the directory, and when uploading / writing the corresponding data piece to Ceph, the separation of the file data and the directory information is achieved by synchronizing the directory information to the cache service.

[0014] The second aspect of the present application provides a data operation device, which is applied to a communication device, wherein the communication device mounts a distributed storage system Ceph, and the distributed storage system Ceph includes at least one data block, each data block includes at least one data slice, and the device includes: an acquisition module, which is used to determine a target file to be operated and obtain a target directory of the target file in Ceph, wherein the target file includes a data slice to be downloaded / read; a determination module, which is used to determine, based on the target directory, a target data slice corresponding to the data slice to be downloaded / read in Ceph and a target data block where the target data slice is located; and a cache module, which is used to return the target data slice or target data block when the target data slice or target data block has been cached, so as to realize downloading / reading the target data slice or target data block.

[0015] A third aspect of the present application provides a communication device, which mounts a distributed storage system Ceph and includes a memory and a processor coupled to each other, the distributed storage system Ceph includes at least one data block, each data block includes at least one data slice, and the processor is used to execute program instructions stored in the memory to implement the data operation method in the above-mentioned first aspect.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the data operation method in the above-mentioned first aspect when the program instructions are executed by a processor.

[0017] In the above scheme, by obtaining the target directory of the target file in Ceph, the target data piece and target data block corresponding to the data piece to be downloaded / read of the target file in Ceph are determined according to the target directory, and then when the target data piece or target data block has been cached, the target data piece or target data block is returned to achieve downloading / reading the target data piece or target data block, thereby improving the speed of downloading / reading data and solving the problem of repeated downloading / reading. .

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 is a flowchart of a data operation method according to an embodiment of the present application;

[0021] Figure 2 is a flowchart of a data operation method according to an embodiment of the present application;

[0022] Figure 3 is a flowchart of a data operation method according to an embodiment of the present application;

[0023] Figure 4 is a flowchart of a data operation method according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of the framework of a data operation device according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of a framework of a communication device according to an embodiment of the present application;

[0026] Figure 7 It is a schematic diagram of the framework of the computer-readable storage medium of an embodiment of the present application. DETAILED DESCRIPTION

[0027] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0028] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0029] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] See also Figure 1 , Figure 1 : is a flow chart of the data operation method of an embodiment of the present application. The method is applied to a communication device, wherein the communication device mounts a distributed storage system Ceph, the distributed storage system Ceph includes at least one data block, and each data block includes at least one data slice. Regarding the distributed storage system Ceph, this application only describes the parts related to this application, and the parts not described in this application are not explained here. The communication device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the data operation method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0031] Specifically, if Figure 1 As shown, the method may include the following steps:

[0032] Step S11: Determine the target file to be operated, and obtain the target directory of the target file in Ceph.

[0033] The target file includes the data piece to be downloaded / read.

[0034] If the target file is a file stored in Ceph, it has a target directory in Ceph. The target file occupies a data block of Ceph, namely the target data block. The target file is divided into different data slices. Accordingly, different data slices of the target file correspond to different data slices of the target data block stored in Ceph. For example, the target file includes data slices to be downloaded / read. Accordingly, the target data block includes corresponding target data slices.

[0035] The target file to be operated can be determined according to actual needs. For example, when a file in Ceph needs to be downloaded / read, the file is determined as the target file.

[0036] The file structure information in Ceph is an efficient Json line structure, and the directory information of the files in Ceph is stored in a cache service, for example, an efficient cache service such as a redis cache service. The cache service can be deployed locally on the communication device, or it can be built on a server that is connected to the communication device. Taking the cache service as a redis cache service as an example, the redis cache service can be deployed locally on the communication device, or it can be built on a server that is connected to the communication device. When the redis cache service is deployed locally on the communication device, the directory information of the files stored or provided by it can be directly obtained. When the redis cache service is built on a server, that is, the redis cache server, the directory information of the files stored or provided by it is obtained through the communication connection.

[0037] Step S12: According to the target directory, determine the target data piece corresponding to the data piece to be downloaded / read in Ceph and the target data block where the target data piece is located.

[0038] The target data slice and target data block can be found in Ceph according to the target directory.

[0039] Step S13: Determine whether the target data slice or target data block has been cached.

[0040] When the target data slice or target data block has been cached, step S14 is executed; when the target data slice or target data block has not been cached, step S15 is executed.

[0041] When the target data slice or the target data block has been cached, the target data slice or the target data block may be cached in a cache module of the communication device.

[0042] Step S14: Return the target data slice or target data block to download / read the target data slice or target data block.

[0043] When the target data slice or target data block has been cached, the target data slice or target data block is returned to the bottom layer so that the target data slice or target data block can be downloaded / read through the bottom layer.

[0044] Step S15: caching the target data slice or target data block.

[0045] When the target data slice or target data block is not cached, the target data slice or target data block is cached in a cache module of the communication device, so that it can be directly called from the cache module of the communication device during subsequent operations.

[0046] The target data slice or target data block is cached, that is, the cache is performed in the form of data blocks and / or data slices.

[0047] In an implementation scenario, the above steps S14 and S15 can be executed in sequence, for example, step S14 is executed first, and then step S15; or, step S15 is executed first, and then step S14. The specific settings can be made according to actual applications and are not limited here.

[0048] In the above scheme, by obtaining the target directory of the target file in Ceph, the target data piece and target data block corresponding to the data piece to be downloaded / read of the target file in Ceph are determined based on the target directory, and then when the target data piece or target data block has been cached, the target data piece or target data block is returned to realize downloading / reading the target data piece or target data block, thereby improving the speed of downloading / reading data and solving the problem of repeated downloading / reading.

[0049] See also Figure 2 , Figure 2 : is a flow chart of the data operation method of the embodiment of the present application. Based on the above embodiment, the method may further include the following steps:

[0050] Step S21: Determine a preset number of data blocks that are located after the target data block in Ceph.

[0051] After determining the target data block, a preset number of data blocks located after the target data block is determined. The preset number can be set according to actual applications and is not limited here.

[0052] In one implementation scenario, the above step S21 and step S12 can be performed in a sequential order, for example, step S12 is performed first, and then step S21; or, step S21 is performed first, and then step S12. In another implementation scenario, the above step S21 and step S12 can be performed simultaneously, which can be set according to actual applications and is not limited here.

[0053] Step S22: Determine whether a preset number of data blocks have been cached.

[0054] When the preset number of data blocks are not cached, step S23 is executed. When the preset number of data blocks are cached, no processing is performed, and the cached data blocks are used for subsequent downloading / reading of corresponding data blocks in Ceph.

[0055] Step S23: caching a preset number of data blocks.

[0056] When a preset number of data blocks are not cached, they are cached for subsequent downloading / reading of corresponding data blocks in Ceph, thereby increasing the speed of subsequent downloading / reading.

[0057] In the above solution, preloading is achieved by caching the data blocks after the target data block, thereby improving the speed of sequential downloading / reading of the file.

[0058] As described above, the directory information of the file in Ceph is stored in a cache service, such as a redis service. In some embodiments, obtaining the target directory of the target file in Ceph includes: obtaining the directory tree structure of Ceph through the cache service, and the target tree structure includes a target node for representing the target directory. The target node includes detailed information and an identifier of the target file, and the identifier of the target file is used to indicate the target data block of the target file in Ceph.

[0059] In the directory tree structure, each node corresponds to a file, and the node contains detailed information and an identifier of the file. The identifier of each file indicates the data block occupied by the file in Ceph. The target node is a node in the directory tree structure. Accordingly, the target node includes detailed information and an identifier of the target file. The identifier of the target file is used to indicate the target data block of the target file in Ceph. In some examples, the identifier of the target file is represented by a hash value.

[0060] In the disclosed embodiment, the directory tree structure of Ceph is obtained through the cache service, and the target tree structure includes a target node for representing the target directory; wherein the target node includes detailed information and an identifier of the target file, and the identifier of the target file is used to indicate the target data block of the target file in Ceph, thereby improving the access speed of Ceph.

[0061] In some embodiments, obtaining the directory tree structure of Ceph is performed in response to the input file structure identifier matching the target tree structure. The input file structure identifier refers to the file structure identifier input by the user into the communication device, and the file structure identifier can be any preset information representing the file structure, for example, the header information of the file structure. The file structure identifier input into the communication device is received, and the file structure identifier is compared with the target tree structure. If the input file structure identifier matches the target tree structure, the directory tree structure of Ceph is obtained through the cache service.

[0062] In the disclosed embodiment, by obtaining the Ceph directory tree structure through the cache service in response to the input file structure identifier matching the target tree structure, the reading speed of the file directory is greatly improved without scanning all files under the directory.

[0063] See also Figure 3 , Figure 3 : is a flow chart of the data operation method of the embodiment of the present application. Based on the above embodiment, the method may also include the following steps:

[0064] Step 31: Get the preset list.

[0065] The preset list includes at least one cached data piece to be uploaded / written, and the at least one data piece to be uploaded / written is a data piece to be uploaded / written to a target file.

[0066] As can be seen from the above, the target file is a file stored in Ceph. When a data piece needs to be uploaded / written to the target file, it is cached and added to the preset list.

[0067] In one implementation scenario, the above step S31 and step S12 can be performed in a sequential order, for example, step S12 is performed first, and then step S31; or, step S31 is performed first, and then step S12. In another implementation scenario, the above step S31 and step S12 can be performed simultaneously, which can be set according to actual applications and is not limited here.

[0068] Step S32: Upload / write the cached at least one data slice to be uploaded / written to the corresponding data slice in the target data block of the target file in Ceph.

[0069] After obtaining the preset list, the cached data slices in the preset list can be uploaded / written to the corresponding data slices in the target data block where the target file is located after the preset time. In this way, when the target file is modified, only part of the data content is modified and uploaded, thereby achieving efficient data modification. There is no need to modify the entire target file before uploading it. At the same time, it avoids repeatedly uploading the same data, thereby improving data utilization efficiency.

[0070] In the disclosed embodiment, through a preset list, the cached data slices are uploaded / written to the corresponding data slices in the target data block where the target file is located, thereby achieving efficient data modification without the need to modify the entire target file before uploading it. At the same time, it avoids repeatedly uploading the same data, thereby improving data utilization efficiency.

[0071] As described above, the cached data pieces are uploaded / written to the corresponding data pieces in the target data block. In some embodiments, in response to at least one data piece to be uploaded / written being uploaded / written to the corresponding data piece in the target data block, the target directory is updated and synchronized to the cache service. By updating the target directory and synchronizing it to the cache service, the reading speed of the file directory is greatly improved, and there is no need to scan all files in the directory.

[0072] See also Figure 4 , Figure 4 : is a flow chart of the data operation method of the embodiment of the present application. Based on the above embodiment, the method can also be applied to uploading / writing files, specifically, it can also include the following steps:

[0073] Step 41: Determine the file to be uploaded / written.

[0074] The file to be uploaded / written includes at least one data piece to be uploaded / written and directory information.

[0075] The file to be uploaded / written refers to a new file to be uploaded / written to Ceph. The new file may have one data piece to be uploaded / written, or may have multiple data pieces to be uploaded / written.

[0076] In one implementation scenario, the above step S11 and step S41 can be performed in a sequential order, for example, step S11 is performed first and then step S41; or step S41 is performed first and then step S11. In another implementation scenario, the above step S11 and step S41 can be performed simultaneously, which can be set according to actual applications and is not limited here.

[0077] Step 42: Determine the data block of the file to be uploaded / written in Ceph based on the directory information.

[0078] The data blocks of the files to be uploaded / written in Ceph can be determined based on the target information.

[0079] Step 43: Get the preset list.

[0080] The preset list includes at least one cached data piece to be uploaded / written.

[0081] When a piece of data of the new file needs to be uploaded / written to Ceph, it is cached and added to the preset list.

[0082] Step 44: Upload / write the cached at least one data piece to be uploaded / written to the corresponding data piece in the data block of the file to be uploaded / written in Ceph.

[0083] After obtaining the preset list, you can upload / write the cached data slices in the preset list to the corresponding data slices in the data block where the new file is located in Ceph after the preset time. In this way, when the new file is modified, only part of the data content is modified and uploaded, so as to achieve efficient data modification without modifying the entire file before uploading it. At the same time, it avoids uploading the same data repeatedly, thus improving data utilization efficiency.

[0084] In the disclosed embodiment, through a preset list, the cached data slices are uploaded / written to the corresponding data slices in the target data block where the file is located, thereby achieving efficient data modification without the need to modify the entire file before uploading it. At the same time, it avoids repeatedly uploading the same data, thereby improving data utilization efficiency.

[0085] As described above, the cached data pieces are uploaded / written to the corresponding data pieces in the data block where the new file is located in Ceph. In some embodiments, in response to at least one data piece to be uploaded / written being uploaded / written to the corresponding data piece in the data block of the file to be uploaded / written in Ceph, the directory information is synchronized to the cache service. By synchronizing the directory information to the cache service, the reading speed of the file directory is greatly improved without scanning all the files under the directory, and when uploading / writing the corresponding data piece to Ceph, the separation of the file data and the directory information is achieved by synchronizing the directory information to the cache service.

[0086] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0087] See also Figure 5 , Figure 5 Schematic diagram of the framework of the data operation device of the embodiment of the present application. The data operation device 50 includes an acquisition module 51, a determination module 52 and a cache module 53. The acquisition module 51 is used to determine the target file to be operated and obtain the target directory of the target file in Ceph, wherein the target file includes the data piece to be downloaded / read; the determination module 52 is used to determine the target data piece corresponding to the data piece to be downloaded / read in Ceph and the target data block where the target data piece is located according to the target directory; the cache module 53 is used to return the target data piece or target data block when the target data piece or target data block has been cached, so as to download / read the target data piece or target data block.

[0088] In the above scheme, by obtaining the target directory of the target file in Ceph, the target data piece and target data block corresponding to the data piece to be downloaded / read of the target file in Ceph are determined based on the target directory, and then when the target data piece or target data block has been cached, the target data piece or target data block is returned to realize downloading / reading the target data piece or target data block, thereby improving the speed of downloading / reading data and solving the problem of repeated downloading / reading.

[0089] In some embodiments, the determination module 52 is further used to determine a preset number of data blocks located after the target data block in Ceph; the cache module 53 is further used to determine whether the preset number of data blocks have been cached; and if not, cache the preset number of data blocks. By caching the data blocks after the target data block, preloading is achieved, thereby improving the speed of sequential downloading / reading of files.

[0090] In some embodiments, obtaining a target directory of a target file in Ceph includes: obtaining a directory tree structure of Ceph through a cache service, wherein the target tree structure includes a target node for representing the target directory; wherein the target node includes detailed information and an identifier of the target file, and the identifier of the target file is used to indicate a target data block of the target file in Ceph. Obtaining the directory tree structure of Ceph through the cache service improves the access speed of Ceph.

[0091] In some embodiments, obtaining the Ceph directory tree structure is performed in response to the input file structure identifier matching the target tree structure. By performing the Ceph directory tree structure acquisition through the cache service in response to the input file structure identifier matching the target tree structure, the reading speed of the file directory is greatly improved without scanning all files under the directory.

[0092] In some embodiments, the identity of the target file is represented by a hash value.

[0093] In some embodiments, it also includes: an acquisition module for acquiring a preset list, the preset list includes at least one cached data piece to be uploaded / written, and at least one data piece to be uploaded / written is a data piece to be uploaded / written to a target file; an upload / write module for uploading / writing at least one cached data piece to be uploaded / written to a corresponding data piece in a target data block of the target file in Ceph. Through the preset list, the cached data pieces are uploaded / written to the corresponding data pieces in the target data block where the target file is located, so as to achieve efficient data modification without modifying the entire target file before uploading it, and at the same time, avoid uploading the same data repeatedly, so as to improve the efficiency of data use.

[0094] In some embodiments, it further includes: a directory module, which is used to update the target directory and synchronize it to the cache service in response to at least one data piece to be uploaded / written being uploaded / written to the corresponding data piece in the target data block. By updating the target directory and synchronizing it to the cache service, the reading speed of the file directory is greatly improved, and there is no need to scan all files in the directory.

[0095] In some embodiments, it also includes: an acquisition module for determining a file to be uploaded / written, the file to be uploaded / written includes at least one data piece to be uploaded / written and directory information; based on the directory information, determining the data block of the file to be uploaded / written in Ceph; acquiring a preset list, the preset list includes at least one cached data piece to be uploaded / written; and an upload / write module for uploading / writing at least one cached data piece to be uploaded / written to the corresponding data piece in the data block of the file to be uploaded / written in Ceph. Through the preset list, the cached data pieces are uploaded / written to the corresponding data pieces in the target data block where the file is located, so as to achieve efficient data modification without modifying the entire file before uploading it. At the same time, it is avoided to upload the same data repeatedly, so as to improve the efficiency of data use.

[0096] In some embodiments, it further includes: a directory module, which is used to synchronize directory information to a cache service in response to at least one data piece to be uploaded / written being uploaded / written to a corresponding data piece in a data block of a file to be uploaded / written in Ceph. By synchronizing the directory information to the cache service, the reading speed of the file directory is greatly improved without scanning all files under the directory, and when uploading / writing the corresponding data piece to Ceph, the separation of file data and directory information is achieved by synchronizing the directory information to the cache service.

[0097] See also Figure 6 , Figure 6 60 is a schematic diagram of the framework of the communication device of the embodiment of the present application. The communication device 60 mounts the distributed storage system Ceph 60' and includes a memory 61 and a processor 62 coupled to each other. The processor 62 is used to execute the program instructions stored in the memory 61 to implement the steps in any of the above data operation method embodiments. In a specific implementation scenario, the communication device 60 may include but is not limited to: a microcomputer, a server, and in addition, the communication device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0098] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in any of the above data operation embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0099] In the above scheme, by obtaining the target directory of the target file in Ceph, the target data piece and target data block corresponding to the data piece to be downloaded / read of the target file in Ceph are determined based on the target directory, and then when the target data piece or target data block has been cached, the target data piece or target data block is returned to realize downloading / reading the target data piece or target data block, thereby improving the speed of downloading / reading data and solving the problem of repeated downloading / reading.

[0100] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 701 that can be executed by a processor, and the program instructions 701 are used to implement the steps in any of the above data operation method embodiments.

[0101] In the above scheme, by obtaining the target directory of the target file in Ceph, the target data piece and target data block corresponding to the data piece to be downloaded / read of the target file in Ceph are determined based on the target directory, and then when the target data piece or target data block has been cached, the target data piece or target data block is returned to realize downloading / reading the target data piece or target data block, thereby improving the speed of downloading / reading data and solving the problem of repeated downloading / reading.

[0102] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0103] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0104] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0105] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A data operation method, applied to a communication device, characterized in that: The communication device mounts a distributed storage system Ceph, the distributed storage system Ceph includes at least one data block, each of the data blocks includes at least one data slice, and the method includes: Determine a target file to be operated, and obtain a target directory of the target file in the Ceph, wherein the target file includes a data piece to be downloaded / read; Determine, according to the target directory, a target data piece corresponding to the data piece to be downloaded / read in the Ceph and a target data block where the target data piece is located; Determining whether the target data slice or the target data block has been cached; and If yes, returning the target data slice or the target data block to download / read the target data slice or the target data block; Determining a preset number of data blocks located after the target data block in the Ceph; Determining whether the preset number of data blocks has been cached; and If not, the preset number of data blocks are cached.

2. The method according to claim 1, characterized in that The step of obtaining the target directory of the target file in the Ceph comprises: Obtaining, through a cache service, a directory tree structure of the Ceph, wherein the target tree structure includes a target node for representing the target directory; The target node includes detailed information and an identifier of the target file, and the identifier of the target file is used to indicate the target data block of the target file in the Ceph.

3. The method according to claim 2, characterized in that The obtaining of the Ceph directory tree structure is performed in response to the input file structure identifier matching the target tree structure.

4. The method according to claim 2, characterized in that: The identification of the target file is represented by a hash value.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: Obtaining a preset list, wherein the preset list includes at least one cached data piece to be uploaded / written, and the at least one data piece to be uploaded / written is a data piece to be uploaded / written to the target file; The at least one cached data slice to be uploaded / written is uploaded / written to a corresponding data slice in the target data block of the target file in the Ceph.

6. The method according to claim 5, characterized in that Further including: In response to the at least one data piece to be uploaded / written being uploaded / written to a corresponding data piece in the target data block, the target directory is updated and synchronized to the cache service.

7. The method according to any one of claims 1 to 4, characterized in that Also includes: Determine a file to be uploaded / written, wherein the file to be uploaded / written includes at least one data piece to be uploaded / written and directory information; Determine the data block of the file to be uploaded / written in the Ceph according to the directory information; Obtaining a preset list, wherein the preset list includes the at least one cached data slice to be uploaded / written; The at least one cached data slice to be uploaded / written is uploaded / written to the corresponding data slice in the data block of the file to be uploaded / written in the Ceph.

8. The method according to claim 7, characterized in that Further including: In response to the at least one data piece to be uploaded / written being uploaded / written to a corresponding data piece in a data block of the file to be uploaded / written in the Ceph, the directory information is synchronized to the cache service.

9. A data operation device, applied to a communication device, characterized in that: The communication device mounts a distributed storage system Ceph, the distributed storage system Ceph includes at least one data block, each of the data blocks includes at least one data slice, and the apparatus includes: An acquisition module, used to determine a target file to be operated, and to acquire a target directory of the target file in the Ceph, wherein the target file includes a data piece to be downloaded / read; A determination module, configured to determine, based on the target directory, a target data piece corresponding to the data piece to be downloaded / read in the Ceph and a target data block where the target data piece is located; A cache module is used to return the target data slice or the target data block when the target data slice or the target data block has been cached, so as to download / read the target data slice or the target data block.

10. A communication device, characterized in that: A distributed storage system Ceph is mounted and includes a memory and a processor coupled to each other, wherein the distributed storage system Ceph includes at least one data block, each of the data blocks includes at least one data slice, and the processor is used to execute program instructions stored in the memory to implement the data operation method described in any one of claims 1 to 8.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the data operation method according to any one of claims 1 to 8 is implemented.