Data processing method and related equipment

Abstract

The invention discloses a data processing method and related equipment, and the method comprises the steps: obtaining a to-be-stored current data stream, and segmenting the current data stream based on historical storage information of a data storage space to obtain a plurality of to-be-written data; writing the to-be-written data into the data storage space to obtain target data corresponding to each piece of to-be-written data; constructing metadata information corresponding to the target data based on the to-be-written data; according to the metadata information, generating object data corresponding to the target data, and storing the object data to an object storage; and when a data reading request sent by the terminal is received, reading the target object data in the object storage, and sending the target object data to the terminal. According to the method and the device, the mapping relation among the target data, the metadata information and the object data can be constructed, so that the object storage data can be supported to be accessed, and the file can be efficiently read and written.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and related equipment. Background Art

[0002] Object storage is a computer data storage architecture that manages data as objects, unlike other storage architectures (such as file systems that manage data as a hierarchy of files, and block storage that manages data as sectors and blocks within tracks). Each object typically consists of the data itself, a varying amount of metadata, and a globally unique identifier. Object storage can be implemented at multiple levels, including the device level (object storage devices), the system level, and the interface level. In each case, object storage attempts to implement capabilities that other storage architectures do not have, such as interfaces that can be programmed directly by applications, namespaces that can span multiple physical hardware instances, and data management functions such as data replication and data distribution at object-level granularity.

[0003] In the prior art, the files to be stored are directly stored as objects in the object storage, and the reading and writing of a single file data stream will be limited by the capacity of the previous object data in the object storage. In addition, the application of the prior art is not suitable for random reading of data and will also reduce cache management efficiency. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and related equipment. The related equipment may include a data processing device, an electronic device, a computer-readable storage medium and a computer program product. By constructing a mapping relationship between target data, metadata information, and object data, it is possible to support access to object storage data while achieving efficient reading and writing of files.

[0005] The present application provides a data processing method, including:

[0006] Acquire a current data stream to be stored, and divide the current data stream based on historical storage information of the data storage space to obtain a plurality of data to be written;

[0007] Writing the data to be written into the data storage space to obtain target data corresponding to each data to be written;

[0008] Based on the data to be written, construct metadata information corresponding to the target data;

[0009] Generate object data corresponding to the target data according to the metadata information, and store the object data in object storage;

[0010] When a data read request sent by a terminal is received, target object data is read from the object storage, and the target object data is sent to the terminal.

[0011] Accordingly, an embodiment of the present application provides a data processing device, including:

[0012] An acquisition unit, used for acquiring a current data stream to be stored, and dividing the current data stream based on historical storage information of the data storage space to obtain a plurality of data to be written;

[0013] A writing unit, used for writing the data to be written into the data storage space to obtain target data corresponding to each data to be written;

[0014] A construction unit, configured to construct metadata information corresponding to the target data based on the data to be written;

[0015] A storage unit, configured to generate object data corresponding to the target data according to the metadata information, and store the object data in an object storage;

[0016] The reading unit is used to read the target object data in the object storage and send the target object data to the terminal when receiving a data reading request sent by the terminal.

[0017] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to stop writing data if the data to be written has not yet filled up the target data in the data storage space, and determine the target data that has not yet been filled up as the first target data; based on the data to be written that has been written in the first target data, construct first metadata information corresponding to the first target data; based on the first metadata information, generate first object data corresponding to the first target data, and store the first object data in the object storage.

[0018] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to write the data to be written into the remaining space of the first target data; based on the data to be written written into the remaining space of the first target data, construct second metadata information corresponding to the first target data; based on the second metadata information, generate second object data corresponding to the first target data, and store the second object data in the object storage.

[0019] Optionally, in some embodiments of the present application, the reading unit can be specifically used to determine, when receiving a data reading request sent by a terminal, target metadata information corresponding to the data to be read based on the data reading request; based on the target metadata information, read target object data corresponding to the data to be read in the object storage; and send the target object data to the terminal.

[0020] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to obtain a first random data stream to be stored and a first write offset; based on the historical storage information of the data storage space and the first write offset, the first random data stream is divided to obtain multiple first random data; based on the first write offset, the first random data is written into the data storage space to obtain third target data corresponding to each first random data, wherein there is no overlapping part between the first random data and the data to be written that has been written in the first target data; based on the first random data, third metadata information corresponding to the third target data is constructed; according to the third metadata information, third object data corresponding to the third target data is generated, and the third object data is stored in the object storage.

[0021] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to obtain a second random data stream to be stored and a second write offset; based on the historical storage information of the data storage space and the second write offset, write the second random data stream completely over the target data of the data storage space; based on the second random data stream, construct fourth metadata information corresponding to the target data; based on the fourth metadata information, generate fourth object data corresponding to the target data, and store the fourth object data to the object storage.

[0022] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to obtain a third random data stream to be stored and a third write offset; based on the historical storage information of the data storage space and the third write offset, write the third random data stream over the continuous target data of the data storage space; based on the third random data stream, construct fifth metadata information corresponding to the target data located at the first position in the continuous target data, and sixth metadata information corresponding to the target data located at the last position in the continuous target data; based on the fifth metadata information, generate fifth object data corresponding to the target data located at the first position, and store the fifth object data in the object storage; based on the sixth metadata information, generate sixth object data corresponding to the target data located at the last position, and store the sixth object data in the object storage.

[0023] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to obtain a data truncation request, and determine a data truncation position based on the data truncation request; determine the target data at the data truncation position; and adjust the metadata information corresponding to the target data at the data truncation position based on the offset of the data truncation position in the target data.

[0024] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps in the data processing method provided in the embodiment of the present application.

[0025] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in the data processing method provided in the embodiment of the present application.

[0026] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the data processing method provided in the embodiment of the present application.

[0027] The embodiment of the present application provides a data processing method and related equipment, which can obtain the current data stream to be stored, and based on the historical storage information of the data storage space, the current data stream is segmented to obtain multiple data to be written; the data to be written is written into the data storage space to obtain the target data corresponding to each data to be written; based on the data to be written, the metadata information corresponding to the target data is constructed; according to the metadata information, the object data corresponding to the target data is generated, and the object data is stored in the object storage; when a data read request sent by the terminal is received, the target object data is read in the object storage, and the target object data is sent to the terminal. The present application can use the object storage module to manage array resources, and abstract the metadata service module to manage metadata logic. In addition, for the flat KV structure of the object storage, the directory tree is abstracted, and the mapping relationship between the node and the data block is constructed. The technical solution of the present application can support POSIX semantic access to object storage data while achieving efficient reading and writing of files, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. The method can achieve strong consistency of data and can be expanded in parallel to provide high availability and high reliability service guarantees for the metadata service module. Moreover, this method can achieve storage and computing separation and is applicable to multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a scenario schematic diagram of the data processing method provided in the embodiment of the present application;

[0030] Figure 2 is a first flow chart of the data processing method provided in an embodiment of the present application;

[0031] Figure 3 is a second flow chart of the data processing method provided in an embodiment of the present application;

[0032] Figure 4 is a third flow chart of the data processing method provided in the embodiment of the present application;

[0033] Figure 5 is a fourth flow chart of the data processing method provided in an embodiment of the present application;

[0034] Figure 6 is a fifth flow chart of the data processing method provided in the embodiment of the present application;

[0035] Figure 7 is a sixth flow chart of the data processing method provided in an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of the overall architecture of the data processing system provided in the embodiment of the present application;

[0037] Fig. 9 It is a schematic diagram of the directory tree structure provided by the embodiment of the present application;

[0038] Fig.10 It is a schematic diagram of the reading and writing process provided by the embodiment of the present application;

[0039] Fig.11 It is a schematic diagram of the control process provided by the embodiment of the present application;

[0040] Fig.12 It is a schematic diagram of the mapping relationship between data provided in the embodiment of the present application;

[0041] Fig.13 This is a normal sequence writing schematic diagram provided by an embodiment of the present application;

[0042] Fig.14 This is a sequential write Flush diagram provided in an embodiment of the present application;

[0043] Fig.15 It is a schematic diagram of data reading provided by an embodiment of the present application;

[0044] Fig.16 This is a schematic diagram of random write without overwriting provided by an embodiment of the present application;

[0045] Fig.17 This is a first schematic diagram of random writing with coverage provided by an embodiment of the present application;

[0046] Fig.18 It is a second schematic diagram of random writing with overlay provided in an embodiment of the present application;

[0047] Fig.19 It is a first schematic diagram of data truncation provided in an embodiment of the present application;

[0048] Fig. 20 is a second schematic diagram of data truncation provided in an embodiment of the present application;

[0049] Fig.21 It is a schematic diagram of data merging provided in an embodiment of the present application;

[0050] Fig. 22 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0051] Fig.23 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0053] The embodiment of the present application provides a data processing method and related equipment, and the related equipment may include a data processing device, an electronic device, a computer-readable storage medium and a computer program product. The data processing device may be integrated in an electronic device, and the electronic device may be a terminal or a server.

[0054] It is understandable that the data processing method of this embodiment can be executed on a terminal, can be executed on a server, or can be executed by both a terminal and a server. The above examples should not be construed as limiting the present application.

[0055] like Figure 1As shown, a terminal and a server jointly execute a data processing method as an example. The data processing system provided in the embodiment of the present application includes a terminal and a server, etc. The terminal and the server are connected via a network, such as a wired or wireless network connection, etc., wherein the data processing device can be integrated in the terminal.

[0056] Among them, the server can be used to: obtain the current data stream to be stored, and based on the historical storage information of the data storage space, split the current data stream to obtain multiple data to be written; write the data to be written into the data storage space to obtain the target data corresponding to each data to be written; construct the metadata information corresponding to the target data based on the data to be written; generate the object data corresponding to the target data according to the metadata information, and store the object data in the object storage; when receiving the data read request sent by the terminal, read the target object data in the object storage, and send the target object data to the terminal. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In the data processing method or device disclosed in this application, multiple servers can be composed of a blockchain, and the server is a node on the blockchain.

[0057] The terminal can be used to send a data write request to the server, the data write request carries the current data stream to be stored, and receives the target object data sent by the server to complete the data read operation. The terminal can include a mobile phone, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, a tablet computer, a laptop computer, or a personal computer (PC). A client can also be set on the terminal, and the client can be an application client or a browser client, etc.

[0058] The following are detailed descriptions. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. This embodiment will be described from the perspective of a data processing device, which can be integrated in an electronic device, which can be a server or a terminal.

[0059] like Figure 8As shown, the data processing system may include a metadata service module and an object storage module. In layman's terms, the data processing system may divide the current data stream from the client into multiple data blocks, the object storage module may store the object data corresponding to each data block on different replica nodes, and the metadata service module may record the storage location information of each object data, that is, record on which replica node each object data is stored.

[0060] Among them, the client can provide a file reading and writing application programming interface, and can support multiple access protocols. For example, the client can provide a POSIX semantic interface at the back end, so that the client encapsulates the file system semantic interface for the client to use. The access protocol can be HCFS or FUSE. Of course, it can also be other types of general access protocols. This embodiment of the application does not make special restrictions on this. At the same time, the client can also support local metadata and data caching.

[0061] Among them, MDS (Meta Data service) is a metadata service module, which is used to manage the file system directory tree and the metadata information therein. Among them, the metadata service module regards files, directories, links, etc. as index nodes, and each index node contains related attributes and a linked list of storage space handles on each object storage target. In addition, the metadata service module can achieve high availability and active-standby disaster recovery through a distributed election algorithm, thereby maintaining the directory tree structure and achieving efficient access and operation of metadata information.

[0062] The object storage module is a service that stores data in the form of objects and implements the object storage function. It is responsible for allocating actual storage space for data objects, providing a distributed file lock mechanism during data transmission, and providing reliable security verification for data access. The data stream from the client can directly access the object storage database.

[0063] Among them, a directory tree is a virtual "container" containing a digital file system in a computer or related device. It stores a group of files and other directories. A typical file system may contain thousands of directories. Multiple files can be stored in a directory to achieve the purpose of organized file storage. Another directory in a directory is called its subdirectory. In this way, these directories form a hierarchical or tree structure.

[0064] Among them, the POSIX (Portable Operating System Interface) semantic interface is a portable operating system interface, which is a general term for a series of interrelated standards that define application programming interfaces in order to run software on various UNIX operating systems.

[0065] HCFS is a high-performance cluster file system designed to solve the problem of mass storage and implement a stable, network-independent data storage solution. It provides high-throughput data transmission for the cluster, uses an error transfer mechanism to prevent single-point errors from causing system crashes, and can quickly and automatically recover from errors.

[0066] RS (Root server) is an interface for external management and control, including functions such as creating and deleting file systems, creating and deleting mount points, and configuring file system properties. At the same time, the external management and control interface can also distribute the file system evenly to different metadata service modules, and achieve high availability and active-standby disaster recovery through a distributed election algorithm.

[0067] Among them, Proxy is an interface that provides metadata access to the outside world, including functions such as creating and deleting file handles and metadata update interfaces. The client can directly access the interface that provides metadata access to the outside world, and it will perform protocol conversion and forward it to the metadata service module where the file system is located.

[0068] Among them, the file system is a method of storing and organizing computer data, which makes it easy to access and search for it. The file system uses the abstract logical concept of files and tree directories instead of the concept of data blocks used by physical devices such as hard disks and optical disks. When users use the file system to save data, they do not need to care about the address of the data block on the hard disk (or optical disk) where the data is actually saved. They only need to remember the directory and file name to which the file belongs. Before writing new data, users do not need to care about which block address on the hard disk is not in use. The storage space management (allocation and release) function on the hard disk is automatically completed by the file system. Users only need to remember which file the data is written to.

[0069] Inode (index node) refers to a data structure in many "Unix-like file systems" that is used to describe file system objects (including files, directories, device files, sockets, pipes, etc.). Each inode stores the attributes and disk block location of the file system object data. The file system object attributes include various metadata, as well as user groups and permission data.

[0070] Among them, DB Cluster is the backend metadata storage cluster, which can be any type of data storage and can also be extended to a KV storage structure.

[0071] like Figure 2 As shown, the specific process of the data processing method can be as follows:

[0072] 201. Obtain a current data stream to be stored, and divide the current data stream based on historical storage information of the data storage space to obtain a plurality of data to be written.

[0073] The historical storage information of the data storage space may be the historical storage status of the data in the data storage space, and the storage location of the current data stream to be stored may be known according to the historical storage information.

[0074] The embodiment of the present application can be constructed as follows Figure 8 The data processing system shown:

[0075] 1. The metadata service module can store metadata information in the DB / KV cluster, maintain the directory tree and metadata information through the metadata service module itself, and provide a POSIX semantic interface to the outside world. In addition, the state machines of multiple file systems are maintained through a distributed election algorithm, while maintaining various metadata information mapping relationships to ensure strong consistency and correctness of metadata.

[0076] 2. The external management and control interface can assign the file system created by the user and the configured attributes to the specified metadata service module. At the same time, the external management and control interface will record the routing information of the file system for user access.

[0077] 3. The interface that provides metadata access to the outside world can route the request for metadata in the data reading request sent by the user when reading data to the metadata service module recorded by the interface that provides management and control to the outside world.

[0078] 4. The client can encapsulate the portable operating system interface provided by the external metadata access interface to access the file system data and metadata.

[0079] From the overall architecture of the data processing system, we can see that the metadata service module is used to maintain the directory tree and metadata information, while the actual data is stored in the object storage module.

[0080] In the embodiment of the present application, the file system can be represented as a directory tree structure, such as Fig. 9 The directory tree shown includes three directories and two files in total. Files are generally represented as leaf nodes of the tree, while intermediate path nodes are represented as directories, such as the root directory / , other directories / directory 1, and / directory 1 / directory 2. Files include / directory 1 / file 2, and / file 1.

[0081] Each node in the directory tree is called an Inode. Each node has its own attributes, which can be used to identify whether the node belongs to a directory or a file. POSIX semantics include unary operations and binary operations. Generally, nodes are identified by InodeID, so the file system needs to have the ability to quickly find nodes by node ID. However, in scenarios such as HCFS, it is necessary to find node IDs by path and then locate specific nodes by node IDs. Unary operations include adding, deleting, modifying, and checking nodes, modifying node attributes, etc.; binary operations include renaming operations, such as moving file 2 to directory 2, which involves modifying the attributes of two parent directories, that is, pointing the parent directory of file 2 to directory 2, and deleting the content related to file 2 in the child item of directory 1.

[0082] Among them, the structural mapping relationship between metadata and data to be written is as follows: Fig.12 As shown, the file node written by the user through the current data stream can be divided into multiple 4M data blocks, each data block can correspond to one or more metadata information, wherein the size of the metadata information is not fixed and can be determined by the client. For example, the user can manually trigger the upload operation on the client to persist the data, and the persistence operation can trigger data upload. The uploaded data can be in several data blocks. If the uploaded data is in several complete data blocks, each data block corresponds to a metadata information; if the data block is not full due to the user's operation, the part containing data in the unfull data block A will correspond to a metadata information, and if the user continues to write data to the data block A, the subsequent part of the data written in the data block A will also correspond to a metadata information. At this time, the data block A corresponds to more than one metadata information, and these metadata information can record the object storage index and the object storage offset.

[0083] The size of the data block is fixed when each file is created. The size of the data block is a property of the file system and can be obtained when the file system is mounted. The data block is divided into multiple metadata information. One purpose is to meet the semantics of data stream Flush. If the client needs to upload before writing a whole data block, a metadata information is required to record the uploaded data. Another purpose is to meet the semantics of POSIX random write. Since the previously written data may be overwritten, a new metadata information can be constructed for the newly overwritten data, and new object data can be generated for storage. For the part that is not overwritten, the offset attribute in the old metadata information can be changed, and the previous object data can be reused. The purpose of this is to append writes, and only the metadata information is updated to avoid data copying, thereby greatly improving the performance of random writes. The metadata information can also be merged to compress the amount of metadata, and the redundant data in the object data can be handled by the garbage collection module.

[0084] In one embodiment, for example, the direction of the interface can be divided into two categories: one is Fig.11 The control interface shown in the figure; the other is Fig.10 The data stream interface shown in the figure can be further divided into a read data stream interface and a write data stream interface.

[0085] The management and control interface can be applied to the management and control of the file system. The client sends a request to create / delete / change the file system to the interface that provides external management and control. The interface that provides external management and control assigns the metadata service module to the new file system state machine. After receiving the instruction, the metadata service module constructs the directory tree and uses the network log to pre-write WAL to maintain the status of each file system. The metadata service module then provides a metadata operation interface to the outside.

[0086] The write data stream interface can be applied to the data write operation. The user writes the current data stream through FUSE, HCFS and other methods. Whenever the data stream fills a 4M data block, the 4M data block is confirmed as the target data, and the metadata information and object data corresponding to the target data are constructed accordingly. The object data is uploaded to the object storage module, and the metadata information is submitted to the metadata service module and saved through the interface that provides metadata access to the outside. Among them, steps 201 to 204 describe the steps of writing data.

[0087] The read data stream interface can be applied to the data reading operation. When the user reads data through methods such as FUSE and HCFS, the size of the data to be read can be obtained by obtaining the file handle to be read, and the metadata information corresponding to the data to be read can be determined from the metadata service module according to the size of the data block and the size of the data to be read through the interface for providing metadata access to the outside world, and then the data to be read can be obtained from the object storage module according to the object storage index in the metadata information. Among them, step 205 describes the step of reading data.

[0088] 202. Write the data to be written into the data storage space to obtain target data corresponding to each data to be written.

[0089] Among them, the target data is the data block formed after the data to be written is written into the data storage space, that is, one target data can refer to a data block, but in the embodiment of the present application, the data block is an abstract concept and is only used to perform fixed-length segmentation on the data stream.

[0090] Block refers to a data block. In computer technology, a block is a standard length of bytes or bits. A data block is the smallest unit of database management and storage, and is the unit that constitutes a database interval. When processing data streams generated by computer programs, block data can simplify the processing process. Generally, a complete block is read at a time.

[0091] For example, a user can pass in the files he wants to store through the client and form a current data stream to be stored. Based on the historical storage information of the data storage space, the current data stream can be segmented in units of 4M to obtain multiple 4M-sized data to be written, and then these data to be written are written into the data storage space to obtain the target data corresponding to each data to be written, and each 4M-sized target data can be regarded as a data block.

[0092] 203. Based on the data to be written, metadata information corresponding to the target data is constructed.

[0093] Metadata refers to "data that describes data", which is mainly used to describe the attribute information of data to simplify the search process and facilitate use. It is used to support functions such as indicating storage location, historical data, resource search, file records, etc. Metadata is a kind of electronic catalog. In order to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby achieving the purpose of assisting data retrieval.

[0094] The metadata information (Range) can be understood as each node (Inode) is divided into multiple data blocks of a fixed size, and each data block corresponds to a piece of metadata information. However, if a data block of a node is overwritten, the overwritten part will be used as independent metadata information and split the original metadata information. Simply put, this metadata management method will divide a node into multiple data blocks, each data block may correspond to one or more metadata information, and each metadata information corresponds to an object data on the object storage.

[0095] For example, metadata information corresponding to the target data may be constructed according to the attribute information of the data to be written, and the metadata information may include information such as data block identification, data block offset, metadata information size, and object data offset.

[0096] 204. Generate object data corresponding to the target data according to the metadata information, and store the object data in the object storage.

[0097] For example, object data can be generated based on the data content of the target data according to the information recorded in the metadata information, and the object data can be stored in the object storage. In other words, there is a mapping relationship between the target data, the metadata information, and the object data.

[0098] Metadata mapping (Metadata Mapping / Crosswalking) refers to the conversion of different metadata formats using a specific conversion program. This article refers to the mapping relationship between nodes, data blocks, metadata information, and object storage structures.

[0099] As can be seen from steps 201 to 204, the core principle of the write process is to upload part of the data to the object storage module, then construct the metadata information corresponding to the data through the abstract data block concept, and then submit the metadata information to the metadata service module, and use the metadata service module to verify and record the metadata information.

[0100] Optionally, in one embodiment, after the step of “obtaining the current data stream to be stored, and dividing the current data stream based on the historical storage information of the data storage space to obtain a plurality of data to be written”, the following may also be included:

[0101] If the data to be written has not yet filled up the target data in the data storage space, the data writing is stopped, and the target data that has not yet filled up is determined as the first target data;

[0102] Constructing first metadata information corresponding to the first target data based on the data to be written that has been written in the first target data;

[0103] First object data corresponding to the first target data is generated according to the first metadata information, and the first object data is stored in the object storage.

[0104] Among them, in the embodiment of the present application, the target data corresponds to a preset size, that is, the size of the target data will not change, but the data storage state in the target data can change, such as the target data can be full of data or not full of data, for example, Fig.13 As shown, the preset size of the target data is 4M. Currently, the data has been written into four data blocks. Data block 1, data block 2, and data block 3 are in a full state, and data block 4 is not in a full state. 1M of data is written into data block 4.

[0105] For example, if the preset size of the target data is 4M, if the current target data stops writing data when 1M of data is just written because the file transfer is completed or the user triggers an upload operation, then the current target data that is not fully written can be determined as the first target data, that is, Fig.13 The data block 4 can be determined as the first target data. Then, corresponding first metadata information is constructed for the 1M data in the first target data, and first object data corresponding to the first target data is generated according to the first metadata information, and then the first object data is stored in the object storage.

[0106] 1. The normal sequential writing process in the embodiment of the present application is as follows:

[0107] For example, Fig.13 As shown, the user writes data streams into the data storage space through the client. The data stream flows from left to right. When 4M is filled in the data storage space, a data block is recorded. In the embodiment of the present application, the data block is an abstract concept and is only used to divide the data stream into fixed lengths. Fig.13 It can be concluded that data has been written into four data blocks, data block 1, data block 2, and data block 3 are in a full state, and data block 4 is in a partly full state.

[0108] After writing the full 4M length, metadata information 1 corresponding to data block 1, metadata information 2 corresponding to data block 2, metadata information 3 corresponding to data block 3, and metadata information 4 corresponding to data block 4 can be created accordingly. Among them, the metadata information contains the data block identifier BlockID for recording the step length, and the data block offset BlockOffset for recording the offset within the data block step length, etc. In the embodiment of the present application, the data block, metadata information, and object data are one-to-one corresponding. Since no overwriting behavior is generated, the object data corresponding to the metadata information is complete, and the object storage offset is also 0.

[0109] like Fig.13 As shown, in the metadata information 1 corresponding to data block 1, the data block identifier BlockID=0, the data block offset BlockOffset=0, the metadata size is 4M, and the object storage offset ObjectOffset=0; in the metadata information 2 corresponding to data block 2, the data block identifier BlockID=1, the data block offset BlockOffset=0, the metadata size is 4M, and the object storage offset ObjectOffset=0; in the metadata information 3 corresponding to data block 3, the data block identifier BlockID=2, the data block offset BlockOffset=0, the metadata size is 4M, and the object storage offset ObjectOffset=0; in the metadata information 4 corresponding to data block 4, the data block identifier BlockID=3, the data block offset BlockOffset=0, the metadata size is 1M, and the object storage offset ObjectOffset=0.

[0110] The metadata information generated in the embodiment of the present application can be asynchronously submitted to the metadata service module, and object data 1, object data 2, object data 3, and object data 4 can be asynchronously uploaded to the object storage module. When a whole file is written or the user initiates an upload operation, the client can call the submission interface to refresh and wait for all object data to be uploaded, and then submit the final metadata information to the metadata service module, thereby completing the entire process.

[0111] 2. The process of writing Flush in sequence in the embodiment of the present application is as follows:

[0112] For example, Fig.14 As shown, if the preset size of the target data is 4M, if the current target data stops writing data when 3M data is just written because the user triggers an upload operation, then the current target data that is not fully written can be determined as the first target data, that is, Fig.14 Data block 5 can be determined as the first target data. In order to persist the 3M data, the corresponding first metadata information, that is, metadata information 5, can be constructed for the 3M data in the first target data. In the metadata information 5, the data block identifier BlockID=0, the data block offset BlockOffset=0, the metadata size is 3M, and the object storage offset ObjectOffset=0. Then, according to the metadata information 5, the first object data corresponding to the first target data is generated, that is, object data 5, the size of object data 5 is 3M, and then the object data 5 is stored in the object storage, and the metadata information 5 is submitted to the metadata service module. The data after the upload operation is immediately visible to the outside.

[0113] Optionally, in one embodiment, after the step of “generating first object data corresponding to the first target data according to the first metadata information, and storing the first object data in the object storage”, the following may also be included:

[0114] Writing the data to be written into the remaining space of the first target data;

[0115] Constructing second metadata information corresponding to the first target data based on the data to be written in the remaining space of the first target data;

[0116] According to the second metadata information, second object data corresponding to the first target data is generated, and the second object data is stored in the object storage.

[0117] Among them, in the embodiment of the present application, continuing to write data at the position after the upload operation is POSIX append additional writing semantics. Since 3M of data has been stored in the first target data, the remaining 1M space is the remaining space of the first target data, and the data to be written can be written into the remaining space of the first target data, that is, writing 1M of data. Then, 1M of second metadata information is generated for the 1M data, that is, metadata information 6, in which BlockID=0, data block offset BlockOffset=3M, metadata size is 1M, and object storage offset ObjectOffset=0, wherein, since it corresponds to new object data, the object storage offset is 0.

[0118] This ensures that the object data and metadata information generated by each upload operation are newly generated. Uploading and submitting are both overwriting operations. There is no need to first pull, then update, and then submit. This reduces the interaction between data flow and metadata information and improves operation efficiency.

[0119] 3. The random write without overwriting process in the embodiment of the present application is as follows:

[0120] Optionally, in one embodiment, after the step of “generating first object data corresponding to the first target data according to the first metadata information, and storing the first object data in the object storage”, the following may also be included:

[0121] Obtaining a first random data stream to be stored and a first write offset;

[0122] Based on the historical storage information of the data storage space and the first write offset, the first random data stream is segmented to obtain a plurality of first random data;

[0123] Based on the first write offset, the first random data is written into the data storage space to obtain third target data corresponding to each first random data, wherein there is no overlap between the first random data and the data to be written that has been written into the first target data;

[0124] Based on the first random data, construct third metadata information corresponding to the third target data;

[0125] According to the third metadata information, third object data corresponding to the third target data is generated, and the third object data is stored in the object storage.

[0126] For example, the present application can also efficiently support POSIX random write semantics, where random write is mainly divided into two situations: one is a situation where no data overwrite occurs; the other is a situation where data overwrite occurs, where the situation where data overwrite occurs is further divided into two situations: one is that the data of the overwritten part exists in a data block; the other is that the data of the overwritten part has an intersection with two or more data blocks.

[0127] The situations where data coverage is not generated in this application may be as follows: for example, Fig.16 As shown, data of size 3M has been written into data block 5, and the upload instruction has been triggered to perform data upload operation. At this time, data of size 3M already exists in data block 5, wherein data block 5 corresponds to metadata information 5 and object data 5. Then a data write request from the client can be received, and according to the data write request, it can be determined that the size of the first random data stream to be stored is 5M, and the first write offset is 7M, that is, the distance between the starting point of data block 5 and the position point where the first random data stream is written is 7M, and the size of the first random data stream written is 5M. After writing 5M of data, you can continue to write randomly on demand, or you can submit data through the upload operation, that is, generate a 12M data.

[0128] Depend on Fig.16 It can be seen that the embodiment of the present application writes 1M of data into data block 7 and writes 4M of data into data block 8. Fig.16 The shaded part in the , because no valid data is written, is the file black hole in POSIX semantics. This part has no valid data, but it is included in the entire data size. This part is usually filled with byte0. Similarly, if this part of data is read when reading data, byte0 is returned.

[0129] 4. The random write overlay process in the embodiment of the present application is as follows:

[0130] Optionally, in one embodiment, after the step of “generating object data corresponding to the target data according to the metadata information, and storing the object data in the object storage”, the following may also be included:

[0131] Obtaining a second random data stream to be stored and a second write offset;

[0132] Based on the historical storage information of the data storage space and the second write offset, writing the second random data stream completely into the target data of the data storage space;

[0133] Constructing fourth metadata information corresponding to the target data based on the second random data stream;

[0134] Fourth object data corresponding to the target data is generated according to the fourth metadata information, and the fourth object data is stored in the object storage.

[0135] For example, the processing flow for random write with overwriting and the overwritten data part is within a data block is as follows Fig.17 As shown, the core is to segment the existing metadata information according to the coverage range, reuse part of the old object data for the part without coverage data, and generate new metadata information and new object data for the new data.

[0136] like Fig.17 As shown, it is known that a 4M data block 1 has been fully written, and the data block 1 corresponds to metadata information 1 and object data 1. It can be determined based on the acquired data write request that the size of the second random data stream currently to be written is 2M, and the second write offset for data block 1 is 1M, then the 2M second random data stream can be rewritten at the position with an offset of 1M in data block 1, and the written 2M data will generate a new fourth metadata information (that is, metadata information 9) with a size of 2M, and fourth object data (that is, object data 9), and the old metadata information 1 will be divided into two parts, both of which will generate new metadata information 10, but in order to improve the utilization rate of object storage data, the two parts of the new metadata information 10 will reuse the old object data 1, but the offset and effective size of the object data 1 will change. The metadata information 10 on the left corresponds to an object data offset of 0, a size of 1M, and a valid range of 0-1023 for object data 1; while the metadata information 10 on the right corresponds to an object data offset of 3M, a size of 1M, and a valid range of 3145728-4194303 for object data 1. Finally, the newly generated metadata information 10 and metadata information 9 are submitted to the metadata service module.

[0137] Optionally, in one embodiment, after the step of “generating object data corresponding to the target data according to the metadata information, and storing the object data in the object storage”, the following may also be included:

[0138] Obtaining a third random data stream to be stored and a third write offset;

[0139] Based on the historical storage information of the data storage space and the third write offset, overwrite the third random data stream into the continuous target data of the data storage space;

[0140] Based on the third random data stream, construct fifth metadata information corresponding to the target data located at the first position in the continuous target data, and sixth metadata information corresponding to the target data located at the last position in the continuous target data;

[0141] Generate fifth object data corresponding to the first target data according to the fifth metadata information, and store the fifth object data in the object storage;

[0142] According to the sixth metadata information, sixth object data corresponding to the target data at the last position is generated, and the sixth object data is stored in the object storage.

[0143] For example, the random write has an overwrite, and the overwritten data part is in multiple data blocks. Fig.18 As shown, it is known that two consecutive data blocks 1 and data block 2 have been submitted before, data block 1 corresponds to metadata information 1 and object data 1, and data block 2 corresponds to metadata information 2 and object data 2. It can be determined according to the acquired data write request that the size of the third random data stream currently to be written is 2M, and the third write offset for data block 1 is 3M, then the 2M third random data stream can be rewritten at the position of offset 3M in data block 1, then the 2M third random data stream exists in data block 1 and data block 2 at the same time, that is, based on the third random data stream, the fifth metadata information (that is, metadata information 11) and the fifth object data (that is, object data 11) corresponding to data block 1, the sixth metadata information (that is, metadata information 12) and the sixth object data (that is, object data 12) corresponding to data block 2 can be constructed, wherein, since the metadata information is bound to the data block, there will be no metadata information across data blocks.

[0144] Data block 1 will be split into two parts. The left part of data block 1 after being split needs to generate new metadata information 13, but the old object data 1 is reused. The corresponding old object data 1 has an offset of 0, a size of 3M, and a valid range of 0-3145727. Data block 2 will be split into two parts. The right part of data block 2 after being split needs to generate new metadata information 13, but the old object data 2 is reused. The corresponding old object data 2 has an offset of 1M, a size of 3M, and a valid range of 1048576-4194303. The embodiment of the present application is not limited to the covered data part being in two data blocks, and the covered data part can also be in multiple data blocks.

[0145] 205. When a data read request sent by the terminal is received, the target object data is read from the object storage, and the target object data is sent to the terminal.

[0146] Optionally, in one embodiment, the step of “when receiving a data read request sent by a terminal, reading target object data in the object storage and sending the target object data to the terminal” may include:

[0147] When receiving a data reading request sent by the terminal, determining target metadata information corresponding to the data to be read based on the data reading request;

[0148] Based on the target metadata information, read the target object data corresponding to the data to be read in the object storage;

[0149] Send the target object data to the terminal.

[0150] For example, when a data reading request is received from a terminal, the file handle, file size, location to be read, and size of the data to be read corresponding to the file to be read can be first obtained through the metadata service module. Then, based on the above information, the target metadata information corresponding to the data to be read can be obtained through the metadata service module. Then, according to the object storage index recorded in the target metadata information, the target object data corresponding to the data to be read is read in the object storage, and finally the target object data is sent to the terminal to complete the reading step.

[0151] For example, according to Fig.15 The steps shown write 13M of data. At this time, there are already data block 1 with a size of 4M, data block 2 with a size of 4M, data block 3 with a size of 4M, and data block 4 with a size of 1M. Each data block corresponds to a metadata information and an object data.

[0152] Then, you can read the data that has been written, such as Fig.15As shown, according to the data read request, the embodiment of the present application can know that the size of the file to be read is 13M, and it needs to be read from the offset 5M position, and the size of the data to be read is 8M. Fig.15 As can be seen from the above, it is necessary to start reading from the 1M position of data block 2, and respectively read the object data (size is 3M) of the metadata information 2 of the object data 2 corresponding to data block 2, which is 1024-4194303, the complete object data 3 (size is 4M) corresponding to data block 3, and the complete object data 4 (size is 1M) corresponding to data block 4. In the embodiment of the present application, partial object data can be obtained through a data read request, so that the required object data can be obtained no matter where the data is read.

[0153] Optionally, in one embodiment, after the step of “generating object data corresponding to the target data according to the metadata information, and storing the object data in the object storage”, the following may also be included:

[0154] Obtaining a data truncation request, and determining a data truncation position based on the data truncation request;

[0155] Determine the target data at the data truncation position;

[0156] Based on the offset of the data truncation position in the target data, metadata information corresponding to the target data at the data truncation position is adjusted.

[0157] The embodiment of the present application may also support a data truncation function (truncate), which can be divided into two situations: one is that the length of the truncated data is greater than the original length of the data; the other is that the length of the truncated data is less than the original length of the data.

[0158] For example, if the length of the data to be truncated is greater than the original length of the data, there is no need to create any new metadata information or object information, and only the file node attributes need to be changed. Fig.19 As shown in the figure, the original length of the data is 5M, and the length of the data to be truncated is 7M. Then the node size can be updated to 7M. ​​When reading the second data block, only a 1M metadata structure can be obtained, and the remaining 2M shaded part is automatically filled with byte0. Only a unary operation is needed to meet the needs of this scenario.

[0159] For example, if the length of the data to be truncated is less than the original length of the data, such as Fig. 20As shown in the figure, the data truncation position is in the middle of a data block, and the original 4M data block is truncated to 2M. The original data block includes three metadata information, namely 1M, 2M, and 1M. The data truncation position is in the middle of the second old metadata information, which will be split and a new 1M valid metadata information will be created, while the size of the original data will be changed. The remaining 2M metadata information and the last 1M data are handed over to the GC module for asynchronous processing.

[0160] The data truncation method in the embodiment of the present application is similar to the random write method. The data block where the data truncation position is located is determined by the data truncation position, and new metadata information is created and submitted according to the offset of the data truncation position in the target data to overwrite the original metadata information. At the same time, the newly generated metadata information can also reuse part of the data in the original object data. It is not necessary to read the data stream and rewrite the data. It only needs to change the new metadata information and map it to the old object data. Fig. 20 As shown, the newly generated 1M metadata information corresponds to the old object data offset of 0, and the object data size is 1M (2M in total), thereby avoiding the interaction of data streams and improving the performance of metadata operations.

[0161] In addition, it can be seen from the above embodiments that frequent random writing and uploading of data will generate a large amount of metadata information. As the metadata information increases, a large amount of metadata information needs to be obtained from the metadata service module when the client reads it. In order to improve the resource utilization of the metadata service module and reduce the metadata interaction cost and request latency, the embodiments of this application provide two compression methods. The specific metadata compression and data merging process is as follows Figure 3-12 As shown, the core is to merge multiple metadata information in a data block into a new metadata information and merge the object data into one.

[0162] One is that the client request carries a check mark. If the backend turns on this mark when submitting, it will check the number of metadata information. If it exceeds a certain amount of data (16 by default), the batch of metadata information will be returned to the client. The client can reorganize the data because it has data cache, merge the metadata information, and submit it to the metadata service module. This method can utilize client data resources, but the disadvantage is that the client latency will be affected if compression is triggered.

[0163] Therefore, there is another method in which the backend asynchronously merges the data blocks with more metadata information and only updates them. The original metadata information and the corresponding object data are asynchronously cleaned up by the GC module. The advantage of this method is that it has no impact on the client request latency, but the backend needs to bear the cost of data traffic.

[0164] According to the above embodiments, it can be found that frequent random writes and data uploads will generate a large amount of metadata information, and improving resource utilization has strict requirements on compression strategies. It is also difficult to achieve both low latency and reduced traffic costs for data merging and metadata compression. It is necessary to make a trade-off based on actual needs, that is, to improve the efficiency of the compression algorithm while meeting low latency.

[0165] The technical solution of this application can effectively utilize the convenience of object storage data operations and broaden the scenarios of object storage in the field of file systems. While satisfying various POSIX semantic operations in a structured and efficient manner, it makes up for the various pain points and shortcomings of prior art object storage in file system semantics. Renaming and deleting directories only requires one interface call while satisfying atomicity. Due to the specific directory tree structure, there is no need to use Head+List to determine whether a node is a file or a directory, which ensures atomicity while providing efficient access. The metadata mapping method can effectively avoid data stream interaction and use object storage Restful http range to reuse data as much as possible. The metadata uses append overwrite, which effectively reduces the latency of the metadata interface.

[0166] In the existing technology, since object storage is a typical KV structure, there is no concept of directory and it is a typical flat structure. Therefore, the common pain points for supporting POSIX semantics mainly include:

[0167] 1) When renaming a directory, it is necessary to recursively traverse the subdirectories and perform COPY+Delete, which is a non-atomic operation. It requires a large number of interface calls and cannot guarantee atomicity.

[0168] 2) When deleting a directory, you need to recursively traverse the subdirectories and call to delete all keys under the specified prefix one by one.

[0169] 3) When obtaining whether the path is a file or a directory, you need to use Head / a / b, Head / a / b / , and list max-keys=1 to determine whether it exists. For example, you can use PutObject / a / b / c.txt to directly create a file, but object storage will not recursively create a target directory, that is, / a / and / a / b / are non-existent keys. Therefore, if you want to determine whether b is a directory, you can only use list / a / to see whether the returned result contains / a / b / to determine whether b is a directory.

[0170] 4) Object storage has list, head, upload and download frequency control. A single Posix semantic interface may contain multiple object storage interface calls, resulting in request amplification.

[0171] Therefore, in order to solve the shortcomings of object storage in POSIX file system semantics, it is necessary to broaden the use scenarios of object storage in the file system field to meet the needs of scenarios such as Hadoop HCFS and machine learning FUSE. This application designs a set of metadata management methods based on object storage. By constructing a mapping between directory tree nodes and objects, it manages object storage metadata information, solves the above pain points, and can do the following:

[0172] 1) Rename. Since there is a concept of directory tree node, only one call is needed to change the node parent directory, which satisfies the atomic operation.

[0173] 2) Deleting a directory is also a node in the tree. Simply delete the file or directory node from the tree and move it to the GC table for asynchronous deletion while ensuring consistency.

[0174] 3) Since each node in the directory tree has an attr attribute, it can be directly used to determine whether it is a file or a directory;

[0175] 4) The metadata access speed can be improved by caching the directory tree structure to the cache.

[0176] As can be seen from the above, this embodiment can obtain the current data stream to be stored, and based on the historical storage information of the data storage space, the current data stream is segmented to obtain multiple data to be written; the data to be written is written into the data storage space to obtain the target data corresponding to each data to be written; based on the data to be written, the metadata information corresponding to the target data is constructed; according to the metadata information, the object data corresponding to the target data is generated, and the object data is stored in the object storage; when the data read request sent by the terminal is received, the target object data is read in the object storage, and the target object data is sent to the terminal. This application can use the object storage module to manage array resources, and abstract the metadata service module to manage metadata logic. In addition, for the flat KV structure of the object storage, the directory tree is abstracted, and the mapping relationship between the node and the data block is constructed. The technical solution of this application can support POSIX semantic access to object storage data while achieving efficient reading and writing of files, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. This method can achieve strong consistency of data, and can be expanded in parallel to provide high availability and high reliability service guarantees for the metadata service module. Moreover, this method can achieve storage and computing separation and is applicable to multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios.

[0177] According to the method described in the previous embodiment, the following will be further described in detail by taking the data processing device as an example of being specifically integrated into an electronic device. The present application embodiment provides a data processing method, such as Figure 3As shown, the specific process of the data processing method can be as follows:

[0178] 301. The electronic device obtains a current data stream to be stored, and divides the current data stream based on historical storage information of the data storage space to obtain data to be written.

[0179] For example, a user may write a data stream to the data storage space through a client, and the historical storage information of the data storage space indicates that no data is stored in the current data storage space. In this case, the current data stream may be divided into units of 4M, and the current data stream may be divided into data 1 to be written, data 2 to be written, data 3 to be written, and data 4 to be written, wherein the size of data 1 to be written is 4M, the size of data 2 to be written is 4M, the size of data 3 to be written is 4M, and the size of data 4 to be written is 1M.

[0180] 302. The electronic device writes the data to be written into the data storage space to obtain a data block corresponding to each data to be written.

[0181] For example, Fig.13 As shown, data 1 to be written, data 2 to be written, data 3 to be written, and data 4 to be written can be written into the data storage space in sequence, and data block 1 corresponding to data 1 to be written, data block 2 corresponding to data 2 to be written, data block 3 corresponding to data 3 to be written, and data block 4 corresponding to data 4 to be written are obtained, wherein data block 1, data block 2, and data block 3 are in a full state, and data block 4 is in a not full state.

[0182] 303. The electronic device constructs metadata information corresponding to the data block based on the data to be written.

[0183] For example, after writing the full 4M length, metadata information corresponding to the data block can be created accordingly, and the metadata information includes the data block identifier BlockID for recording the step length, and the data block offset BlockOffset for recording the offset within the data block step length, etc. In the embodiment of the present application, the data block, metadata information, and object data are one-to-one corresponding. Since no overwriting behavior occurs, the object data corresponding to the metadata information is complete, and the object storage offset is also 0.

[0184] like Fig.13As shown, in metadata information 1 corresponding to data block 1, data block identifier BlockID=0, data block offset BlockOffset=0, metadata size is 4M, and object storage offset ObjectOffset=0; in metadata information 2 corresponding to data block 2, data block identifier BlockID=1, data block offset BlockOffset=0, metadata size is 4M, and object storage offset ObjectOffset=0; in metadata information 3 corresponding to data block 3, data block identifier BlockID=2, data block offset BlockOffset=0, metadata size is 4M, and object storage offset ObjectOffset=0; in metadata information 4 corresponding to data block 4, data block identifier BlockID=3, data block offset BlockOffset=0, metadata size is 1M, and object storage offset ObjectOffset=0. Then metadata information 1, metadata information 2, metadata information 3, and metadata information 4 can be asynchronously submitted to the metadata service module.

[0185] 304. The electronic device generates object data corresponding to the data block according to the metadata information, and stores the object data in the object storage.

[0186] For example, object data 1 corresponding to data block 1 can be generated based on metadata information 1, object data 2 corresponding to data block 2 can be generated based on metadata information 2, object data 3 corresponding to data block 3 can be generated based on metadata information 3, and object data 4 corresponding to data block 4 can be generated based on metadata information 4, and then object data 1, object data 2, object data 3, and object data 4 can be asynchronously uploaded to the object storage module.

[0187] 305. When receiving the data reading request sent by the terminal, the electronic device reads the target object data in the object storage and sends the target object data to the terminal.

[0188] For example, Fig.15 As shown, according to the data read request from the terminal, it can be known that the size of the file to be read is 13M, it needs to be read from the offset 5M position, and the size of the data to be read is 8M. Fig.15 From the above, it can be seen that it is necessary to start reading from the 1M position of data block 2, and read the object data (size is 3M) of the metadata information 2 of the object data 2 corresponding to data block 2, which is 1024-4194303, the complete object data 3 (size is 4M) corresponding to data block 3, and the complete object data 4 (size is 1M) corresponding to data block 4.

[0189] As can be seen from the above, this embodiment can obtain the current data stream to be stored through the electronic device, and based on the historical storage information of the data storage space, the current data stream is segmented to obtain the data to be written; the data to be written is written into the data storage space to obtain the data block corresponding to each data to be written; based on the data to be written, the metadata information corresponding to the data block is constructed; according to the metadata information, the object data corresponding to the data block is generated, and the object data is stored in the object storage; when the data read request sent by the terminal is received, the target object data is read in the object storage, and the target object data is sent to the terminal. This application can use the object storage module to manage array resources, and abstract the metadata service module to manage metadata logic. In addition, for the flat KV structure of the object storage, the directory tree is abstracted, and the mapping relationship between the node and the data block is constructed. The technical solution of this application can support POSIX semantic access to object storage data while achieving efficient reading and writing of files, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. This method can achieve strong consistency of data, and can be expanded in parallel to provide high availability and high reliability service guarantees for the metadata service module. Moreover, this method can achieve storage and computing separation and is applicable to multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios.

[0190] According to the method described in the previous embodiment, the following will be further described in detail by taking the data processing device as an example of being specifically integrated into an electronic device. The present application embodiment provides a data processing method, such as Figure 4 As shown, the specific process of the data processing method can be as follows:

[0191] 401 . If data writing is stopped before the data to be written fills up the data block 5 , the electronic device constructs metadata information 5 corresponding to the data block 5 .

[0192] For example, Fig.14 As shown, if data writing is stopped because the user triggers an upload operation when 3M of data to be written is written in data block 5, in order to persist the 3M data, metadata information 5 corresponding to the 3M data in data block 5 can be constructed, in which the data block identifier BlockID=0, the data block offset BlockOffset=0, the metadata size is 3M, and the object storage offset ObjectOffset=0.

[0193] 402. The electronic device generates object data 5 corresponding to the data block 5 according to the metadata information 5, and stores the object data 5 in the object storage.

[0194] For example, after determining the metadata information 5, object data 5 corresponding to the data block 5 can be generated based on the metadata information 5. The size of the object data 5 is 3M. ​​Then the object data 5 is stored in the object storage, and the metadata information 5 is submitted to the metadata service module. The data after the upload operation is immediately visible to the outside.

[0195] 403 . The electronic device writes the data to be written into the remaining space of the data block 5 , and constructs metadata information 6 corresponding to the data block 5 .

[0196] For example, the data to be written can be continued to be written to the remaining space in the data block 5, that is, data of 1M size can be written. Then, the metadata information 6 corresponding to the 1M data in the data block 5 is constructed, in which BlockID=0, data block offset BlockOffset=3M, metadata size is 1M, and object storage offset ObjectOffset=0, wherein the object storage offset is 0 because it corresponds to new object data.

[0197] 404. The electronic device generates object data 6 corresponding to the data block 5 according to the metadata information 6, and stores the object data 6 in the object storage.

[0198] This ensures that the object data and metadata information generated by each upload operation are newly generated. Uploading and submitting are both overwriting operations. There is no need to first pull, then update, and then submit. This reduces the interaction between data flow and metadata information and improves operation efficiency.

[0199] 405. When receiving the data reading request sent by the terminal, the electronic device reads the target object data in the object storage and sends the target object data to the terminal.

[0200] As can be seen from the above, if the data writing is stopped when the data to be written is not filled with the data block 5, the electronic device can construct the metadata information 5 corresponding to the data block 5; according to the metadata information 5, the object data 5 corresponding to the data block 5 is generated, and the object data 5 is stored in the object storage; the data to be written is written into the remaining space of the data block 5, and the metadata information 6 corresponding to the data block 5 is constructed; according to the metadata information 6, the object data 6 corresponding to the data block 5 is generated, and the object data 6 is stored in the object storage; when the data read request sent by the terminal is received, the target object data is read in the object storage, and the target object data is sent to the terminal. The present application can use the object storage module to manage array resources, and abstract the metadata service module to manage metadata logic. In addition, for the flat KV structure of the object storage, the directory tree is abstracted, and the mapping relationship between the node and the data block is constructed. The technical solution of the present application can support POSIX semantic access to the object storage data while achieving efficient reading and writing of files, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. The method can achieve strong consistency of data, and can be expanded in parallel to provide high availability and high reliability service guarantees for the metadata service module. Moreover, this method can achieve storage and computing separation and is applicable to multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios.

[0201] According to the method described in the previous embodiment, the following will be further described in detail by taking the data processing device as an example of being specifically integrated into an electronic device. The present application embodiment provides a data processing method, such as Figure 5 As shown, the specific process of the data processing method can be as follows:

[0202] 501 . If data writing is stopped when the data to be written has not yet filled up the data block 5 , the electronic device constructs metadata information 5 corresponding to the data block 5 .

[0203] For example, Fig.16 As shown, if data writing is stopped because the user triggers an upload operation when 3M of data to be written is written in data block 5, in order to persist the 3M data, metadata information 5 corresponding to the 3M data in data block 5 can be constructed, in which the data block identifier BlockID=0, the data block offset BlockOffset=0, the metadata size is 3M, and the object storage offset ObjectOffset=0.

[0204] 502. The electronic device generates object data 5 corresponding to the data block 5 according to the metadata information 5, and stores the object data 5 in the object storage.

[0205] For example, after determining the metadata information 5, object data 5 corresponding to the data block 5 can be generated based on the metadata information 5. The size of the object data 5 is 3M. ​​Then the object data 5 is stored in the object storage, and the metadata information 5 is submitted to the metadata service module. The data after the upload operation is immediately visible to the outside.

[0206] 503. The electronic device obtains a first random data stream to be stored and a first write offset.

[0207] For example, a data write request may be received from a client, and according to the data write request, it may be determined that the size of the first random data stream to be stored is 5M, and the first write offset is 7M.

[0208] 504. The electronic device writes the first random data stream into data block 7 and data block 8 without overwriting.

[0209] For example, Fig.16 As shown, the first random data stream can be written into part of data block 7 and all of data block 8, that is, the distance between the starting point of data block 5 and the position point where the first random data stream is written is 7M. After writing 5M of data, you can continue to write randomly on demand, or submit data through an upload operation, that is, generate 12M of data.

[0210] 505. The electronic device constructs metadata information 7 corresponding to data block 7 and metadata information 8 corresponding to data block 8.

[0211] 506. The electronic device generates object data 7 corresponding to the data block 7 according to the metadata information 7, and stores the object data 7 in the object storage.

[0212] 507. The electronic device generates object data 8 corresponding to the data block 8 according to the metadata information 8, and stores the object data 8 in the object storage.

[0213] For example, by Fig.16 It can be seen that the embodiment of the present application writes 1M of data into data block 7 and writes 4M of data into data block 8. Fig.16 The shaded part in the , because no valid data is written, is the file black hole in POSIX semantics. This part has no valid data, but it is included in the entire data size. This part is usually filled with byte0. Similarly, if this part of data is read when reading data, byte0 is returned.

[0214] 508. When receiving the data reading request sent by the terminal, the electronic device reads the target object data in the object storage and sends the target object data to the terminal.

[0215] As can be seen from the above, if the data writing is stopped when the data to be written has not yet filled the data block 5, the electronic device can construct the metadata information 5 corresponding to the data block 5; according to the metadata information 5, generate the object data 5 corresponding to the data block 5, and store the object data 5 in the object storage; obtain the first random data stream to be stored, and the first write offset; write the first random data stream into the data block 7 and the data block 8 without overwriting; construct the metadata information 7 corresponding to the data block 7, and the metadata information 8 corresponding to the data block 8; generate the object data 7 corresponding to the data block 7 according to the metadata information 7, and store the object data 7 in the object storage; generate the object data 8 corresponding to the data block 8 according to the metadata information 8, and store the object data 8 in the object storage; when receiving the data read request sent by the terminal, read the target object data in the object storage, and send the target object data to the terminal. This application can use the object storage module to manage array resources, and abstract the metadata service module to manage metadata logic. In addition, for the flattened KV structure of the object storage, the directory tree is abstracted, and the mapping relationship between nodes and data blocks is constructed. The technical solution of this application can support POSIX semantic access to object storage data while achieving efficient file reading and writing, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. This method can achieve strong data consistency and can be expanded in parallel to provide high availability and high reliability service guarantees for metadata service modules. In addition, this method can achieve storage and computing separation and is applicable to multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios.

[0216] According to the method described in the previous embodiment, the following will be further described in detail by taking the data processing device as an example of being specifically integrated into an electronic device. The present application embodiment provides a data processing method, such as Figure 6 As shown, the specific process of the data processing method can be as follows:

[0217] 601. The electronic device obtains a current data stream to be stored, and divides the current data stream based on historical storage information of the data storage space to obtain data to be written.

[0218] For example, a user can write a data stream to the data storage space through the client, and the historical storage information of the data storage space shows that no data is stored in the current data storage space. In this case, the current data stream can be divided into data 1 to be written in units of 4M, where the size of data 1 to be written is 4M.

[0219] 602. The electronic device writes the data to be written into the data storage space to obtain a data block corresponding to each data to be written.

[0220] For example, Fig.17As shown, the data 1 to be written can be written into the data storage space, and a data block 1 corresponding to the data 1 to be written can be obtained.

[0221] 603. The electronic device constructs metadata information corresponding to the data block based on the data to be written.

[0222] For example, in metadata information 1 corresponding to data block 1, data block identifier BlockID=0, data block offset BlockOffset=0, metadata size is 4M, and object storage offset ObjectOffset=0. Then, metadata information 1 can be submitted to the metadata service module.

[0223] 604. The electronic device generates object data corresponding to the data block according to the metadata information, and stores the object data in the object storage.

[0224] For example, object data 1 corresponding to data block 1 may be generated according to metadata information 1, and then object data 1 may be uploaded to the object storage module.

[0225] 605. The electronic device obtains a second random data stream to be stored and a second write offset.

[0226] For example, according to the acquired data write request, it can be determined that the size of the second random data stream currently to be written is 2M, and the second write offset for data block 1 is 1M.

[0227] 606. The electronic device writes the second random data stream completely into data block 1 of the data storage space based on the historical storage information of the data storage space and the second write offset.

[0228] For example, a second random data stream of 2 MB can be rewritten at an offset of 1 MB in data block 1. Fig.17 The shaded area represents the second random data stream that is written.

[0229] 607. The electronic device constructs metadata information 9 corresponding to the data block 1 based on the second random data stream.

[0230] 608. The electronic device generates object data 9 corresponding to the data block 1 according to the metadata information 9, and stores the object data 9 in the object storage.

[0231] For example, the written 2M data will generate new metadata information 9 with a size of 2M and object data 9, and the old metadata information 1 will be divided into two parts, both of which will generate new metadata information 10. However, in order to improve the utilization rate of object storage data, the two parts of the new metadata information 10 will reuse the old object data 1, but the offset and effective size of object data 1 will change. The new metadata information 10 on the left side corresponds to the offset of object data 1 of 0, the size of 1M, and the effective range of object data 1 is 0-1023; while the new metadata information 10 on the right side corresponds to the offset of object data 1 of 3M, the size of 1M, and the effective range of object data 1 is 3145728-4194303. Finally, the newly generated metadata information 10 and metadata information 9 are submitted to the metadata service module.

[0232] 609. When receiving the data read request sent by the terminal, the electronic device reads the target object data in the object storage and sends the target object data to the terminal.

[0233] As can be seen from the above, this embodiment can obtain the current data stream to be stored through the electronic device, and based on the historical storage information of the data storage space, the current data stream is segmented to obtain the data to be written; the data to be written is written into the data storage space to obtain the data block corresponding to each data to be written; based on the data to be written, the metadata information corresponding to the data block is constructed; according to the metadata information, the object data corresponding to the data block is generated, and the object data is stored in the object storage; the second random data stream to be stored and the second write offset are obtained; based on the historical storage information of the data storage space and the second write offset, the second random data stream is completely overwritten and written into the data block 1 of the data storage space; based on the second random data stream, the metadata information 9 corresponding to the data block 1 is constructed; according to the metadata information 9, the object data 9 corresponding to the data block 1 is generated, and the object data 9 is stored in the object storage; when a data read request sent by the terminal is received, the target object data is read in the object storage, and the target object data is sent to the terminal. This application can use the object storage module for array resource management, and abstract the metadata service module for metadata logic management. In addition, for the flattened KV structure of the object storage, the directory tree is abstracted, and the mapping relationship between the node and the data block is constructed. The technical solution of this application can support POSIX semantic access to object storage data while achieving efficient file reading and writing, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. This method can achieve strong data consistency and can be expanded in parallel to provide high availability and high reliability service guarantees for metadata service modules. In addition, this method can achieve storage and computing separation and is applicable to multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios.

[0234] According to the method described in the previous embodiment, the following will be further described in detail by taking the data processing device as an example of being specifically integrated into an electronic device. The present application embodiment provides a data processing method, such as Figure 7 As shown, the specific process of the data processing method can be as follows:

[0235] 701. The electronic device obtains a current data stream to be stored, and divides the current data stream based on historical storage information of the data storage space to obtain data to be written.

[0236] For example, a user may write a data stream to a data storage space through a client, and the historical storage information of the data storage space indicates that no data is stored in the current data storage space. In this case, the current data stream may be divided into data 1 to be written and data 2 to be written, with the sizes of both data 1 to be written and data 2 to be written being 4M.

[0237] 702. The electronic device writes the data to be written into the data storage space to obtain a data block corresponding to each data to be written.

[0238] For example, Fig.18 As shown, the data to be written 1 and the data to be written 2 can be written into the data storage space, and a data block 1 corresponding to the data to be written 1 and a data block 2 corresponding to the data to be written 2 are obtained.

[0239] 703. The electronic device constructs metadata information corresponding to the data block based on the data to be written.

[0240] For example, in metadata information 1 corresponding to data block 1, data block identifier BlockID=0, data block offset BlockOffset=0, metadata size is 4M, and object storage offset ObjectOffset=0; in metadata information 2 corresponding to data block 2, data block identifier BlockID=1, data block offset BlockOffset=0, metadata size is 4M, and object storage offset ObjectOffset=0. Then metadata information 1 and metadata information 2 can be submitted to the metadata service module.

[0241] 704. The electronic device generates object data corresponding to the data block according to the metadata information, and stores the object data in the object storage.

[0242] For example, object data 1 corresponding to data block 1 may be generated according to metadata information 1, object data 2 corresponding to data block 2 may be generated according to metadata information 2, and then object data 1 and object data 2 may be uploaded to the object storage module.

[0243] 705. The electronic device obtains a third random data stream to be stored and a third write offset.

[0244] For example, it can be determined according to the acquired data write request that the size of the third random data stream currently to be written is 2M, and the third write offset for data block 1 is 3M.

[0245] 706. The electronic device writes the third random data stream over the data block 1 and the data block 2 of the data storage space based on the historical storage information of the data storage space and the third write offset.

[0246] For example, a third random data stream of 2M may be rewritten at a position with an offset of 3M in data block 1, and then the third random data stream of 2M exists in both data block 1 and data block 2.

[0247] 707. The electronic device constructs metadata information 11 corresponding to data block 1 and metadata information 12 corresponding to data block 2 based on the third random data stream.

[0248] 708. The electronic device generates object data 11 corresponding to the data block 1 according to the metadata information 11, and stores the object data 11 in the object storage.

[0249] 709. The electronic device generates object data 12 corresponding to the data block 2 according to the metadata information 12, and stores the object data 12 in the object storage.

[0250] For example, metadata information 11 and object data 11 corresponding to the third random data stream in data block 1 can be generated, and metadata information 12 and object data 12 corresponding to the third random data stream in data block 2 can be generated. Since the metadata information is bound to the data block, there will be no metadata information across data blocks.

[0251] Data block 1 will be split into two parts. The left part needs to generate new metadata information 13, but reuses part of the old object data 1. The corresponding old object data 1 has an offset of 0, a size of 3M, and a valid range of 0-3145727. Data block 2 will be split into two parts. The right part needs to generate new metadata information 13, but reuses part of the old object data 2. The corresponding old object data 2 has an offset of 1M, a size of 3M, and a valid range of 1048576-4194303.

[0252] 710. When receiving a data read request sent by a terminal, the electronic device reads target object data in the object storage and sends the target object data to the terminal.

[0253] As can be seen from the above, in this embodiment, the current data stream to be stored can be obtained by an electronic device, and based on the historical storage information of the data storage space, the current data stream is segmented to obtain the data to be written; the data to be written is written into the data storage space to obtain the data block corresponding to each data to be written; based on the data to be written, metadata information corresponding to the data block is constructed; according to the metadata information, object data corresponding to the data block is generated, and the object data is stored in the object storage; the third random data stream to be stored and the third write offset are obtained; based on the historical storage information of the data storage space and the third write offset, the third random data stream is overwritten and written into the data block 1 and the data block 2 of the data storage space; based on the third random data stream, metadata information 11 corresponding to the data block 1 and metadata information 12 corresponding to the data block 2 are constructed; according to the metadata information 11, object data 11 corresponding to the data block 1 is generated, and the object data 11 is stored in the object storage; according to the metadata information 12, object data 12 corresponding to the data block 2 is generated, and the object data 12 is stored in the object storage; when a data read request sent by the terminal is received, the target object data is read in the object storage, and the target object data is sent to the terminal. This application can use the object storage module to manage array resources, and abstract the metadata service module to manage metadata logic. In addition, for the flat KV structure of object storage, the directory tree is abstracted, and the mapping relationship between nodes and data blocks is constructed. The technical solution of this application can support POSIX semantic access to object storage data while achieving efficient reading and writing of files, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. This method can achieve strong data consistency and can be expanded in parallel to provide high-availability and high-reliability service guarantees for the metadata service module. In addition, this method can achieve storage and computing separation, and is suitable for multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios.

[0254] In order to better implement the above method, the present application embodiment also provides a data processing device, such as Fig. 22 As shown, the data processing device may include an acquisition unit 2201, a writing unit 2202, a construction unit 2203, a storage unit 2204, and a reading unit 2205, as follows:

[0255] The acquisition unit 2201 is used to acquire the current data stream to be stored, and divide the current data stream based on the historical storage information of the data storage space to obtain multiple data to be written;

[0256] A writing unit 2202, used for writing the data to be written into the data storage space to obtain target data corresponding to each data to be written;

[0257] A construction unit 2203 is used to construct metadata information corresponding to the target data based on the data to be written;

[0258] The storage unit 2204 is used to generate object data corresponding to the target data according to the metadata information, and store the object data in an object storage;

[0259] The reading unit 2205 is configured to read target object data from the object storage and send the target object data to the terminal when receiving a data reading request sent by the terminal.

[0260] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to stop writing data if the data to be written has not yet filled up the target data in the data storage space, and determine the target data that has not yet been filled up as the first target data; based on the data to be written that has been written in the first target data, construct first metadata information corresponding to the first target data; based on the first metadata information, generate first object data corresponding to the first target data, and store the first object data in the object storage.

[0261] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to write the data to be written into the remaining space of the first target data; based on the data to be written written into the remaining space of the first target data, construct second metadata information corresponding to the first target data; based on the second metadata information, generate second object data corresponding to the first target data, and store the second object data in the object storage.

[0262] Optionally, in some embodiments of the present application, the reading unit can be specifically used to determine, when receiving a data reading request sent by a terminal, target metadata information corresponding to the data to be read based on the data reading request; based on the target metadata information, read target object data corresponding to the data to be read in the object storage; and send the target object data to the terminal.

[0263] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to obtain a first random data stream to be stored and a first write offset; based on the historical storage information of the data storage space and the first write offset, the first random data stream is divided to obtain multiple first random data; based on the first write offset, the first random data is written into the data storage space to obtain third target data corresponding to each first random data, wherein there is no overlapping part between the first random data and the data to be written that has been written in the first target data; based on the first random data, third metadata information corresponding to the third target data is constructed; according to the third metadata information, third object data corresponding to the third target data is generated, and the third object data is stored in the object storage.

[0264] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to obtain a second random data stream to be stored and a second write offset; based on the historical storage information of the data storage space and the second write offset, write the second random data stream completely over the target data of the data storage space; based on the second random data stream, construct fourth metadata information corresponding to the target data; based on the fourth metadata information, generate fourth object data corresponding to the target data, and store the fourth object data to the object storage.

[0265] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to obtain a third random data stream to be stored and a third write offset; based on the historical storage information of the data storage space and the third write offset, write the third random data stream over the continuous target data of the data storage space; based on the third random data stream, construct fifth metadata information corresponding to the target data located at the first position in the continuous target data, and sixth metadata information corresponding to the target data located at the last position in the continuous target data; based on the fifth metadata information, generate fifth object data corresponding to the target data located at the first position, and store the fifth object data in the object storage; based on the sixth metadata information, generate sixth object data corresponding to the target data located at the last position, and store the sixth object data in the object storage.

[0266] Optionally, in some embodiments of the present application, the data processing device can also be specifically used to obtain a data truncation request, and determine a data truncation position based on the data truncation request; determine the target data at the data truncation position; and adjust the metadata information corresponding to the target data at the data truncation position based on the offset of the data truncation position in the target data.

[0267] As can be seen from the above, in this embodiment, the current data stream to be stored can be obtained by the acquisition unit 2201, and the current data stream can be segmented based on the historical storage information of the data storage space to obtain multiple data to be written; the data to be written is written into the data storage space by the writing unit 2202 to obtain the target data corresponding to each data to be written; the metadata information corresponding to the target data is constructed by the construction unit 2203 based on the data to be written; the object data corresponding to the target data is generated by the storage unit 2204 according to the metadata information, and the object data is stored in the object storage; when the data read request sent by the terminal is received, the target object data is read in the object storage by the reading unit 2205, and the target object data is sent to the terminal. This application can use the object storage module to manage array resources, and abstract the metadata service module to perform metadata logic management. In addition, for the flattened KV structure of the object storage, the directory tree is abstracted, and the mapping relationship between the node and the data block is constructed. The technical solution of this application can support POSIX semantic access to object storage data while achieving efficient reading and writing of files, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. This method can achieve strong data consistency and can be expanded in parallel to provide high availability and high reliability service guarantees for metadata service modules. In addition, this method can achieve storage and computing separation and is applicable to multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios.

[0268] The present application also provides an electronic device, such as Fig.23 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, and the electronic device may be a terminal or a server, etc. Specifically:

[0269] The electronic device may include components such as a processor 2301 with one or more processing cores, a memory 2302 with one or more computer-readable storage media, a power supply 2303, and an input unit 2304. Those skilled in the art will appreciate that Fig.23 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0270] The processor 2301 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 2302, and calling data stored in the memory 2302. Optionally, the processor 2301 may include one or more processing cores; preferably, the processor 2301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 2301.

[0271] The memory 2302 can be used to store software programs and modules. The processor 2301 executes various functional applications and data processing by running the software programs and modules stored in the memory 2302. The memory 2302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 2302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 2302 may also include a memory controller to provide the processor 2301 with access to the memory 2302.

[0272] The electronic device also includes a power supply 2303 for supplying power to each component. Preferably, the power supply 2303 can be logically connected to the processor 2301 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 2303 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0273] The electronic device may further include an input unit 2304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0274] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 2301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 2302 according to the following instructions, and the processor 2301 will run the application program stored in the memory 2302, thereby realizing various functions, as follows:

[0275] The embodiment of the present application provides a data processing method and related equipment, which can obtain a current data stream to be stored, and based on the historical storage information of the data storage space, divide the current data stream to obtain multiple data to be written; write the data to be written into the data storage space to obtain target data corresponding to each data to be written; based on the data to be written, construct metadata information corresponding to the target data; based on the metadata information, generate object data corresponding to the target data, and store the object data in the object storage; when a data read request sent by a terminal is received, read the target object data in the object storage, and send the target object data to the terminal.

[0276] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0277] As can be seen from the above, this embodiment can provide a data processing method and related equipment in the embodiment of the present application, which can obtain the current data stream to be stored, and based on the historical storage information of the data storage space, the current data stream is segmented to obtain multiple data to be written; the data to be written is written into the data storage space to obtain the target data corresponding to each data to be written; based on the data to be written, the metadata information corresponding to the target data is constructed; according to the metadata information, the object data corresponding to the target data is generated, and the object data is stored in the object storage; when the data read request sent by the terminal is received, the target object data is read in the object storage, and the target object data is sent to the terminal. The present application can use the object storage module to manage array resources, and abstract the metadata service module to manage metadata logic. In addition, for the flattened KV structure of the object storage, the directory tree is abstracted, and the mapping relationship between the node and the data block is constructed. The technical solution of the present application can support POSIX semantic access to object storage data while achieving efficient reading and writing of files, supporting multiple file system standard interfaces such as random reading and writing, data addition, and data truncation. The method can achieve strong consistency of data, and can be expanded in parallel to provide high availability and high reliability service guarantees for the metadata service module. Moreover, this method can achieve storage and computing separation and is applicable to multiple scenarios, such as Hadoop big data tasks, machine learning and other storage demand scenarios.

[0278] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0279] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any data processing method provided in the embodiment of the present application. For example, the instructions can execute the following steps:

[0280] The embodiment of the present application provides a data processing method and related equipment, which can obtain a current data stream to be stored, and based on the historical storage information of the data storage space, divide the current data stream to obtain multiple data to be written; write the data to be written into the data storage space to obtain target data corresponding to each data to be written; based on the data to be written, construct metadata information corresponding to the target data; based on the metadata information, generate object data corresponding to the target data, and store the object data in the object storage; when a data read request sent by a terminal is received, read the target object data in the object storage, and send the target object data to the terminal.

[0281] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0282] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0283] Since the instructions stored in the computer-readable storage medium can execute the steps in any data processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0284] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations of the above-mentioned data processing aspects.

[0285] The above is a detailed introduction to a data processing method and related equipment provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: include: Acquire a current data stream to be stored, and divide the current data stream based on historical storage information of the data storage space to obtain a plurality of data to be written; Writing the data to be written into the data storage space to obtain target data corresponding to each data to be written; Based on the data to be written, construct metadata information corresponding to the target data; Generate object data corresponding to the target data according to the metadata information, and store the object data in object storage; When a data read request sent by a terminal is received, target object data is read from the object storage, and the target object data is sent to the terminal.

2. The data processing method according to claim 1, characterized in that: After obtaining the current data stream to be stored and dividing the current data stream based on the historical storage information of the data storage space to obtain a plurality of data to be written, the method further includes: If the data to be written has not yet filled up the target data in the data storage space, the data writing is stopped, and the target data that has not yet filled up is determined as the first target data; constructing first metadata information corresponding to the first target data based on the data to be written that has been written into the first target data; First object data corresponding to the first target data is generated according to the first metadata information, and the first object data is stored in an object storage.

3. The data processing method according to claim 2, characterized in that: After generating first object data corresponding to the first target data according to the first metadata information and storing the first object data in an object storage, the method further includes: Writing the data to be written into the remaining space of the first target data; constructing second metadata information corresponding to the first target data based on the data to be written in the remaining space of the first target data; Second object data corresponding to the first target data is generated according to the second metadata information, and the second object data is stored in the object storage.

4. The data processing method according to claim 2, characterized in that: After generating first object data corresponding to the first target data according to the first metadata information and storing the first object data in an object storage, the method further includes: Obtaining a first random data stream to be stored and a first write offset; Based on the historical storage information of the data storage space and the first write offset, the first random data stream is segmented to obtain a plurality of first random data; Based on the first write offset, the first random data is written into the data storage space to obtain third target data corresponding to each first random data, wherein there is no overlap between the first random data and the data to be written that has been written in the first target data; Based on the first random data, construct third metadata information corresponding to the third target data; According to the third metadata information, third object data corresponding to the third target data is generated, and the third object data is stored in the object storage.

5. The data processing method according to claim 1, characterized in that: After generating object data corresponding to the target data according to the metadata information and storing the object data in the object storage, the method further includes: Obtaining a second random data stream to be stored and a second write offset; Based on the historical storage information of the data storage space and the second write offset, writing the second random data stream completely into the target data of the data storage space; constructing fourth metadata information corresponding to the target data based on the second random data stream; Fourth object data corresponding to the target data is generated according to the fourth metadata information, and the fourth object data is stored in the object storage.

6. The data processing method according to claim 1, characterized in that: After generating object data corresponding to the target data according to the metadata information and storing the object data in the object storage, the method further includes: Obtaining a third random data stream to be stored and a third write offset; Based on the historical storage information of the data storage space and the third write offset, writing the third random data stream over the continuous target data of the data storage space; Based on the third random data stream, construct fifth metadata information corresponding to the target data located at the first position in the continuous target data, and sixth metadata information corresponding to the target data located at the last position in the continuous target data; generating fifth object data corresponding to the first target data according to the fifth metadata information, and storing the fifth object data in the object storage; According to the sixth metadata information, sixth object data corresponding to the target data at the last position is generated, and the sixth object data is stored in the object storage.

7. The data processing method according to claim 1, characterized in that: When receiving a data read request sent by a terminal, reading target object data in the object storage and sending the target object data to the terminal includes: When receiving a data reading request sent by the terminal, determining target metadata information corresponding to the data to be read based on the data reading request; Based on the target metadata information, reading target object data corresponding to the to-be-read data in the object storage; The target object data is sent to the terminal.

8. The data processing method according to claim 1, characterized in that: After generating object data corresponding to the target data according to the metadata information and storing the object data in the object storage, the method further includes: Obtaining a data truncation request, and determining a data truncation position based on the data truncation request; Determine the target data at the data truncation position; Based on the offset of the data truncation position in the target data, metadata information corresponding to the target data at the data truncation position is adjusted.

9. A data processing device, characterized in that: include: An acquisition unit, used for acquiring a current data stream to be stored, and dividing the current data stream based on historical storage information of the data storage space to obtain a plurality of data to be written; A writing unit, used for writing the data to be written into the data storage space to obtain target data corresponding to each data to be written; A construction unit, configured to construct metadata information corresponding to the target data based on the data to be written; A storage unit, configured to generate object data corresponding to the target data according to the metadata information, and store the object data in an object storage; The reading unit is used to read the target object data in the object storage and send the target object data to the terminal when receiving a data reading request sent by the terminal.

10. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the operations in the data processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the data processing method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

Images