A data processing method, device and storage system

Abstract

The present application proposes a data processing method, device and storage system. The above storage system includes multiple interfaces corresponding to multiple storage service modes, multiple data service units corresponding to the above interfaces, and a storage medium shared by the above storage service modes. Among them, the above method may include receiving a data processing request. Determine metadata information corresponding to the data processing based on the above data processing request. Share the above metadata information to each data service unit included in the above storage system so that the above storage system responds to a processing request for storage data in the above storage medium received by any interface, and processes the storage data corresponding to the metadata stored by the data service unit corresponding to the above interface, thereby improving the storage space utilization rate, simplifying the process of user data extraction, and reducing the error rate.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a data processing method, device and storage system. Background Art

[0002] Currently, a variety of storage methods are used for data processing. Among them, the above storage methods may include distributed storage methods. The above distributed storage methods may include object storage, file storage and block storage.

[0003] In the related art, different storage methods need to correspond to different storage interfaces and storage systems, and different storage systems are independent of each other and cannot be shared.

[0004] For example, object storage needs to use the S3 (Amazon Simple Storage Service) interface, and file storage needs to use the Posix (Portable Operating System Interface) interface. When users collect data through different data collection tools, since different collection tools correspond to different storage methods, it is necessary to at least maintain a file storage system that provides a Posix interface and an object storage system that provides an S3 interface to meet the storage needs of data collected by users using different collection tools. When users need to obtain all the collected data, they need to extract the data from the access file system and the object storage system respectively, and then group them together.

[0005] It can be seen that, on the one hand, data stored in different storage methods need to be stored in different storage systems, and the storage space utilization rate is low; on the other hand, the steps for processing the stored data are relatively cumbersome and prone to errors. Summary of the invention

[0006] In order to solve the problem that data stored in different storage methods need to be stored in different storage systems, the storage space utilization rate is low, and the steps for processing the stored data are cumbersome and prone to errors, the present application discloses at least one data processing method, which is applied to a storage system. The above storage system includes multiple interfaces corresponding to multiple storage service methods, multiple data service units corresponding to the above multiple interfaces, and a storage medium shared by the above multiple storage service methods;

[0007] Such methods may include:

[0008] receiving data processing requests;

[0009] Determine metadata information corresponding to the data processing based on the data processing request;

[0010] The metadata information is shared with each data service unit included in the storage system so that the storage system responds to a processing request for storage data in the storage medium received by any interface, and processes the storage data corresponding to the metadata stored by the data service unit corresponding to the interface.

[0011] In some of the illustrated embodiments, the determining of metadata information corresponding to the data processing based on the data processing request includes:

[0012] The metadata information corresponding to the data processing is determined based on the data processing request through the first data service unit corresponding to the interface receiving the data processing request.

[0013] In some of the illustrated embodiments, the storage system includes a message queue service;

[0014] The above method further includes:

[0015] Registering services for the plurality of data service units in the message queue service respectively so that the plurality of data service units can monitor each other;

[0016] The above-mentioned sharing of the above-mentioned metadata information to each data service unit included in the above-mentioned storage system includes:

[0017] Sending the metadata information to the message queue through the first data service unit;

[0018] After monitoring the metadata information, the other data service units included in the storage system except the first data service unit acquire and store the metadata information from the message queue.

[0019] In some of the illustrated embodiments, the step of sharing the metadata information to each data service unit included in the storage system includes:

[0020] Distributing the metadata information to other data service units included in the storage system through the first data service unit;

[0021] The other data service unit stores the metadata information in response to the received metadata information.

[0022] In some of the illustrated embodiments, when the data processing request includes a processing request for stored data, the step of determining metadata information corresponding to the data processing based on the data processing request includes:

[0023] Obtaining the stored data information carried in the above data processing request;

[0024] Generate corresponding metadata information based on the above-mentioned stored data information, and determine the generated metadata information as metadata information corresponding to the data processing;

[0025] The above method further includes:

[0026] After the metadata information is determined, the stored data corresponding to the metadata is processed based on the metadata information.

[0027] In some of the illustrated embodiments, when the data processing request includes a processing request for metadata, the step of determining metadata information corresponding to the data processing based on the data processing request includes:

[0028] Obtaining metadata search conditions carried in the above data processing request;

[0029] Based on the above retrieval conditions, metadata information satisfying the above retrieval conditions is obtained, and the obtained metadata information is determined as metadata information corresponding to the data processing.

[0030] In some of the illustrated embodiments, the step of acquiring metadata satisfying the search condition based on the search condition includes:

[0031] By metadata retrieval or log redo, metadata satisfying the above retrieval condition is obtained from the stored metadata information.

[0032] In some of the illustrated embodiments, the above search conditions include any N of the following:

[0033] Retrieve metadata whose generation time is within a preset time interval; retrieve metadata whose storage path is within a preset path; retrieve metadata whose storage path is within a preset path and whose generation time is within a preset time interval.

[0034] In some of the illustrated embodiments, the storage system comprises a distributed storage system;

[0035] The storage medium includes a plurality of storage groups; the storage group includes a plurality of storage nodes; the method further includes:

[0036] When the data to be stored is received, metadata information corresponding to the data to be stored is generated through a data service unit corresponding to the interface receiving the data to be stored;

[0037] Determine, according to the storage path indicated by the metadata information, a storage group corresponding to the data to be stored in the storage medium and a storage node corresponding to the storage group;

[0038] The data to be stored is stored in the storage node.

[0039] In some of the illustrated embodiments, the method further includes:

[0040] Generate a copy corresponding to the above data to be stored;

[0041] The generated copy is stored in other storage groups included in the above storage medium.

[0042] The present application also proposes a data processing device, which is applied to a storage system. The storage system includes multiple interfaces corresponding to multiple storage service modes, multiple data service units corresponding to the multiple interfaces, and a storage medium shared by the multiple storage service modes. The device may include:

[0043] A receiving module, used for receiving a data processing request;

[0044] A determination module, configured to determine metadata information corresponding to the data processing based on the data processing request;

[0045] A sharing module shares the above metadata information to each data service unit included in the above storage system so that the above storage system responds to a processing request for storage data in the above storage medium received by any interface, and processes the storage data corresponding to the metadata stored by the data service unit corresponding to the above interface.

[0046] The present application also includes a storage system, the system comprising:

[0047] An interface layer, a data service layer communicatively connected to the interface layer, a storage engine layer communicatively connected to the data service layer, and a storage medium communicatively connected to the storage engine layer;

[0048] The interface layer includes multiple interfaces corresponding to multiple storage service modes;

[0049] The data service layer includes a plurality of data servers corresponding to the plurality of interfaces respectively; the data service layer provides metadata information sharing services between the plurality of data servers;

[0050] The above-mentioned storage medium includes the storage medium shared by the above-mentioned multiple storage service modes;

[0051] The storage engine layer is used to store data stored in any of the above-mentioned storage service modes to the storage medium;

[0052] The above system executes the data processing method shown in any of the above embodiments.

[0053] In the present application, on the one hand, by deploying a storage medium shared by multiple storage methods in a storage system, data stored in multiple storage methods can be stored in the storage medium, thereby improving the storage space utilization rate. On the other hand, by sharing metadata information for accessing stored data among multiple storage service units included in the storage system, the above-mentioned storage system can respond to a processing request for storage data in the above-mentioned storage medium received by any interface, and process the storage data corresponding to the metadata stored by the data service unit corresponding to the above-mentioned interface, so that users can access all data stored in the above-mentioned storage system through only one interface, simplifying the process of users extracting data and reducing the error rate.

[0054] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 A schematic diagram of the structure of a storage system shown in this application;

[0057] Figure 2 A method flow chart of a data processing method shown in this application;

[0058] Figure 3 A schematic diagram of a storage system architecture shown in this application;

[0059] Figure 4 A schematic diagram of a topological structure of a storage system shown in this application;

[0060] Figure 5 A schematic diagram of a metadata sharing process shown in this application;

[0061] Figure 6 This is a schematic diagram of the structure of a data processing device shown in this application. DETAILED DESCRIPTION

[0062] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0063] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "above", and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0064] In view of this, the present application proposes a data processing method. On the one hand, the method deploys a storage medium shared by multiple storage methods in a storage system, so that data stored in multiple storage methods can be stored in the storage medium, thereby improving the storage space utilization rate. On the other hand, the method shares metadata information for accessing stored data among multiple storage service units included in the storage system, so that the above-mentioned storage system can respond to a processing request for storage data in the above-mentioned storage medium received by any interface, and process the storage data corresponding to the metadata stored by the data service unit corresponding to the above-mentioned interface, so that users can access all data stored in the above-mentioned storage system through only one interface, simplifying the process of users extracting data and reducing the error rate.

[0065] The method can be applied to a storage system. The storage system comprises a plurality of interfaces corresponding to a plurality of storage service modes, a plurality of data service units corresponding to the plurality of interfaces, and a storage medium shared by the plurality of storage service modes.

[0066] In this application, multiple may refer to greater than or equal to two, and the specific number can be set according to the actual business, and is not specifically limited here. For example, multiple interfaces may refer to more than two interfaces. Multiple data service units may refer to more than two data service units corresponding to the interfaces.

[0067] The storage system may be a single-point storage system or a distributed storage system. In some examples, in order to optimize storage performance, the storage system may be a distributed storage system.

[0068] The following description is made by taking the above storage system as a distributed storage system as an example. It is understandable that the description of the single-point storage system can refer to the description of the distributed storage system, which will not be described in detail in this application.

[0069] See also Figure 1 , Figure 1 This is a schematic diagram of a storage system shown in this application. It should be noted that: Figure 1 This is only for illustrative purposes; other similar structures may be used in actual applications.

[0070] like Figure 1 As shown, the storage system may include multiple interfaces, multiple data service units, and storage media. The multiple interfaces may respectively perform data interaction with multiple data service units. The multiple data service units may perform data interaction with the storage media.

[0071] Figure 1 The interfaces shown can interact with users. For example, the above multiple interfaces can include a Posix interface that supports file storage and an S3 interface that supports object storage. In a distributed storage system, the above interfaces can be deployed in a computing node that interacts with a user client.

[0072] The above interfaces correspond to Figure 1 For example, if the user processes data through a Posix interface, the data processing can be performed through a data service unit corresponding to the Posix interface. If the user processes data through an S3 interface, the data processing can be performed through a data service unit corresponding to the S3 interface.

[0073] The above data service unit is used for data management services. Among them, the data service units corresponding to different interfaces use different storage methods to manage data.

[0074] For example, when the above-mentioned interface is a file storage interface, the data service unit corresponding to the interface can generate file metadata (i.e., descriptive data of the stored data) corresponding to the stored data (data collected by the user and stored in the storage medium) to realize data management of the file storage method.

[0075] For another example, when the above interface is an object storage interface, the data service unit corresponding to the interface can generate object metadata (i.e., descriptive data of the stored data) corresponding to the stored data (data collected by the user and stored in the storage medium) to realize data management in an object storage manner.

[0076] It should be noted that the metadata of the same storage data stored in different types of data service units can be converted to each other, and the conversion method is described in detail here. For example, file metadata can be converted to object metadata; object metadata can also be converted to file metadata. Among them, the above file metadata and the above object metadata can point to the same storage data.

[0077] The metadata is specifically the description data of the stored data. The metadata can be generated by the data service unit based on the stored data. The metadata can include the storage path, data name, data size, storage time and other information of the stored data.

[0078] The above metadata may be used to access the storage data stored in the storage medium.

[0079] For example, when a data service unit of a file storage type stores a certain file metadata, the unit can display a display view to the user based on the file metadata stored by itself (the above display view includes displaying metadata to the user in a preset display form). When the user needs to access the storage data corresponding to the above file metadata, the file metadata displayed by the above display view can be packaged into a data access request. The request is then sent to the above storage system through the file interface. The above storage system can determine the storage address of the above storage data through the storage path indicated by the above file metadata, and then obtain and return the above storage data to the user based on the storage address.

[0080] In a distributed storage system, the above data service unit may be deployed in a data service node.

[0081] Figure 1 The storage medium shown is specifically a storage medium shared by the above-mentioned multiple storage service modes. Data received through any data interface can be stored in the above-mentioned storage medium.

[0082] In some examples, the storage medium includes several storage groups. The storage group may include several storage nodes. When the data to be stored is received, metadata information corresponding to the data to be stored may be generated by a data service unit corresponding to the interface receiving the data to be stored. Then, the storage group corresponding to the data to be stored in the storage medium and the corresponding storage node in the storage group are determined according to the storage path indicated by the metadata information. Finally, the data to be stored is stored in the storage node.

[0083] In the above example, distributed storage can be implemented to balance the storage pressure of each storage node and improve the storage performance of the distributed storage system.

[0084] In some examples, the storage system may also generate a copy corresponding to the data to be stored, and then store the generated copy in other storage groups included in the storage medium.

[0085] In the above example, the reliability of data storage can be guaranteed through the replication mechanism.

[0086] In some examples, the storage system may further include a storage engine unit configured to store the corresponding storage data to the storage medium based on the metadata information.

[0087] See also Figure 2 , Figure 2 The method flow chart of a data processing method shown in the present application. The method is applied to a storage system. The storage system includes a plurality of interfaces corresponding to a plurality of storage service modes, a plurality of data service units corresponding to the plurality of interfaces, and a storage medium shared by the plurality of storage service modes. Figure 1 The following is an example of an embodiment.

[0088] like Figure 2 As shown, the above method may include:

[0089] S202, receiving a data processing request.

[0090] In some examples, the user can interact with any interface provided by the storage system through the client installed by the user to send a data processing request to the storage system. The storage system can receive the data processing request through the interface. This application does not specifically limit the interface protocol.

[0091] The data processing request may include a processing request for stored data or a processing request for metadata. The processing request for stored data may include a request for storing, modifying, deleting, etc. the stored data. The processing request for metadata may include a metadata sharing request. The metadata sharing request may include sharing the stored metadata to each data service unit included in the storage system.

[0092] After receiving the data processing request, S204 may be executed.

[0093] S204: Determine metadata information corresponding to the data processing based on the data processing request.

[0094] In some examples, the data processing request may include information such as stored data information or metadata information corresponding to the stored data or metadata retrieval conditions.

[0095] After receiving the data processing request, the data information carried in the data processing request may be parsed, and after acquiring the data information, the metadata information may be determined based on the data information.

[0096] For example, if the data processing request carries storage data information, corresponding metadata information may be generated by a first data service unit corresponding to the interface receiving the data processing request to determine metadata information corresponding to the data processing.

[0097] If the data processing request carries a metadata retrieval condition, metadata information satisfying the retrieval condition may be obtained from the stored metadata information, and the obtained metadata information may be determined as metadata information corresponding to the data processing.

[0098] In some examples, in order to be compatible with existing storage service modes and reduce the difficulty of storage system development, the above step S204 may be performed by a first data service unit corresponding to the interface that receives the above request.

[0099] For example, when the above interface is a file storage interface, the corresponding data service unit is a file storage service unit. At this time, when S204 is executed by the file storage service unit, the existing file storage method can be used to perform the relevant steps, so there is no need to develop additional code in the storage system, so that it can be compatible with the existing storage service method, reducing the difficulty of storage system development.

[0100] After the metadata information corresponding to the data processing is determined, S206 may be executed.

[0101] S206, sharing the metadata information to the data service units included in the storage system so that the storage system responds to a processing request for storage data in the storage medium received by any interface, and processes the storage data corresponding to the metadata stored by the data service unit corresponding to the interface.

[0102] For example, the storage system may distribute the determined metadata information to each data service unit to complete the above-mentioned sharing.

[0103] The above-mentioned processing request for the storage data in the above-mentioned storage medium may include processing requests such as access, deletion, and modification of the above-mentioned storage data. In some examples, the above-mentioned processing request may be an access request.

[0104] In the above scheme, on the one hand, since the above storage system includes a storage medium shared by multiple storage methods, data stored by multiple storage methods are stored in the common storage medium, thereby improving the utilization rate of storage space; on the other hand, since the metadata information that can be used to access the stored data can be shared to each data service unit included in the above storage system, the above storage system can respond to a processing request for the stored data in the above storage medium received by any interface, and process the stored data corresponding to the metadata stored by the data service unit corresponding to the above interface, so that the user can access all the data stored in the above storage system through only one interface, which simplifies the process of users extracting data and reduces the error rate.

[0105] In some examples, metadata information sharing can be achieved by using a message queue service. The storage system can provide a message queue service. In the message queue service, services are registered for the multiple data service units respectively so that the multiple data service units can monitor each other.

[0106] For example, the registration may be completed when the storage system is initialized.

[0107] Afterwards, when the metadata information is shared with each data service unit included in the storage system, the metadata information can be sent to the message queue through the first data service unit corresponding to the interface receiving the request. Then, after monitoring the metadata information, other data service units included in the storage system except the first data service unit can obtain and store the metadata information from the message queue.

[0108] In the above example, the metadata information sharing between the data service units is guaranteed by the above message queue service, thereby ensuring that users can access all data stored in the above storage system through one interface, simplifying the process of user data extraction and reducing the error rate.

[0109] In some examples, metadata information sharing can be achieved through a message notification service. The storage system can provide a message notification service. In the message notification service, services are registered for the multiple data service units respectively so that the multiple data service units can notify each other of messages.

[0110] For example, the registration may be completed when the storage system is initialized.

[0111] Afterwards, when the metadata information is shared with each data service unit included in the storage system, the metadata information may be distributed to the other data service units included in the storage system except the first data service unit through the first data service unit corresponding to the interface receiving the request. Then, the other data service units may store the metadata information in response to the received metadata information.

[0112] In the above example, the metadata information sharing between the data service units is guaranteed by the message notification service, thereby ensuring that users can access all data stored in the above storage system through one interface, simplifying the process of users extracting data and reducing the error rate.

[0113] In some examples, metadata sharing of the stored data may be performed synchronously when processing a data processing request initiated by a user for the stored data.

[0114] At this time, the data processing request includes a data processing request for storage data.

[0115] The following describes data processing requests for storing data in two categories, wherein the first category may include requests for creating data such as data storage requests, and the second category may include requests for updating data such as data modification or deletion requests.

[0116] It is understandable that in the first type of request, the corresponding storage data is not stored in the above storage system, so there is no corresponding metadata information. At this time, when the data processing request is initiated, the request carries the storage data information. In the second type of request, the corresponding storage data has been stored in the above storage system, so the corresponding metadata information has also been stored. At this time, when the data processing request is initiated, the request carries the metadata information corresponding to the storage data.

[0117] For the first type of request:

[0118] When the data processing request includes a storage request for storage data, when executing S204, the storage data information carried by the data processing request may be obtained, and then corresponding metadata information may be generated based on the storage data information, and the generated metadata information may be determined as metadata information corresponding to the data processing.

[0119] In some examples, after receiving the request, the request may be transmitted to a first data service unit corresponding to the interface receiving the request, and the unit then generates and stores metadata information corresponding to the stored data information.

[0120] Then, based on the message queue mechanism, the unit may send the metadata information to the message queue, and then other data service units other than the first data service unit may obtain and store the metadata information to achieve sharing of metadata messages.

[0121] Afterwards, the storage data may be stored in the storage medium based on the generated metadata information.

[0122] In some examples, the storage system may include a storage engine for storing data. The storage engine may manage the storage medium and divide the medium into a plurality of storage groups. The storage groups may include a plurality of storage nodes or storage disks.

[0123] The storage engine can determine the storage group corresponding to the data to be stored in the storage medium and the corresponding storage node in the storage group according to the storage path indicated by the metadata information, and then store the data to be stored in the storage node.

[0124] In some examples, the storage engine may also generate a copy corresponding to the data to be stored, and then the generated copy may be stored in other storage groups included in the storage medium.

[0125] In the above example, on the one hand, after receiving a storage request for stored data, the metadata corresponding to the stored data can be shared during the data storage process, so that users can access the stored data immediately through any interface after the storage of the stored data is completed, thereby improving the user experience. On the other hand, the replica mechanism can be used to ensure the reliability of data storage.

[0126] For the second type of request:

[0127] When the data processing request includes an update request such as modification or deletion of stored data, when executing S204, the updated stored data information carried by the data processing request may be obtained, and then corresponding metadata update information is generated based on the updated stored data information, and the generated metadata update information is determined as metadata information corresponding to the data processing.

[0128] In some examples, after receiving the request, the request may be transmitted to a first data service unit corresponding to the interface receiving the request, and the unit then generates and stores metadata update information based on the updated storage data information.

[0129] The unit may then complete the sharing of the metadata update information, which will not be described in detail here.

[0130] Thereafter, based on the metadata update information, the stored data corresponding to the metadata may be processed.

[0131] For example, if the data processing request is an update processing request, the storage address of the storage data can be determined based on the storage path indicated by the metadata update information, and then the updated storage data can be stored at the address to complete the update of the original storage data.

[0132] In the above example, after receiving an update request for stored data, the metadata corresponding to the stored data can be shared during the data update process, so that users can immediately access the updated stored data through any interface after the storage data is updated, thereby improving the user experience.

[0133] In some examples, metadata sharing may be delayed to preserve storage system performance.

[0134] At this time, the above data processing request may include a data processing request for metadata. That is, when a user initiates a processing request for stored data, the storage system only completes the data processing operation, and then implements metadata sharing after receiving a sharing request for metadata initiated by the user.

[0135] In some examples, the metadata sharing request may include a search condition for the metadata to be processed, and metadata that meets the search condition may be shared, thereby achieving flexible sharing of metadata and improving user experience.

[0136] In some examples, when the data processing request includes a request for processing metadata, when executing S204, the search condition carried in the data processing request may be obtained. Then, based on the search condition, metadata information satisfying the search condition may be obtained, and the obtained metadata information may be determined as metadata information corresponding to the data processing.

[0137] The above search conditions may include any N of the following:

[0138] Retrieve metadata whose generation time is within a preset time interval; retrieve metadata whose storage path is within a preset path; retrieve metadata whose storage path is within a preset path and whose generation time is within a preset time interval. Wherein, the above N is a positive integer less than or equal to 3.

[0139] It is understandable that setting the above search conditions can enable users to perform metadata sharing processing more flexibly.

[0140] In some examples, after receiving the above request, the request can be passed to the data service unit corresponding to the interface that receives the request. Then, based on the above retrieval conditions, the unit retrieves metadata that meets the above retrieval conditions from the stored metadata information through metadata retrieval or log redo. Then, the unit can pass the retrieved metadata to other data service units through a message queue mechanism, thereby realizing metadata sharing.

[0141] In the above example, since the storage system can share metadata that meets the retrieval conditions in response to metadata sharing requests flexibly initiated by users, on the one hand, metadata sharing can be delayed to ensure storage system storage access performance; on the other hand, user experience can be improved.

[0142] The application also proposes a storage system.

[0143] See also Figure 3 , Figure 3 A schematic diagram of a storage system architecture shown in this application.

[0144] like Figure 3 As shown, the above system may include an interface layer, a data service layer communicatively connected to the above interface layer, a storage engine layer communicatively connected to the above data service layer, and a storage medium communicatively connected to the above storage engine layer.

[0145] The interface layer includes a plurality of interfaces corresponding to a plurality of storage service modes.

[0146] The data service layer includes a plurality of data servers corresponding to the plurality of interfaces respectively; the data service layer provides metadata information sharing services among the plurality of data servers.

[0147] The above-mentioned storage medium includes storage media shared by the above-mentioned multiple storage service modes.

[0148] The storage engine layer is used to store data stored in any of the above-mentioned storage services to the storage medium.

[0149] The above system can execute the data processing method shown in any of the above embodiments.

[0150] The embodiment is described by taking the storage system compatible with file storage and object storage as an example.

[0151] The above storage system (hereinafter referred to as system) is a distributed storage system, which may be a distributed storage system with a central node or a distributed storage system without a central node, which is not limited here.

[0152] See also Figure 4 , Figure 4 This is a schematic diagram of a topological structure of a storage system shown in this application. Figure 4 The topological structure shown is only for illustration and may have many variations in actual applications.

[0153] Indicative, such as Figure 4 As shown, the above storage system may include an interface layer, a data service layer communicatively connected to the interface layer, a storage engine layer communicatively connected to the data service layer, and a storage medium communicatively connected to the storage engine layer.

[0154] The interface layer may include several computing nodes (not shown in the figure), which are used to interact with users.

[0155] The computing nodes provide a file interface (eg, Posix interface) for file storage and an object interface (eg, S3 interface) for object storage. The interface layer can receive data processing requests from users or return storage data to users.

[0156] The data service layer may include a file metadata server (hereinafter referred to as a file server) that manages data in a file storage manner. The file server corresponds to the Posix interface. Data processing requests received through the Posix interface may be passed to the file server for processing.

[0157] The data service layer may include an object gateway server (hereinafter referred to as an object server) that manages data in an object storage manner. The object server corresponds to the S3 interface. Data processing requests received through the S3 interface may be transferred to the object server for processing.

[0158] The data service layer can also deploy metadata retrieval services and message queue services in the file server and object server. It is understandable that, usually when the system is initialized, the file server and object server can be registered in the message queue service so that the two servers can monitor each other.

[0159] The data service layer can also display the metadata stored in itself to the user in the form of a display view. The user can access the data stored in the system through the metadata shown in the display view.

[0160] The above storage medium may include several storage nodes or storage disks. The above storage medium is a storage medium shared by file storage and object storage.

[0161] The storage engine layer may include several storage engine servers. The storage engine layer may divide the storage medium into several storage groups through management logic. The storage groups include several storage disks or storage nodes.

[0162] The storage engine layer can store data stored in file storage or object storage in the storage medium, thereby enabling the sharing of storage media.

[0163] The storage engine layer can also copy the stored data to other storage groups through a replication mechanism, thereby ensuring the reliability of data storage.

[0164] The above system can provide synchronous metadata sharing and delayed metadata sharing functions.

[0165] Synchronize metadata sharing service:

[0166] See also Figure 5 , Figure 5 A schematic diagram of a metadata sharing process shown in this application.

[0167] like Figure 5 As shown, when the system is initialized, step 1 can be executed first to complete the registration service of the file server and the object server in the message queue service so that the two can monitor each other.

[0168] When a user collects data A from the network through a crawler tool (assuming that the crawler tool only supports file storage), the user can construct a data storage request based on the data A and send the storage request to the above system through the Posix interface of the above system.

[0169] After receiving the storage request, the system may send the request to the file server ( Figure 5 2 shown).

[0170] The file server may parse the storage request and obtain the data A carried in the request. Then, the corresponding metadata information may be constructed based on the information carried in the data A and the metadata information may be stored.

[0171] Afterwards, the file server can send the metadata information and the data A to the storage engine server ( Figure 5 The storage engine server may determine the corresponding storage group and storage node based on the storage path indicated by the metadata information to complete the storage of data A, and may generate several copies based on data A and store them in other storage groups.

[0172] On the other hand, the file server can send the above metadata information to the message queue ( Figure 5 After the object server monitors the metadata information uploaded by the file server, it can obtain the metadata information from the message queue and store it ( Figure 5 4 shown).

[0173] At this point, the storage of the data A is completed, and the sharing of the metadata information corresponding to the data A is synchronously completed.

[0174] It is not difficult to understand that the data A has been stored in the storage medium, and the metadata information of the data A is stored in the file server and the object server (hereinafter referred to as the two servers). Afterwards, the two servers can display the metadata corresponding to the data A to the user in a display view, so that the user can access the data A in real time based on the metadata through the file interface or object interface provided by the system.

[0175] Delayed metadata sharing feature:

[0176] The following continues to combine Figure 5 The process shown is described.

[0177] like Figure 5 As shown, when the system is initialized, step 1 can be executed first to complete the registration service of the file server and the object server in the message queue service so that the two can monitor each other.

[0178] To ensure the performance of the storage system, users can choose to turn off the synchronous sharing function when storing data. In this case, users can use the system idle time to share metadata.

[0179] The user can set the search condition to search for metadata stored in the time period from 0:00 to 23:59 on the same day and stored in disk D. Then, a metadata sharing request can be constructed based on the search condition and sent to the above system through the Posix interface.

[0180] After receiving the storage request, the system may send the request to the file server ( Figure 5 2 shown).

[0181] The file server can parse the storage request and obtain the search condition carried by the request. Then, based on the search condition, the file server can use the metadata search service to search for metadata information stored in the file server itself, which is stored in the D disk and stored between 0:00 and 23:59 on the same day.

[0182] Afterwards, the file server can send the retrieved metadata information to the message queue ( Figure 5 After the object server monitors the metadata information uploaded by the file server, it can obtain the metadata information from the message queue and store it ( Figure 5 4 shown).

[0183] At this point, the delayed sharing of metadata information is completed.

[0184] It is not difficult to understand that at this time, the file server and the object server (hereinafter referred to as the two servers) both store the above metadata information. Afterwards, the two servers can display the above metadata to the user in the form of a display view, so that the user can access the data stored in the storage system based on the above metadata through the file interface or object interface provided by the system.

[0185] Corresponding to any of the above embodiments, the present application also proposes a data processing device. The device can be applied to a storage system. The above storage system includes multiple interfaces corresponding to multiple storage service modes, multiple data service units corresponding to the above multiple interfaces, and a storage medium shared by the above multiple storage service modes.

[0186] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a data processing device shown in this application.

[0187] like Figure 6 As shown, the above device 60 may include:

[0188] A receiving module 61, used for receiving a data processing request;

[0189] A determination module 62, configured to determine metadata information corresponding to the data processing based on the data processing request;

[0190] The sharing module 63 is used to share the above-mentioned metadata information to each data service unit included in the above-mentioned storage system so that the above-mentioned storage system responds to a processing request for storage data in the above-mentioned storage medium received by any interface, and processes the storage data corresponding to the metadata stored by the data service unit corresponding to the above-mentioned interface.

[0191] In some of the illustrated embodiments, the determination module 62 is specifically configured to:

[0192] The metadata information corresponding to the data processing is determined based on the data processing request through the first data service unit corresponding to the interface receiving the data processing request.

[0193] In some of the illustrated embodiments, the storage system includes a message queue service; the apparatus 60 further includes:

[0194] A monitoring module 64, used to register services for the multiple data service units in the message queue service respectively so that the multiple data service units can monitor each other;

[0195] The above-mentioned sharing module 63 is specifically used for:

[0196] Sending the metadata information to the message queue through the first data service unit;

[0197] After monitoring the metadata information, the other data service units included in the storage system except the first data service unit acquire and store the metadata information from the message queue.

[0198] In some of the illustrated embodiments, the sharing module 63 is specifically used for:

[0199] Distributing the metadata information to other data service units included in the storage system through the first data service unit;

[0200] The other data service unit stores the metadata information in response to the received metadata information.

[0201] In some of the illustrated embodiments, when the data processing request includes a processing request for stored data, the determination module 62 is specifically configured to:

[0202] Obtaining the stored data information carried in the above data processing request;

[0203] Generate corresponding metadata information based on the above-mentioned stored data information, and determine the generated metadata information as metadata information corresponding to the data processing;

[0204] The above-mentioned device 60 further includes:

[0205] The modification module is used to process the storage data corresponding to the metadata based on the metadata information after determining the metadata information.

[0206] In some of the illustrated embodiments, when the data processing request includes a processing request for metadata, the determination module 62 is specifically configured to:

[0207] Obtaining metadata search conditions carried in the above data processing request;

[0208] Based on the above retrieval conditions, metadata information satisfying the above retrieval conditions is obtained, and the obtained metadata information is determined as metadata information corresponding to the data processing.

[0209] In some of the illustrated embodiments, the determination module 62 is specifically configured to:

[0210] By metadata retrieval or log redo, metadata satisfying the above retrieval condition is obtained from the stored metadata information.

[0211] In some of the illustrated embodiments, the above search conditions include any N of the following:

[0212] Retrieve metadata whose generation time is within a preset time interval; retrieve metadata whose storage path is within a preset path; retrieve metadata whose storage path is within a preset path and whose generation time is within a preset time interval.

[0213] In some of the embodiments shown, the storage system includes a distributed storage system; the storage medium includes a plurality of storage groups; the storage group includes a plurality of storage nodes; the device 60 further includes:

[0214] A data storage module, configured to, when receiving the data to be stored, generate metadata information corresponding to the data to be stored through a data service unit corresponding to the interface receiving the data to be stored;

[0215] Determine, according to the storage path indicated by the metadata information, a storage group corresponding to the data to be stored in the storage medium and a storage node corresponding to the storage group;

[0216] The data to be stored is stored in the storage node.

[0217] In some of the illustrated embodiments, the device 60 further includes:

[0218] A replica storage module, used to generate a replica corresponding to the above data to be stored;

[0219] The generated copy is stored in other storage groups included in the above storage medium.

[0220] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0222] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0223] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0224] The embodiments of the subject matter and functional operations described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that may include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0225] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The above processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0226] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0227] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0228] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work as mentioned above in some combinations and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0229] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0230] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0231] The above are only preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. A data processing method, applied to a storage system; It is characterized in that The storage system includes a plurality of interfaces corresponding to the plurality of storage service modes, a plurality of data service units corresponding to the plurality of interfaces, and a storage medium shared by the plurality of storage service modes; The interface is used to interact with the user client; The data service unit is used to manage the stored data in the storage medium; The data service units corresponding to different interfaces use different storage methods for data management; The method comprises: receiving a data processing request based on any one of the multiple interfaces; Determining metadata information corresponding to the data processing based on the data processing request through a first data service unit corresponding to the interface receiving the data processing request; the metadata information is description data of the stored data and is used to access the stored data in the storage medium; Sharing the metadata information to each data service unit included in the storage system so that the storage system responds to a processing request for storage data in the storage medium received by any interface, and processes the storage data corresponding to the metadata stored by the data service unit corresponding to the interface; Each data service unit included in the storage system stores the shared metadata information.

2. The method according to claim 1, It is characterized in that The storage system includes a message queue service; The method further comprises: Registering services for the multiple data service units in the message queue service respectively so that the multiple data service units can monitor each other; The step of sharing the metadata information to each data service unit included in the storage system includes: sending the metadata information to the message queue through the first data service unit; After monitoring the metadata information, other data service units included in the storage system except the first data service unit acquire and store the metadata information from the message queue.

3. The method according to claim 1, It is characterized in that The step of sharing the metadata information to each data service unit included in the storage system includes: Distributing the metadata information to other data service units included in the storage system through the first data service unit; The other data service unit stores the metadata information in response to the received metadata information.

4. The method according to any one of claims 1 to 3, It is characterized in that When the data processing request includes a processing request for stored data, determining metadata information corresponding to the data processing based on the data processing request includes: Obtaining the storage data information carried by the data processing request; generating corresponding metadata information based on the stored data information, and determining the generated metadata information as metadata information corresponding to the data processing; The method further comprises: After determining the metadata information, storage data corresponding to the metadata is processed based on the metadata information.

5. The method according to any one of claims 1 to 3, It is characterized in that When the data processing request includes a processing request for metadata, the determining metadata information corresponding to the data processing based on the data processing request includes: Obtaining metadata retrieval conditions carried in the data processing request; Based on the search condition, metadata information satisfying the search condition is acquired, and the acquired metadata information is determined as metadata information corresponding to the data processing.

6. The method according to claim 5, It is characterized in that The acquiring metadata satisfying the search condition based on the search condition includes: By metadata retrieval or log redoing, metadata satisfying the retrieval condition is obtained from the stored metadata information.

7. The method according to claim 5, It is characterized in that The search conditions include any N of the following: Retrieve metadata whose generation time is within a preset time interval; retrieve metadata whose storage path is within a preset path; retrieve metadata whose storage path is within a preset path and whose generation time is within a preset time interval.

8. The method according to claim 1, It is characterized in that The storage system includes a distributed storage system; The storage medium includes a plurality of storage groups; the storage groups include a plurality of storage nodes; the method further includes: When the data to be stored is received, metadata information corresponding to the data to be stored is generated through a data service unit corresponding to the interface receiving the data to be stored; Determine, according to the storage path indicated by the metadata information, a storage group corresponding to the data to be stored in the storage medium and a storage node corresponding to the storage group; The data to be stored is stored in the storage node.

9. The method according to claim 8, It is characterized in that The method further comprises: Generating a copy corresponding to the data to be stored; The generated copy is stored in other storage groups included in the storage medium.

10. A data processing device, applied to a storage system; It is characterized in that The storage system includes a plurality of interfaces corresponding to the plurality of storage service modes, a plurality of data service units corresponding to the plurality of interfaces, and a storage medium shared by the plurality of storage service modes; The interface is used to interact with the user client; The data service unit is used to manage the stored data in the storage medium; The data service units corresponding to different interfaces use different storage methods for data management; The device comprises: A receiving module, configured to receive a data processing request based on any one of the multiple interfaces; a determination module, configured to determine metadata information corresponding to the data processing based on the data processing request through a first data service unit corresponding to the interface receiving the data processing request; the metadata information is description data of the stored data and is used to access the stored data in the storage medium; a sharing module, which shares the metadata information to each data service unit included in the storage system so that the storage system processes the storage data corresponding to the metadata stored by the data service unit corresponding to the interface in response to a processing request for storage data in the storage medium received by any interface; Each data service unit included in the storage system stores the shared metadata information.

11. A storage system, It is characterized in that The system comprises: An interface layer, a data service layer communicatively connected to the interface layer, a storage engine layer communicatively connected to the data service layer, and a storage medium communicatively connected to the storage engine layer; The interface layer includes multiple interfaces corresponding to multiple storage service modes respectively; The data service layer includes a plurality of data servers corresponding to the plurality of interfaces respectively; the data service layer provides metadata information sharing services between the plurality of data servers; The storage medium includes a storage medium shared by the multiple storage service modes; The storage engine layer is used to store data stored in any of the multiple storage service modes into the storage medium; The system executes the data processing method as described in any one of claims 1-9.

Images