A data processing method and system

Abstract

The application discloses a data processing method and system, wherein the data processing method comprises the following steps: a data collection node acquires metadata information of stored data in an allocated data fragment and sends the metadata information to an information management node corresponding to the data collection node; the information management node receives the metadata information; the information management node stores the metadata information in an associated information cache library; and the information management node periodically synchronizes the metadata information in the associated information cache library to an information database. Through the technical scheme, the application improves the processing performance of the information database on metadata and improves the robustness of the information database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and system. BACKGROUND

[0002] At present, a distributed database based on a single machine database realizes horizontal expansion of data through data sharding. Meanwhile, global transaction control and data routing of computing nodes are supplemented to realize distributed transaction management.

[0003] However, this management mode of metadata under a distributed architecture has problems: each data node currently stores metadata, but there is no comprehensive data node to uniformly manage the metadata information in these data nodes, so that the conditions of the data shards corresponding to the metadata information in these data nodes cannot be obtained, the metadata information of the data stored in the data shards cannot be obtained, the processing performance of the database for metadata is reduced, and the robustness of the database is reduced. SUMMARY

[0004] The present application provides a data processing method and system to improve the processing performance of the information database for metadata and improve the robustness of the information database.

[0005] According to an aspect of the present application, a data processing method is provided, comprising:

[0006] The data collection node obtains metadata information of data stored in the allocated data shard and sends the metadata information to the information management node corresponding to the data collection node;

[0007] The information management node receives the metadata information;

[0008] The information management node stores the metadata information in the associated information cache library;

[0009] The information management node periodically synchronizes the metadata information in the associated information cache library to the information database.

[0010] According to another aspect of the present application, a data processing system is provided, which is applied to the data processing method of any embodiment of the present application, and the system comprises a data collection node, an information management node, an information cache library associated with the information management node, and an information database;

[0011] The data collection node is configured to obtain metadata information of data stored in the allocated data shard and send the metadata information to the corresponding information management node;

[0012] The information management node is configured to receive the metadata information;

[0013] The information management node is configured to store the metadata information in the associated information cache library;

[0014] An information management node is configured to periodically synchronize the metadata information in the associated information cache to the information database.

[0015] According to the technical scheme of the embodiment of the present application, the data collection node acquires the metadata information of the stored data in the allocated data shard and sends the metadata information to the information management node corresponding to the data collection node. Then, the information management node receives the metadata information. Subsequently, the information management node stores the metadata information in the associated information cache. Furthermore, the information management node periodically synchronizes the metadata information in the associated information cache to the information database. According to the technical scheme, the information management node pre-stores the metadata information in the information cache, thereby improving the real-time read-write speed of the metadata information. Meanwhile, the metadata information in the information cache is periodically uploaded to the information database, so that the metadata information can be periodically stored in the disk, and the loss of the metadata information in the information cache can be recovered. The collection, merging and integration of the metadata information of the stored data in the data shard are realized, thereby improving the processing performance of the metadata of the information database and further improving the robustness of the information database.

[0016] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flowchart of a data processing method according to the first embodiment of the present application;

[0019] Figure 2A is a flowchart of a data processing method according to the second embodiment of the present application;

[0020] Figure 2B is a flowchart of metadata change according to the second embodiment of the present application;

[0021] Figure 3A is a structural diagram of a data processing system according to the third embodiment of the present application;

[0022] Figure 3B is another structural diagram of a data processing system according to the third embodiment of the present application;

[0023] Figure 3C The present invention provides a flowchart of the overall process for collecting, merging and integrating metadata information of data stored in data fragments in a data processing system according to Embodiment 3 of the present invention.

[0024] Figure 3D This is a structural diagram of another data processing system provided according to Embodiment 3 of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the data processing method of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "main," "secondary," "target," and "other," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. In the technical solutions of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of data all comply with relevant laws and regulations and do not violate public order and good morals.

[0028] Example 1

[0029] Figure 1 This is a flowchart of a data processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to the unified collection, merging, and integration of metadata information of data stored in data nodes of a single-machine distributed database. This method can be executed by a data processing system, which can be implemented in hardware and / or software, and can be configured within a distributed database system. Figure 1 As shown, the method includes:

[0030] S101, the data collection node acquires metadata information of the stored data in the allocated data shard and sends the metadata information to the information management node corresponding to the data collection node.

[0031] The data collection node is configured to collect the metadata information of the stored data. The stored data is stored in the data node, and the metadata information of the stored data can be collected by the data collection node. The stored data can be business data, and can also include metadata of the data node. The business data can include user-associated data and task-associated data, etc. The data collection node can be the same as the data node, or can be different from the data node. When a data collection component (Cluster Group Metadata Collector, CGMC) is configured in the data node, the data node can serve as a data collection node to collect the metadata information of the stored data. The stored data in the data node is sharded according to a preset rule to form data shards. One data shard stores part of the data in the data node, or all of the data in the data node. The metadata information refers to data attributes of the data and statistical information generated during running of the data node, and is used to describe information about data definition, running state, and statistics in different dimensions, etc. related key data generated during storage and use of the stored data, such as description information about business, structure, definition, storage, running state, and incremental changes of the stored data.

[0032] The information management node, i.e., a metadata server (Metadata Server, MDS), is responsible for unified management of global metadata information, and is configured to receive, store, and report the metadata information collected by the corresponding data collection node. The information management node corresponds to the data collection node, and the information management node allocates at least one data shard to at least one data collection node. A correspondence between the information management node and the data shards, a correspondence between the data collection nodes and the data shards, and a correspondence between the information management node and the data collection nodes are established. The data collection nodes upload the metadata information of the stored data in the corresponding data shards to the corresponding information management nodes.

[0033] S102, the information management node receives the metadata information.

[0034] Specifically, the information management node receives the metadata information collected by the corresponding data collection node.

[0035] S103, the information management node stores the metadata information in an associated information cache library.

[0036] The information cache library is associated with the information management node, and the information management node stores the received metadata information in the information cache library associated therewith. The information cache library, i.e., a metadata cache library (MDB), is used to store the metadata information received by the information management node. The information cache library can be a memory library, and the data reading and writing speed is fast. When the number of information management nodes is multiple, there is an associated information cache library corresponding to each information management node. The metadata information received by different information management nodes is different, and the metadata information stored in the associated information cache library is different, so that the multiple information cache libraries respectively store the metadata information of different data shards, and the distributed storage of metadata information is realized. The information management node can perform read and write operations on the information cache library associated therewith.

[0037] In S104, the information management node periodically synchronizes the metadata information in the associated information cache library to the information database.

[0038] The periodic synchronization of metadata information can mean that the metadata information is synchronized once every preset fixed time. The preset fixed time is 60 seconds by default and can be configured by the user. For example, the metadata information is sent once every 60 seconds.

[0039] The information database, i.e., a relational database (RDB), is used to store metadata information and can provide global metadata query services in different dimensions. Different dimensions refer to different angles, such as the dimensions of a column of a table and an index of a table. Global metadata refers to the metadata information of all information cache libraries stored in the information database. When there are multiple information management nodes, all information management nodes upload the metadata information in the associated information cache library to the same information database, i.e., the information database stores full metadata information.

[0040] For example, the information management node sends the metadata information in the associated information cache library to the information database once every 60 seconds.

[0041] The technical scheme of the embodiment of the present application acquires the metadata information of the stored data in the allocated data shard by the data collection node, and sends the metadata information to the information management node corresponding to the data collection node, then the information management node receives the metadata information, and then the information management node stores the metadata information in the associated information cache library, and then the information management node periodically synchronizes the metadata information in the associated information cache library to the information database. The above technical scheme pre-stores the metadata information in the information cache library through the information management node, improves the real-time read-write speed of the metadata information, at the same time, periodically uploads the metadata information in the information cache library to the information database, so that the metadata information can be periodically stored in the disk, and the loss of the metadata information in the information cache library can be recovered, the collection, merging and integration of the metadata information of the stored data in the data shard are realized, thereby improving the processing performance of the information database on the metadata, and further improving the robustness of the information database.

[0042] On the basis of the above embodiment, as an optional mode of the embodiment of the present application, the number of information databases is multiple, and the data stored in each information database is the same.

[0043] Specifically, the information database has at least two, and the data stored in each information database is the same, that is, at least one backup of the information database is obtained to realize multiple copy storage data. In addition, each information database stores the data of each information cache library, that is, each information database stores full metadata information. In addition, one information database can be set as a master information database, and the other information databases are slave information databases.

[0044] The above technical scheme sets the number of information databases to be multiple, and the data stored in each information database is the same, and the slave information database can be understood as the backup of the master information database, which can avoid data loss caused by failure of the master information database, and ensures the collection, merging and integration of the metadata information of the stored data in the data shard, thereby improving the robustness of the information database.

[0045] Embodiment two

[0046] Figure 2A A flowchart of a data processing method provided for the embodiment two of the present application, the embodiment of the present application further refines the "data collection node acquires the metadata information of the stored data in the allocated data shard" on the basis of the above embodiment, and provides an optional implementation scheme. As shown in the figure, the data processing method specifically includes: Figure 2A

[0047] ​S201, when detecting that the metadata information of the data stored in the data shard is changed due to the definition change of the data, the information management node generates a metadata collection task of the data stored in the data shard and sends it to the collection group corresponding to the data shard.

[0048] The data stored in the data shard refers to a database object, and the metadata information change caused by the definition change of the data stored in the data shard can refer to the metadata information change caused by the change of the attribute of the database object itself, and can include the metadata information change caused by the change of the identification information, type, length, additional information or shard information of the database object. The shard information can refer to the information of the data shard to which the database object is allocated; the additional information can be the index type and index column information added to the database object.

[0049] The metadata collection task is used to instruct the data collection node to collect the metadata information. The collection group is composed of at least one data collection node. The data shard corresponds to the collection group, and the collection group collects the metadata information of the data stored in the corresponding data shard. Correspondingly, the data shard corresponds to each data collection node in the collection group.

[0050] The DDL change log can be detected to detect whether the definition change of the data stored in the data shard causes the change of the metadata information. For example, when the definition of the data stored in the data shard is newly created or modified, the metadata information is changed, a DDL change log is generated, or the corresponding change information is added in the DDL change log.

[0051] S202, the target collection node in the collection group receives the metadata collection task.

[0052] The target collection node can refer to the data collection node to be received by the metadata collection task, and the target collection node is also used to allocate the data collection node to execute the metadata collection task in the collection group. The target collection node can be a CGMC leader in the collection group. There is only one CGMC leader in each collection group, and the CGMC leader is designated by the information management node corresponding to it. The CGMC leader is responsible for the allocation of the metadata collection task in the collection group, the state monitoring of the data collection component, the fault switching in the collection group, the dynamic capacity expansion and the communication with the MDS. The dynamic capacity expansion refers to dynamically increasing or reducing the number of data collection nodes in the working mode in the current collection group according to the demand.

[0053] S203, the target collection node selects a data collection node in the collection group and allocates the metadata collection task to the data collection node.

[0054] Specifically, the target collection node selects a suitable data collection node in the collection group in which the target collection node is located according to a preset selection rule, and assigns the metadata collection task to the data collection node. The preset selection rule can include selection according to the load capacity of the data collection node, selection according to the working state of the data collection node, selection according to the fault state of the data collection node, or random selection, etc.

[0055] S204, the data collection node collects metadata information of the stored data in the data shard according to the metadata collection task, and sends the metadata information to the information management node corresponding to the data collection node.

[0056] S205, the information management node receives the metadata information.

[0057] S206, the information management node stores the metadata information in the associated information cache library.

[0058] S207, the information management node periodically synchronizes the metadata information in the associated information cache library to the information database.

[0059] The technical scheme of the embodiment of the application determines whether the information management node generates a metadata collection task by detecting the metadata information change caused by the defined change of the stored data in the data shard, avoids repeated collection, and collects metadata information in real time according to the change information, thereby improving the metadata information update efficiency. At the same time, the target collection node selects a data collection node in the collection group and assigns a metadata collection task to the data collection node, thereby realizing effective allocation of the metadata collection task and enabling the metadata collection task to be accurately executed.

[0060] On the basis of the above-mentioned embodiment, as an optional mode of the embodiment of the application, the selection of the data collection node in the collection group by the target collection node is specified as follows: the target collection node acquires nodes in a working mode in the collection group; and the target collection node selects a data collection node from the nodes in the working mode.

[0061] The working mode refers to a mode in which a metadata collection task can be executed. The working mode represents the collection availability of the data collection node. The node in the working mode can normally process a metadata collection task.

[0062] For example, the data collection nodes in the collection group are pre-configured with modes including a working mode according to a preset mode configuration rule of the data collection node. Then, the target collection node acquires data collection nodes in the working mode in the collection group, and selects at least one data collection node from the data collection nodes in the working mode.

[0063] The technical scheme of the embodiment of the present application acquires the nodes in the working mode in the collection group by the target collection node, and then selects the data collection node from the nodes in the working mode by the target collection node. The above technical scheme selects the data collection node in the working mode in the collection group to execute the metadata collection task, can distribute the collection task to the available data collection node in the collection group, and further improves the processing performance of the information database on the metadata.

[0064] On the basis of the above embodiment, as an optional mode of the embodiment of the present application, the data processing method can further include: adjusting the mode type of the nodes included in the collection group by the target collection node according to the node state information of the nodes included in the collection group, the mode type including the working mode or the silent mode, and the node state information including the load state of the nodes of the collection group or the fault state of the nodes in the collection group.

[0065] The silent mode is opposite to the working mode, the working mode refers to the mode that can execute the metadata collection task, and the silent mode refers to the mode that cannot execute the metadata collection task. Specifically, the mode of the data collection node that at least one of the following conditions is met can be configured as the silent mode: the number of tasks being processed reaches the maximum processing number, the data node fails and cannot work normally, or the data collection node is in the energy-saving state.

[0066] The node state information is used for recording the current state of the nodes in the collection group. The load state is used for indicating the state of the nodes in the collection group processing the metadata collection task, and the working capacity of the nodes can be determined according to the load state of the nodes. The fault state can include the fault state and the normal state. The load state of the nodes of the collection group and the fault state of the nodes in the collection group are used for determining the load capacity of the collection group.

[0067] For example, the mode type of the nodes included in the collection group is adjusted by the target collection node according to the load state and the fault state of the nodes included in the collection group. If the number of the nodes in the working mode in the collection group is greater than the number of the nodes in the working mode required by the current metadata collection task, the mode type of the redundant nodes in the collection group is adjusted from the working mode to the silent mode; if the number of the nodes in the working mode in the collection group is less than or equal to the number of the nodes in the working mode required by the current metadata collection task, the mode type of part of the nodes in the silent mode in the collection group is adjusted from the silent mode to the working mode. Similarly, if the load capacity of the collection group is greater than the load capacity required by the current metadata collection task, the mode type of part of the nodes in the working mode in the collection group is adjusted from the working mode to the silent mode; if the load capacity of the collection group is less than or equal to the load capacity required by the current metadata collection task, the mode type of part of the nodes in the silent mode in the collection group is adjusted from the silent mode to the working mode.

[0068] The technical scheme of the embodiment of the present application provides a method for determining the mode type of nodes in a collection group according to node state information of nodes included in the collection group, realizes balanced distribution of metadata collection tasks in the collection group, realizes real-time monitoring of the load state of nodes in the collection group, and thus improves the processing performance of the information database for metadata, avoids data collection failure caused by high load, and improves the stability of the data collection system.

[0069] On the basis of the above embodiment, a flowchart of metadata management is provided, as shown in Figure 2B

[0070] S21, the MDS periodically reads the DDL change log.

[0071] Specifically, the MDS reads the DDL change log once every preset fixed time. The preset time is 60 seconds by default and can be configured by the user.

[0072] S22, it is detected whether there is a metadata object changed in the DDL change log, if yes, S23 is executed, and if no, S21 is executed.

[0073] The metadata object refers to the aforementioned stored data.

[0074] S23, the metadata information in the MDB is updated.

[0075] For example, the record of the metadata object in the MDB is added or marked for deletion.

[0076] S24, the MDS issues a metadata collection task of the incremental object to a collection group.

[0077] The incremental object can refer to the metadata object changed. For example, the MDS divides the metadata object into a certain data shard according to a preset data shard rule. After the incremental object is determined, the data shard to which the incremental object belongs can be determined. The MDS assigns the data shard to a certain collection group in advance and establishes and stores the corresponding relationship between the data shard and the collection group. The metadata collection task is generated according to the incremental object and is issued to the collection group.

[0078] S25, the collection group assigns the metadata collection task of the incremental object.

[0079] ​For example, after receiving the metadata collection task of the incremental object issued by the MDS, the collection group distributes the metadata collection task of the incremental object to the corresponding data collection node according to the preset task distribution rule (for example, the Identity Document (ID) of the incremental object is taken as the remainder). In addition, the data collection node can also be configured in a mode, which can specifically include a working mode and a silent mode. Specifically, the collection group assigns the metadata collection task of the incremental object to the data collection node in the working mode.

[0080] In addition, if the load of the data collection node in the working mode exceeds the load threshold, the data collection node in the silent mode in the collection group is enabled, and the metadata collection task of the incremental object is assigned to the data collection node. In the above metadata management process, the data collection node in the silent mode can be enabled in the collection group. If the enabling fails, an alarm is triggered, so that the collection group still forces the metadata collection task of the incremental object to be issued to the data collection node in the silent mode according to the predetermined rule. The load threshold refers to the number of metadata collection tasks that can be processed by the data collection node in the working mode; the alarm is used to provide prompt information to the user.

[0081] For example, the data collection node in the working mode fails, and there is no data collection node in the working mode, so the data collection node in the silent mode needs to be enabled. When the enabling fails, an alarm is triggered to prompt the user that there is no data collection node in the silent mode available at present. However, the data collection node in the working mode fails and cannot process the metadata collection task, so the collection group still forces the metadata collection task of the incremental object to be issued to the data collection node in the silent mode according to the predetermined rule.

[0082] On the basis of the above embodiment, as an optional mode of the embodiment of the application, the data processing method can further include: the information management node acquires a data query request; the information management node queries in the information cache library associated with the information management node and the information cache library associated with other information management nodes according to the data query request, wherein the collection group corresponding to the information management node and the collection group corresponding to other information management nodes are different, and the collection group corresponding to the information management node includes the data collection node corresponding to the information management node; and in the case that the query result is empty, the information management node queries in the information database according to the data query request.

[0083] The data query request is used to query metadata information. There can be at least two information management nodes. The information management node that is currently receiving the data query request is one of these nodes; other information management nodes are those distinct from it. Both other information management nodes and the information management node periodically report metadata information from their associated information cache to the information database.

[0084] The difference between the data collection groups corresponding to the information management node and those corresponding to other information management nodes means that each information management node corresponds to a different data collection group. The information management node receives metadata information reported by its corresponding data collection group and stores it in the information cache library associated with that node. Other information management nodes receive metadata information reported by their respective data collection groups and store it in their own associated information cache libraries. In practice, since each information management node corresponds to a different data collection group, and the data fragments and metadata information differ between these groups, the data cached in the information cache library associated with the information management node differs from the data cached in the information cache libraries associated with other information management nodes.

[0085] The metadata information retrieved by a data query request can be stored in the information cache repository associated with the information management node, or it can be stored in the information cache repository associated with other information management nodes. Each information management node that receives a data query request can query the metadata information in all information cache repositories.

[0086] Cases where the query result is empty may include situations where the information management node does not find the metadata information corresponding to the stored data query request in its associated information cache library or other information cache libraries associated with other information management nodes, or where the metadata information in the information cache libraries associated with the information management node or other information cache libraries associated with other information management nodes has been cleared. In this case, the metadata information corresponding to the data query request can be queried in the information database.

[0087] In fact, the information cache store data in a cached form, which has a fast read and write speed, while the information database stores data written to disk, which has a slow read and write speed. Prioritizing queries in the information cache and then in the information database can improve data retrieval efficiency.

[0088] The technical scheme of the embodiment of the present application acquires a data query request by an information management node, then the information management node queries in an information cache library associated with the information management node and an information cache library associated with other information management nodes according to the data query request, and then queries in an information database according to the data query request when there is no information in the information cache library (i.e. when the query result is empty), thereby reducing the pressure of information database access, realizing multi-level management of metadata information, preferentially querying data in the cache library with fast read-write speed, and improving data query speed.

[0089] On the basis of the above embodiment, as an optional mode of the embodiment of the present application, the data processing method can further include: when the information management node is a target management node, the target management node acquires the failure state of other information management nodes; when there is an other information management node with a failure state of failure, the target management node allocates the collection group corresponding to the other information management node with the failure state to other information management nodes with a failure state of normal, so that the other information management nodes with the failure state of normal receive the metadata information reported by the collection group corresponding to the other information management node with the failure state.

[0090] The target management node can be an information management node leader (MDS Leader), and generally one MDS Leader exists in a plurality of information management nodes, i.e. the information management node and the other information management nodes, and the remaining information management nodes can be information management node followers (MDS Follower). The target management node is also used to coordinate all information management nodes. In addition, the information management node can also be an information management node follower, and an information management node in the other information management nodes is an information management node leader. The failure state is used to describe whether the node is failure, whether it can work normally, etc. The target management node detects the failure state of other information management nodes by a heartbeat packet or other ways. The failure state can include a failure state and a normal state, etc.

[0091] For example, the target management node acquires the failure state of other information management nodes in a pre-set table about information management nodes, allocates the collection group corresponding to the other information management nodes with the failure state of failure to other information management nodes with the failure state of normal according to the acquired failure state of other information management nodes, so that the other information management nodes with the failure state of normal receive the metadata information reported by the collection group corresponding to the other information management node with the failure state, to avoid the loss of the metadata information reported by the collection group corresponding to the other information management node with the failure state.

[0092] The technical scheme of the embodiment of the present application is that, in the case that the information management node is the target management node, the target management node acquires the fault state of other information management nodes; in the case that there is an other information management node with a fault state of fault, the target management node distributes the collection group corresponding to the other information management node with fault to other information management nodes with a fault state of normal, so that the other information management nodes with a fault state of normal receive the metadata information reported by the collection group corresponding to the other information management node with fault. The above technical scheme realizes the fault switching between information management nodes, avoids the situation that metadata information cannot be reported, avoids data loss, and improves the data transmission accuracy and the metadata information integrity in the database.

[0093] In addition, in the data query process, any one information management node receiving a data query request can be forwarded to the target management node, which preferentially queries in each information cache library and then queries in the information database. The other information management nodes do not perform the query. Alternatively, any one information management node can perform the query, and the performance of the information management node can be specifically limited.

[0094] Embodiment three

[0095] Figure 3A A structural diagram of a data processing system provided by the embodiment three of the present application. The embodiment can be applicable to the case of uniformly collecting, merging and integrating the metadata information of the data stored in the single-computer distributed database data node. The data processing system can be realized in the form of hardware and / or software, and the data processing system can be configured in the distributed database system. As shown in the figure, the data processing system comprises a data collection node 10, an information management node 20, an information cache library 30 associated with the information management node 20 and an information database 40. Figure 3A Figure 3A ​In the figure, there can be multiple data collection nodes 10, information management nodes 20, information caches 30 and information databases 40. The information management nodes 20 correspond to the data collection nodes 10, the information caches 30 correspond to the information management nodes 20, the information management nodes 20 store the received metadata information in the information caches 30 corresponding thereto, when there are multiple information management nodes 20, there are information caches 30 corresponding to each information management node 20, the information management nodes 20 receive different metadata information, and the corresponding information caches 30 store different metadata information. RDB(M) represents the main information database, the corresponding rectangle is gray, indicating that the main information database is currently working; RDB(S) represents the slave information database, MDS represents the information management node, and MDB represents the information cache. The RDB(M) and RDB(S) store full metadata information. The RDB(S) is a copy of the RDB(M).

[0096] The data collection node 10 is configured to obtain metadata information of data stored in the allocated data shard and send the metadata information to the corresponding information management node 20. The information management node 20 is configured to receive the metadata information. The information management node 20 is configured to store the metadata information in the associated information cache 30. The information management node 20 is configured to periodically synchronize the metadata information in the associated information cache 30 to the information database 40.

[0097] The data collection node 10 can be the same as the data node or different from the data node. When the CGMC is configured in the data node, the data node can serve as the data collection node 10.

[0098] The information management node 20 corresponds to the data collection node 10, and the information management node 20 allocates at least one data shard to at least one data collection node 10. The data collection node 10 uploads the collected metadata information of data stored in the allocated data shard to the information management node 20.

[0099] The technical scheme of the embodiment of the present application acquires the metadata information of the stored data in the allocated data shard by the data collection node 10, and sends the metadata information to the information management node 20 corresponding to the data collection node, then the information management node 20 receives the metadata information, and then the information management node 20 stores the metadata information in the associated information cache library 30, and then the information management node 20 periodically synchronizes the metadata information in the associated information cache library 30 to the information database 40. The above data processing system pre-stores the metadata information in the information cache library through the information management node, improves the real-time read-write speed of the metadata information, at the same time, periodically uploads the metadata information in the information cache library to the information database, so that the metadata information can be periodically stored in the disk, and the loss of the metadata information in the information cache library can be recovered, realizes the collection, merging and integration of the metadata information of the stored data in the data shard, improves the processing performance of the metadata in the information database in the system, and improves the robustness of the information database in the system. At the same time, the system can provide the user with the query function of the global metadata information, and realizes the real-time monitoring of the global metadata information.

[0100] On the basis of the above embodiment, as an optional way of the embodiment of the present application, as shown in Figure 3B The data processing system can further include a collection group 50, the collection group 50 including a target collection node 60 and a data collection node 10; the collection group 50 being composed of at least one data collection node. Figure 3B In the figure, the rectangle with gray color and marked with CGMC represents the target data collection node 60, and the other rectangles marked with CGMC represent the data collection node 10, there are six collection groups 50 in the figure, each of the collection groups 50 includes four data collection nodes 10, and each of the collection groups 50 has and only has one target data collection node 60. In addition, one collection group can be configured with more (such as 6) or less (such as 1) data collection nodes, and the number of data collection nodes included in one collection group is not specifically limited.

[0101] The information management node 20 is configured to generate a metadata collection task and send the metadata collection task to the collection group 50 corresponding to the data shard when detecting that the metadata information is changed due to the defined change of the stored data in the data shard.

[0102] The metadata information change caused by the definition change of the data stored in the data shard can be a metadata information change caused by a change in the attribute of the database object itself, and can include a metadata information change caused by a change in the identification information, type, length, additional information, or shard information of the database object.

[0103] For example, when the metadata information of the data stored in the data shard changes due to a definition change, a DDL change log is generated. The information management node 20 periodically reads the DDL change log, and when it is checked that the metadata information of the data stored in the data shard has changed, the metadata information in the information cache library 30 is first updated, and then the information management node 20 generates a metadata collection task of the changed metadata information according to a preset storage data shard rule, and sends it to the collection group 50 corresponding to the data shard.

[0104] The target collection node 60 is configured to receive the metadata collection task, and the target collection node 60 is configured to select a data collection node 10 in the collection group 50 and assign the metadata collection task to the data collection node 10.

[0105] Specifically, the target collection node 60 selects a suitable data collection node 10 in the collection group according to a selection rule, and assigns the metadata collection task to the data collection node 10. The selection rule can include selection according to the load capacity of the data collection node 10, selection according to the working state of the data collection node 10, selection according to the fault state of the data collection node 10, or random selection, etc.

[0106] For example, the target collection node 60 selects a data collection node 10 in the collection group according to the load capacity and working state of the data collection node 10. If the load capacity of the data collection node 10 is redundant and the working state of the data collection node 10 is in the working mode, the data collection node 10 can be selected; if the load capacity of the data collection node 10 is insufficient or the working state of the data collection node 10 is in the silent mode, the data collection node 10 cannot be selected. One data collection node 10 is selected from the data collection nodes 10 that can be selected, and the metadata collection task is assigned to the data collection node 10.

[0107] The data collection node 10 is configured to collect the metadata information of the data stored in the data shard according to the metadata collection task.

[0108] The technical scheme of the embodiment of the present application adds a collection group to the system, allocates metadata collection tasks to the collection group, and uses the collection group as the smallest metadata information management unit, selects data collection nodes in the collection group through a target collection node, and allocates metadata collection tasks to the data collection nodes, thereby realizing load balancing of the data collection nodes in the collection group, realizing effective allocation of the metadata collection tasks, and enabling the metadata collection tasks to be accurately executed. Meanwhile, the system can be used for early warning.

[0109] In addition, the overall flowchart of the collection, merging and integration of the metadata information of the stored data in the data shard by the data processing system is shown in Figure 3C as follows, and specifically includes:

[0110] S31, the system installation stage allocates MDS and a collection group and the mapping relationship therebetween; the CGMC is configured to the data node, the CGMC is allocated a role, and the data node is allocated managed storage data.

[0111] The MDS is a global manager of metadata information, and includes two roles, MDS Leader and MDS Follower. There is only one MDS Leader globally, which is used to coordinate all information management nodes to report metadata information to an information database regularly. The MDS Follower is used to execute the task allocated by the MDS Leader. The mapping relationship between the MDS and the collection group can be that the CGMC Leader and the CGMC Follower are allocated to the collection group according to the similar rules of the MDS allocation role. The CGMC Leader is responsible for the allocation of the metadata collection task in the collection group, the monitoring of the state of the data collection component, the fault switching in the collection group, the dynamic expansion and contraction, and the communication with the MDS. The CGMC Follower collects the metadata information of the data node thereof regularly according to the metadata collection task issued by the CGMC Leader, and reports the same to the CGMC Leader.

[0112] S32, the CGMC collects the metadata information of the managed storage data thereof regularly, and reports the same to the CGMC Leader.

[0113] The CGMC can be the CGMC Follower, or the CGMC Leader itself.

[0114] S33, the CGMC Leader is responsible for the integration, proofreading and reporting of the metadata information in the collection group, and is responsible for the management of the CGMC in the collection group and the allocation of the metadata collection task.

[0115] In addition, the CGMC Leader re-connects communication with the CGMC that does not timely report, and after connection timeout fails, lists the CGMC in a to-be-assigned list, triggers a metadata collection task re-assignment mechanism of the CGMC Leader, or triggers a standby CGMC to take over the failed metadata collection task, re-executes the metadata collection task, and reports to the MDS after collection succeeds.

[0116] S34, each MDS manages the reporting flow of the collection group and the integration flow of the metadata information in the collection group.

[0117] For example, each MDS checks the metadata information in the collection group, adds a timestamp of metadata information update to the metadata information that has changed, and uploads the updated metadata information to the MDB.

[0118] S35, the MDS Leader is responsible for the overall scheduling of global metadata information, and periodically aggregates the global metadata information to the main RDB.

[0119] Specifically, the MDS Leader is responsible for the collection, merging and integration of global metadata information, and periodically aggregates the integrated global metadata information to the main RDB.

[0120] On the basis of the above embodiment, as an optional mode of the embodiment of the application, the data processing system, the target information management node 20, can further include: other information management nodes 70, such as Figure 3D As shown, the rectangle marked with MDS and in gray color is the target information management node 20, the corresponding associated MDB is the target information cache 30, and the other rectangle marked with MDS is the other information management node 70, and the corresponding associated MDB is the other information cache 80.

[0121] The target information management node 20 is configured to acquire the fault state of the other information management node 70, and the target information management node 20 is configured to, in the case that there is the other information management node 70 in the fault state of failure, assign the collection group 50 corresponding to the other information management node 70 in the fault state of failure to the other information management node 70 in the fault state of normal, so that the other information management node 70 in the fault state of normal receives the metadata information reported by the collection group 50 corresponding to the other information management node 70 in the fault state of failure.

[0122] The target information management node 20, i.e., the MDS Leader, is configured to coordinate all information management nodes to periodically report metadata information to the information database. The fault state is comprehensively judged by the process state and the link state.

[0123] The technical scheme of the embodiment of the present application distinguishes the information management nodes in the data processing system, divides them into target management nodes and other information management nodes, and gives them different functions, so that the global metadata information can be uniformly reported to the information database by the target management nodes, ensuring the consistency of the global metadata information, and realizing the unified collection, merging and integration of the metadata information of the stored data in the data shards.

[0124] The data processing system provided by the embodiment of the present application can execute the data processing method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of executing each data processing method.

[0125] Embodiment four

[0126] Figure 4 A structural schematic diagram of an electronic device 400 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0127] As shown in Figure 4 The electronic device 400 includes at least one processor 401, and a memory, such as a read-only memory (ROM) 402, a random access memory (RAM) 403, etc., which is communicatively connected to the at least one processor 401, wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 401 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 402 or the computer program loaded from the storage unit 408 into the random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0128] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0129] The processor 401 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 401 performs various methods and processes described above, such as a data processing method.

[0130] In some embodiments, the data processing method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded onto the RAM 403 and executed by the processor 401, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the processor 401 can be configured to perform the data processing method by any other appropriate means, such as by means of firmware.

[0131] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0132] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package and partially on a remote machine or entirely on a remote machine or server.

[0133] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0135] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0136] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0137] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in series, or executed in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0138] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: The data collection node obtains metadata information of the data stored in the allocated data shard and sends the metadata information to the information management node corresponding to the data collection node; The information management node receives the metadata information; The information management node stores the metadata information in the associated information cache library; The information management node periodically synchronizes the metadata information in the associated information cache library to the information database; The data collection node obtains metadata information of the data stored in the allocated data shard, which comprises the following steps: When detecting that the metadata information of the data stored in the data shard changes due to a definition change of the data, the information management node generates a metadata collection task of the data stored in the data shard and sends the metadata collection task to the collection group corresponding to the data shard; wherein the data stored in the data shard is a database object, and the metadata information change caused by the definition change of the data stored in the data shard refers to the change of the metadata information caused by the change of the attribute of the database object; The target collection node in the collection group receives the metadata collection task; The target collection node selects the data collection node in the collection group and allocates the metadata collection task to the data collection node; The data collection node collects the metadata information of the data stored in the data shard according to the metadata collection task.

2. The method of claim 1, wherein, The target collection node selects the data collection node in the collection group, which comprises the following steps: The target collection node obtains the nodes in the working mode in the collection group; The target collection node selects the data collection node from the nodes in the working mode.

3. The method of claim 2, wherein, The method further comprises the following steps: The target collection node adjusts the mode type of the nodes included in the collection group according to the node state information of the nodes included in the collection group, wherein the mode type comprises a working mode or a silent mode, and the node state information comprises a load state of the nodes in the collection group or a fault state of the nodes in the collection group.

4. The method of claim 1, wherein, The number of information databases is multiple, and the data stored in each information database is the same.

5. The method of claim 1, wherein, The method further comprises the following steps: The information management node obtains a data query request; The information management node queries in the information cache library associated with the information management node and the information cache library associated with other information management nodes according to the data query request, wherein the collection group corresponding to the information management node is different from the collection group corresponding to the other information management nodes, and the collection group corresponding to the information management node comprises the data collection node corresponding to the information management node; In the case that the query result is empty, the information management node queries in the information database according to the data query request.

6. The method of claim 1, wherein, Further comprising the following steps: In the case that the information management node is a target management node, the target management node obtains the fault state of other information management nodes; In the case that the other information management node in the fault state is faulty, the target information management node allocates the collection group corresponding to the faulty other information management node to the other information management node in the normal state, so that the other information management node in the normal state receives the metadata information reported by the collection group corresponding to the faulty other information management node.

7. A data processing system, characterized by The data processing method is applied to any one of claims 1-6, and comprises a data collection node, an information management node, an information cache library and an information database associated with the information management node; The data collection node is configured to acquire metadata information of stored data in the allocated data shard and send the metadata information to the corresponding information management node; The information management node is configured to receive the metadata information; The information management node is configured to store the metadata information in the associated information cache library; The information management node is configured to periodically synchronize the metadata information in the associated information cache library to the information database.

8. The data processing system of claim 7, wherein, Further comprising: A collection group, the collection group comprising a target collection node and the data collection node; The information management node is configured to generate a metadata collection task and send the metadata collection task to the collection group corresponding to the data shard when detecting that the metadata information of stored data in the data shard is changed; The target collection node is configured to receive the metadata collection task; The target collection node is configured to select the data collection node in the collection group and allocate the metadata collection task to the data collection node; The data collection node is configured to collect the metadata information of stored data in the data shard according to the metadata collection task.

9. The data processing system of claim 7, wherein, The information management node is a target information management node, and further comprises an other information management node; The target information management node is configured to acquire a fault state of the other information management node; In the case that the other information management node in the fault state is faulty, the target information management node allocates the collection group corresponding to the faulty other information management node to the other information management node in the normal state, so that the other information management node in the normal state receives the metadata information reported by the collection group corresponding to the faulty other information management node.

Images