A data storage system and method

Abstract

Embodiments of the present application provide a data storage system and method, the data storage system comprises a data input end, a master node, at least two slave nodes, a first database and a second database, the master node and the at least two slave nodes are communicatively connected, and each of the slave nodes is communicatively connected, the master node is used for reading initial data from the data input end and sending the initial data to the at least two slave nodes, the at least two slave nodes are used for calculating the initial data according to a preset data processing flow to obtain device data and metadata, the master node is further used for receiving the metadata and storing the metadata in a mode into a HIVE table of the first database, and the at least two slave nodes are further used for storing the device data into the second database, the metadata calculated is stored in a mode, the problems of low data storage efficiency and high cost caused by using a JSON format for storage are avoided, and the semantic integrity of the initial data is ensured, and the readability of the data is not affected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular, relates to a data storage system and method. BACKGROUND

[0002] Industrial Internet is a new generation of information communication technology and industrial economy deep integration of new infrastructure, application mode and industrial ecology, through the overall connection of people, machines, things, systems, etc., a new manufacturing and service system covering the whole industry chain and the whole value chain is built, which provides an implementation approach for the development of industrial digitalization, networking and intelligence. It is obvious that the importance of developing industrial internet cannot be overstated. With the development of industrial internet, the data of industrial devices grows rapidly and is huge in size, and the storage of industrial device data has become a new problem.

[0003] The traditional storage method often converts industrial device data into row-column storage, which easily destroys the semantic integrity of the data, resulting in poor readability of the data, and is often stored in JSON format, which is low in data storage efficiency and high in cost. Therefore, a data storage scheme is urgently needed to ensure the storage integrity of industrial device data and improve the storage efficiency of industrial device data. SUMMARY

[0004] Therefore, one of the purposes of the present application is to provide a data storage system and method, which can at least solve part of the above technical problems.

[0005] In a first aspect, the embodiments of the present application provide a data storage system, which comprises a data input end, a master node, at least two slave nodes, a first database and a second database, the master node and the at least two slave nodes are in communication connection, and each slave node is in communication connection with each other, wherein,

[0006] The master node is configured to read initial data from the data input end and send the initial data to the at least two slave nodes, and the at least two slave nodes are configured to calculate the initial data according to a preset data processing flow to obtain device data and metadata corresponding to the device data.

[0007] The master node is further configured to receive the metadata and store the metadata in a mode into a HIVE table of the first database, and the at least two slave nodes are further configured to store the device data into the second database.

[0008] In a possible implementation, the first database is configured to check the mode according to a preset mode table, wherein,

[0009] If the mode is a brand new mode, the first database is configured to generate a new HIVE table and store the brand new mode into the new HIVE table.

[0010] In a possible implementation, the preset data processing procedure comprises at least one of data cleaning, data grouping and data aggregation, wherein,

[0011] The target node is configured to perform data cleaning on the target data to filter out abnormal target data.

[0012] The target node is further configured to obtain device identifiers in the target data and send target data of the same device identifier to the same target node.

[0013] The target node is further configured to calculate the target data according to a preset clustering algorithm to obtain the device data and metadata of the device data.

[0014] The target node is any of the slave nodes, the target data is initial data received by the target node, and the abnormal target data comprises initial data lacking the device data and / or the metadata.

[0015] In a possible implementation, each slave node is further configured to read the initial data from the data input end, and in a case where the master node is down and the master node and each slave node still fail to recover communication within a preset time period, the at least two slave nodes are further configured to select one slave node as a new master node from the slave nodes according to a preset selection mechanism.

[0016] In a possible implementation, the data storage system further comprises a metadata buffer configured to buffer the metadata calculated by each slave node, and the master node and the at least two slave nodes are respectively connected to the metadata buffer in communication.

[0017] In a possible implementation, the master node is further configured to send the initial data to each slave node in equal amounts according to a preset distribution algorithm, and the preset distribution algorithm comprises at least one of data size sharding and Hashing according to a preset condition.

[0018] In a possible implementation, the master node is further configured to read the initial data from the data input end in a polling manner.

[0019] In a second aspect, an embodiment of the present application provides a data storage method, applied to a data storage system, the data storage system comprising a data input end, a master node, at least two slave nodes, a first database and a second database, the master node and the at least two slave nodes being connected in communication, and each slave node being connected in communication, the data storage method comprising:

[0020] The master node reads initial data from the data input end and sends the initial data to the at least two slave nodes, and the at least two slave nodes calculate the initial data according to a preset data processing flow to obtain the device data and the metadata corresponding to the device data;

[0021] The master node receives the metadata and stores the metadata in a mode into a HIVE table of the first database, and the at least two slave nodes are further configured to store the device data into the second database.

[0022] In a possible implementation, the data storage method comprises:

[0023] The first database is configured to verify the mode according to a preset mode table;

[0024] If the mode is a brand-new mode, the first database generates a new HIVE table and stores the brand-new mode into the new HIVE table.

[0025] In a possible implementation, the data storage method comprises:

[0026] The master node sends the initial data to each slave node according to a preset distribution algorithm, wherein the preset distribution algorithm comprises at least one of data size sharding and Hash according to a preset condition.

[0027] The data storage system and method provided by the embodiments of the present application can obtain data from a data input end and read initial data, and calculate device data of the initial data and metadata corresponding to the device data, and then each slave node stores the calculated device data into a second database, and the calculated metadata is stored into a HIVE table in a first database in a mode, thereby avoiding the problem of low data storage efficiency and high cost caused by using a JSON format for storage, and the mode storage mode ensures the semantic integrity of the initial data and does not affect the readability of the data. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0029] Figure 1 Fig. 1 shows a functional module schematic diagram of a data storage system provided by an embodiment of the present application;

[0030] Figure 2 A method flow diagram of a data storage method provided by an embodiment of the present application is shown.

[0031] Icon:

[0032] Data input end 100; master node 110; slave node 120; first database 130; second database 140. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0034] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor fall within the scope of protection of the present application.

[0035] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0036] In the description of the present application, it should be noted that if the terms "upper", "lower", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is usually placed, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, therefore, it cannot be understood as a limitation on the present application.

[0037] In addition, if the terms "first", "second" and the like appear, they are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0038] It should be noted that the features in the embodiments of the present application can be combined with each other without conflict.

[0039] Please refer to Figure 1 , Figure 1A functional module schematic diagram of a data storage system provided by an embodiment of the present application is provided. The data storage system comprises a data input end 100, a master node 110, at least two slave nodes 120, a first database 130 and a second database 140. The master node 110 and the at least two slave nodes 120 are communicatively connected, and each of the slave nodes 120 is communicatively connected with each other. Wherein,

[0040] The master node 110 is configured to read initial data from the data input end 100 and send the initial data to the at least two slave nodes 120. The at least two slave nodes 120 are configured to calculate the initial data according to a preset data processing flow to obtain device data and metadata corresponding to the device data.

[0041] The master node 110 is further configured to receive the metadata and store the metadata in a mode in a HIVE table of the first database 130. The at least two slave nodes 120 are further configured to store the device data in the second database 140.

[0042] In the embodiment, the master node 110 and the at least two slave nodes 120 can constitute a master-slave architecture.

[0043] The data input end 100 is a data input interface of the data storage system and can be connected with a client providing data. The initial data in the embodiment generally refers to machine data in the industrial field. In the industrial field, there are many types of machines, and the storage form of data related to the machines not only affects the efficiency of data storage, but also affects the query efficiency of subsequent data.

[0044] The master node 110 and the slave node 120 can constitute a master-slave architecture, and the master node 110 and the slave node 120 can be understood as a master-slave mode. The master node 110 can be regarded as a master node, which is a master node of a computing task and is mainly responsible for scheduling and state management of the computing task, and is the brain of the entire computing task. The slave node 120 can be regarded as a slave node, which is an execution node of the computing task and is responsible for executing the initial data distributed by the master node.

[0045] The first database 130 and the second database 140 mainly implement the storage function of data, wherein the first database 130 can be used to store the metadata calculated by the node 120, and the second database 140 can be used to store the device data calculated by the node 120. It should be noted that each node 120 independently performs a computing task, and the efficiency of data processing is high. After each node 120 completes the calculation, the device data calculated by each node 120 can be stored in the second database 140 independently. The metadata calculated by each node 120 is stored in the HIVE table of the first database 130 in the form of a schema. In the embodiment, the "schema" is schema, that is, the metadata in the embodiment is stored in the HIVE table of the first database 130 in the form of schema, which avoids the problem caused by the storage in the form of JSON in the traditional mode, does not damage the semantics of the initial data, and has high storage efficiency. Subsequent queries will also be more convenient.

[0046] Optionally, the data storage system further comprises a metadata buffer, and the metadata buffer is used to buffer the metadata calculated by each node 120, wherein the master node 110 and the at least two nodes 120 are respectively connected with the metadata buffer in communication.

[0047] In addition to the communication connection between the master node 110 and each node 120, the master node 110 and each node 120 can also be connected through the metadata buffer in the embodiment. The metadata buffer can be used to temporarily store the metadata calculated by each node 120, and the master node 110 does not need to store the metadata, which can reduce the data cache pressure of the master node 110.

[0048] In summary, the data storage system provided by the embodiment of the application can obtain data from the data input end and read initial data by establishing a communication connection between the master node and the slave node, and calculate the device data of the initial data and the metadata corresponding to the device data. Then each node stores the calculated device data to the second database, and stores the calculated metadata in the HIVE table in the first database in the form of schema, which avoids the problem of low data storage efficiency and high cost caused by the storage in the form of JSON, and ensures the semantic integrity of the initial data in the form of schema storage, and does not affect the readability of the data.

[0049] In a possible implementation, the first database 130 is configured to verify the schema according to a preset schema table.

[0050] If the schema is a brand new schema, the first database 130 is configured to generate a new HIVE table and store the brand new schema in the new HIVE table.

[0051] In the embodiment, the preset schema table is stored in the first database 130, the first database 130 can compare the schema stored by the master node 110 with the preset schema table, if the schema information is not in the preset schema table, the first database 130 can generate a new HIVE table and store the schema information not in the preset schema table into the newly generated HIVE table. Moreover, if the first database 130 detects that the schema stored by the master node 110 and the schema in the preset schema table are not completely consistent, the first database 130 can find the HIVE table corresponding to the schema that is not completely consistent and store the schema stored by the master node 110 into the HIVE table to replace the originally stored schema, thereby realizing the update of the HIVE table.

[0052] In a possible implementation, the preset data processing flow includes at least one of data cleaning, data grouping and data aggregation.

[0053] The target node is configured to perform data cleaning on the target data to filter out abnormal target data.

[0054] The target node is further configured to acquire device identifiers in the target data and send target data with the same device identifier to the same target node.

[0055] The target node is further configured to calculate the target data according to a preset clustering algorithm to obtain the device data and metadata of the device data.

[0056] In the embodiment, the target node is any one of the slave nodes 120, the target data is initial data received by the target node, and the abnormal target data includes initial data lacking the device data and / or the metadata.

[0057] In the embodiment, the slave node 120 calculates the initial data distributed by the master node 110 to obtain the device data and metadata of the device data in the initial data. Specifically, data cleaning can filter out abnormal databases. For example, the A group initial data {“t”: “”, “a”: “2”, “b”: “bb”,} is taken as an example, where t, a and b can represent metadata, 2 represents device data of the metadata a, and bb represents device data of the metadata b. Obviously, the device data of the metadata t is missing, and in this case, the A group initial data can be determined as abnormal data. Correspondingly, if any of the metadata t, a and b is missing, the A group initial data can also be determined as abnormal data.

[0058] The initial data is cleaned, and the initial data of the same device type is allocated to the same slave node 120 for processing according to the device type corresponding to the initial data. The same slave node 120 processes the initial data of the same device type, which can obviously speed up the data processing speed of the slave node 120.

[0059] In a possible implementation, each slave node 120 is further configured to read the initial data from the data input end 100. In the case that the master node 110 is down and the master node 110 and each slave node 120 still fail to recover communication within a preset time period, the at least two slave nodes 120 are further configured to select one slave node 120 from each slave node 120 as a new master node 110 according to a preset selection mechanism.

[0060] The master node 110 and each slave node 120 are in communication connection, and each slave node 120 is also in direct communication connection. When the master node 110 fails and cannot work normally, the data storage will be inevitably affected. In this case, a new master node 110 needs to be determined from the normally working slave nodes 120 to ensure the normal progress of data storage. Specifically, in this embodiment, when the communication between the slave node 120 and the master node 110 is interrupted and still fails to recover communication within a preset time period, such as 2 minutes, each slave node 120 can select one slave node 120 from all slave nodes 120 as the master node 110 according to a preset selection mechanism. The preset selection mechanism can adopt the master-slave selection mechanism of Zookeeper.

[0061] Optionally, the master node 110 is further configured to send the initial data to each slave node 120 in equal amounts according to a preset allocation algorithm. The preset allocation algorithm includes at least one of data size fragmentation and Hash according to a preset condition. Specifically, the algorithm used in data size fragmentation includes a ring cutting algorithm of Hash, and the preset condition includes device type information.

[0062] In this embodiment, the master node 110 can allocate all initial data to each slave node 120 in equal amounts according to the preset allocation algorithm, which realizes full use of the slave nodes 120 and improves the data storage efficiency.

[0063] Optionally, the master node 110 is further configured to read the initial data from the data input end 100 in a polling manner.

[0064] Specifically, the master node 110 polls to obtain the initial data, which realizes automatic data acquisition of the master node 110 and also improves the data storage efficiency.

[0065] In summary, the data storage system provided by the embodiment of the application can obtain data from the data input end and read initial data, and calculate device data of the initial data and metadata corresponding to the device data, and then each slave node stores the calculated device data to the second database and stores the calculated metadata to the HIVE table in the first database in a mode, thereby avoiding the problem of low data storage efficiency and high cost caused by using the JSON format for storage, and ensuring the semantic integrity of the initial data in the mode storage manner and not affecting the readability of the data.

[0066] Corresponding to the method embodiment, the application further provides a data storage method. Please refer to Figure 2 , Figure 2 The method flowchart of the data storage method provided by the embodiment of the application can be applied to the data storage system in the above embodiment, the data storage system comprises a data input end 100, a master node 110, at least two slave nodes 120, a first database 130 and a second database 140, the master node 110 and the at least two slave nodes 120 are in communication connection, each slave node 120 is in communication connection, and the data storage method comprises the following steps.

[0067] S210, the master node 110 reads initial data from the data input end 100 and sends the initial data to the at least two slave nodes 120, and the at least two slave nodes 120 calculate the initial data according to a preset data processing flow to obtain device data and metadata corresponding to the device data;

[0068] S220, the master node 110 receives the metadata and stores the metadata to the HIVE table of the first database 130 in a mode, and the at least two slave nodes 120 are further used to store the device data to the second database 140.

[0069] The specific implementation process in the embodiment can refer to the description of the above data storage embodiment, which will not be repeated here.

[0070] In a possible implementation, the data storage method comprises:

[0071] The first database 130 is used to verify the mode according to a preset mode table;

[0072] If the mode is a brand-new mode, the first database 130 generates a new HIVE table and stores the brand-new mode to the new HIVE table.

[0073] The specific implementation process in the embodiment can refer to the description of the above data storage embodiment, which will not be repeated here.

[0074] Optionally, the data storage method comprises:

[0075] The master node 110 sends the initial data to each slave node 120 in equal amounts according to a preset allocation algorithm, wherein the preset allocation algorithm comprises at least one of data size slicing and Hashing according to a preset condition.

[0076] The data storage method provided by the embodiments of the present application can obtain data from a data input end and read initial data, and calculate device data of the initial data and metadata corresponding to the device data, and then each slave node stores the calculated device data to a second database and stores the calculated metadata to a HIVE table in the first database in a mode, thereby avoiding the problem of low data storage efficiency and high cost caused by using a JSON format for storage, and ensuring the semantic integrity of the initial data in a mode storage manner and not affecting the readability of the data.

[0077] The above merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data storage system, characterized by, The data storage system comprises a data input end, a master node, at least two slave nodes, a first database and a second database, the master node and the at least two slave nodes are in communication connection, and each slave node is in direct communication connection with another, wherein The master node is configured to read initial data from the data input end, the initial data refers to machine data in the industrial field, and send the initial data to the at least two slave nodes, and the at least two slave nodes are configured to calculate the initial data according to a preset data processing flow to obtain device data and metadata corresponding to the device data; The master node is further configured to receive the metadata and store the metadata in a Schema mode into a HIVE table of the first database, and the at least two slave nodes are configured to store the device data into the second database which is independent of the first database; Each slave node is further configured to read the initial data from the data input end, and in the case that the master node is down and the master node and each slave node still fail to restore communication within a preset time period, the at least two slave nodes are further configured to select one slave node as a new master node according to a Zookeeper mechanism, and continue to receive the metadata generated by the slave node by the new master node; The first database is configured to check the mode according to a preset mode table, wherein if the mode is a brand-new mode, the first database is configured to generate a new HIVE table and store the brand-new mode into the new HIVE table; The data storage system further comprises a metadata buffer, and the metadata buffer is configured to buffer the metadata calculated by each slave node, wherein the master node and the at least two slave nodes are in communication connection with the metadata buffer respectively.

2. The data storage system of claim 1, wherein, The preset data processing flow comprises at least one of data cleaning, data grouping and data aggregation, wherein The target node is configured to perform data cleaning on target data to filter out abnormal target data; The target node is further configured to obtain device identifiers in the target data and send target data with the same device identifier to the same target node; The target node is further configured to calculate the target data according to a preset clustering algorithm to obtain the device data and metadata of the device data; The target node is any one of the slave nodes, the target data is the initial data received by the target node, and the abnormal target data includes initial data lacking the device data and / or the metadata.

3. The data storage system of claim 1, wherein, The master node is further configured to send the initial data to each slave node in equal amounts according to a preset allocation algorithm, wherein the preset allocation algorithm comprises at least one of data size sharding and Hashing according to a preset condition.

4. The data storage system of claim 1, wherein, The master node is further configured to read the initial data from the data input end in a polling manner.

5. A data storage method, characterized by, The data storage method is applied to a data storage system, the data storage system comprises a data input end, a master node, at least two slave nodes, a first database and a second database, the master node and the at least two slave nodes are in communication connection, and each slave node is in direct communication connection with another, and the data storage method comprises: The master node reads initial data from the data input end, the initial data refers to machine data in the industrial field, and sends the initial data to the at least two slave nodes, the at least two slave nodes calculate the initial data according to a preset data processing flow to obtain device data and metadata corresponding to the device data; The master node receives the metadata and stores the metadata in a Schema mode into a HIVE table of the first database, and the at least two slave nodes are configured to store the device data into the second database which is independent of the first database; Each slave node is also configured to read the initial data from the data input end, and in a case that the master node is down and the master node and each slave node still fail to restore communication within a preset time period, the at least two slave nodes are also configured to select one slave node as a new master node from each slave node according to a Zookeeper mechanism, and continue to receive the metadata generated by the slave nodes by the new master node; The first database is configured to check the mode according to a preset mode table, and if the mode is a brand-new mode, the first database is configured to generate a new HIVE table and store the brand-new mode into the new HIVE table; The data storage system further comprises a metadata buffer configured to buffer the metadata calculated by each slave node, and the master node and the at least two slave nodes are respectively communicatively connected with the metadata buffer.

6. The data storage method of claim 5, wherein, The data storage method comprises: The first database is configured to check the mode according to a preset mode table; If the mode is a brand-new mode, the first database generates a new HIVE table and stores the brand-new mode into the new HIVE table.

7. The data storage method of claim 5, wherein, The data storage method comprises: The master node sends the initial data to each slave node according to a preset distribution algorithm, and the preset distribution algorithm comprises at least one of data size sharding and Hashing according to a preset condition.

Images