Data management method and related device

Abstract

The invention discloses a data management method and a related device, and relates to the technical field of the Internet of Things, and the method comprises the steps: a cluster collection device in a distributed management system collects real-time values of a plurality of data points, converges the real-time values of associated data points in the plurality of data points to a same group of network switches, and transmits the real-time values to a server; therefore, the federated database node in the distributed management system receives the real-time value of the associated data point from the corresponding group of network switches, and stores the received real-time value of the associated data point into the database of the federated database node. According to the method, the real-time values of the associated data points are transmitted to the same federated database node, so that the degree of influence of network delay and node performance on the associated data points is the same, and the problem of real-time consistency is solved; and the autonomous federated database nodes are adopted, so that when one federated database node is abnormal, services on other federated database nodes are not influenced, only the abnormal federated database node is maintained, and the operation cost is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of Internet of Things, and in particular to a data management method and related devices. Background Art

[0002] With the rapid development of the Internet of Things, the generation and processing of massive real-time data has become a major challenge. Traditional centralized data management methods have performance bottlenecks and poor reliability problems, while distributed database management technology can effectively solve these problems and improve the flexibility and scalability of the system.

[0003] At present, distributed database management technology generally uses consistent hashing algorithms to evenly divide the data of IoT devices into various distributed nodes for storage. This division method allows some data with logical correlation to be divided into different distributed nodes. Affected by network delays and node performance, data with logical correlation may have real-time consistency issues when stored in different distributed nodes. In addition, distributed database management technology using consistent hashing algorithms also has the problem of high operating costs. Summary of the invention

[0004] In view of the above problems, the present application provides a data management method and related devices to achieve the purpose of real-time consistency of data storage with logical correlation and reduce operating costs. The specific scheme is as follows:

[0005] The first aspect of the present application provides a data management method, which is applied to a cluster collection device in a distributed management system, comprising:

[0006] Collecting real-time values ​​of a plurality of data points, wherein each of the plurality of data points has a globally unique identifier, the globally unique identifier includes a correlation factor identifier, and the correlation factor identifiers of associated data points having logical correlation among the plurality of data points are the same;

[0007] The real-time values ​​of the associated data points among the multiple data points are aggregated to the same group of network switches, so that the federal database nodes in the distributed management system receive the real-time values ​​of the associated data points from the corresponding group of network switches, and store the received real-time values ​​of the associated data points in the database of the federal database nodes.

[0008] In a possible implementation, the correlation factor identifier refers to an identifier of a correlation factor, and the correlation factor includes one or more of the following factors: a source of a data point, a collection protocol of a data point, and a scope of action of a data point.

[0009] In a possible implementation, the globally unique identifier is composed of a correlation factor identifier and a data point identifier, and the globally unique identifier is defined using a uniform resource identifier standard specification.

[0010] In a possible implementation, aggregating the real-time values ​​of the associated data points among the multiple data points to the same group of network switches includes:

[0011] A wired network communication method is used to aggregate the real-time values ​​of the associated data points to the same group of network switches;

[0012] And / or, the real-time values ​​of the associated data points are aggregated to the same group of network switches using a wireless network communication method.

[0013] In a possible implementation, the same group of network switches includes multiple network switches, and the multiple network switches are in a stacked structure.

[0014] A second aspect of the present application provides a data management method, which is applied to a federated service node in a distributed management system, comprising:

[0015] Get the globally unique identifier of the data point to be accessed;

[0016] Extracting a correlation factor identifier from the global unique identifier of the data point to be accessed as a target correlation factor identifier;

[0017] The federal database node corresponding to the target correlation factor identifier is queried from a preconfigured correlation factor identifier directory, so as to process the data point to be accessed based on the queried federal database node, wherein the queried federal database node is used to receive the real-time value of the associated data point collected by the cluster collection device from the corresponding group network switch and store it in its own database, and the associated data point refers to a data point with logical correlation and the same correlation factor identifier among multiple data points collected by the cluster collection device.

[0018] A third aspect of the present application provides a data management device, which is applied to a cluster collection device in a distributed management system, comprising:

[0019] A data collection unit, used for collecting real-time values ​​of a plurality of data points, wherein each of the plurality of data points has a globally unique identifier, the globally unique identifier includes a correlation factor identifier, and the correlation factor identifiers of associated data points having logical correlation among the plurality of data points are the same;

[0020] A data storage unit is used to aggregate the real-time values ​​of the associated data points among the multiple data points to the same group of network switches, so that the federal database nodes in the distributed management system receive the real-time values ​​of the associated data points from the corresponding group of network switches and store the received real-time values ​​of the associated data points in the database of the federal database nodes.

[0021] A fourth aspect of the present application provides a data management device, which is applied to a federated service node in a distributed management system, including:

[0022] An identification acquisition unit, used to acquire a globally unique identification of a data point to be accessed;

[0023] An identifier extraction unit, used to extract a correlation factor identifier from the global unique identifier of the data point to be accessed as a target correlation factor identifier;

[0024] A data processing unit, used for querying the federal database node corresponding to the target correlation factor identifier from a preconfigured correlation factor identifier directory, so as to process the data point to be accessed based on the queried federal database node, wherein the queried federal database node is used to receive the real-time value of the associated data point collected by the cluster collection device from the corresponding group network switch and store it in its own database, and the associated data point refers to a data point with logical correlation and the same correlation factor identifier among multiple data points collected by the cluster collection device.

[0025] A fifth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0026] The memory is used to store computer programs;

[0027] The processor is used to execute the computer program so that the electronic device can implement the data management method of the first aspect or any implementation of the first aspect, or implement the data management method of the second aspect.

[0028] In a sixth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the data management method of the first aspect or any implementation method of the first aspect, or implement the data management method of the second aspect.

[0029] By means of the above technical scheme, the data management method provided by the present application can collect the real-time values ​​of multiple data points through the cluster collection device in the distributed management system, and aggregate the real-time values ​​of the associated data points in the multiple data points to the same group of network switches, so that the federal database node in the distributed management system receives the real-time values ​​of the associated data points from the corresponding group of network switches, and stores the received real-time values ​​of the associated data points in the database of the federal database node. The present application can transmit the real-time values ​​of the associated data points collected by the cluster collection device to the same federal database node through the same group of network switches for storage, thereby making the associated data points affected by the network delay and node performance to the same extent, so that there will be no real-time consistency problems between the associated data points. At the same time, since the federal database node can be autonomous, its online and offline will only affect the autonomous business, and will not affect the business of other federal database nodes, so that if a federal database node is abnormal, only the federal database node with the abnormality can be maintained, reducing operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0031] Figure 1 A schematic diagram of a flow chart of a data management method for a cluster collection device in a distributed management system provided by the present application;

[0032] Figure 2 A flowchart of a data management method for a federated service node in a distributed management system provided by the present application;

[0033] Figure 3 A schematic diagram of the structure of a distributed management system provided for this application;

[0034] Figure 4 A schematic diagram of the structure of another distributed management system provided for this application;

[0035] Figure 5 A schematic diagram of the structure of a data management device for a cluster collection device applied to a distributed management system provided by the present application;

[0036] Figure 6 A schematic diagram of the structure of a data management device for a federated service node in a distributed management system provided by the present application;

[0037] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0039] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0040] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0041] In order to enable those skilled in the art to better understand the data management method and related devices provided by the present application, the relevant terms used in the present application are explained as follows.

[0042] Cluster collection: Cluster collection is a method of distributed data collection in a clustered manner. It is an energy-saving and efficient data collection technology suitable for large-scale, distributed sensor networks. In cluster collection, nodes are divided into several clusters, each of which consists of a cluster head node and multiple slave nodes. The cluster head node is mainly responsible for aggregating the data collected by the slave nodes and forwarding it to other nodes or external devices. The slave nodes only need to transmit the collected data to the cluster head node, which reduces the communication volume and energy consumption between nodes, and also improves the efficiency of data collection and the reliability of data transmission.

[0043] Federal database: A federal database combines multiple independent and autonomous databases to form a large, unified database network that spans different institutions and regions.

[0044] Existing distributed database management technologies mainly include the following:

[0045] (1) Distributed File System (DFS): This method divides large files into small blocks and stores them among multiple nodes. Each node only stores the blocks it owns. For example, Hadoop's HDFS (Hadoop Distributed File System) and Google's GFS (Google File System) are typical representatives of distributed file systems.

[0046] (2) Distributed Database System (DDS): This method distributes data to multiple nodes for storage and provides cross-node operations such as query, insert, and delete. For example, MySQLCluster, Apache Cassandra, and Amazon DynamoDB are all typical representatives of distributed database systems.

[0047] (3) Distributed Object Storage System (DOSS): This method stores data as objects and stores and schedules them among multiple nodes to achieve data reliability and high performance. For example, Apache Kafka and RabbitMQ are typical representatives of distributed object storage systems.

[0048] (4) Distributed Cache System (DCS): This method stores data in cache to improve access speed and responsiveness. For example, Memcached and Redis are typical representatives of distributed cache systems.

[0049] The above-mentioned distributed database management technologies can all realize distributed storage of massive IoT data. However, in terms of data storage methods, the above-mentioned systems all use consistent hashing algorithms to evenly divide the data of IoT devices into various distributed nodes for storage. As a result, when some data with logical correlation is divided into different distributed nodes, it will be affected by network delays and node performance, resulting in real-time consistency problems.

[0050] In addition, the distributed database management technology that uses the consistent hashing algorithm still has the problem of high operating costs.

[0051] In order to solve the problems existing in the above-mentioned prior art, the inventor of this case conducted in-depth research and finally proposed a data management method based on a distributed management system, which is introduced in detail below.

[0052] The present application provides a data management method, which can be applied to a cluster collection device in a distributed management system. The data management method applied to a cluster collection device in a distributed management system according to an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0053] Reference Figure 1 , Figure 1 A flow chart of a data management method for a cluster collection device in a distributed management system provided in an embodiment of the present application, the method may include:

[0054] Step S101: Collect real-time values ​​of multiple data points.

[0055] The multiple data points each have a globally unique identifier, the globally unique identifier includes a correlation factor identifier, and the correlation factor identifiers of the associated data points with logical correlation among the multiple data points are the same.

[0056] The above data points can also be called signal points, which refer to the signals that can be collected by various devices in the Internet of Things.

[0057] In the field of IoT big data federation governance, the identifiers of data points themselves may have the same name. For example, multiple IoT devices can collect temperature signals, so the identifiers of the temperature signals of multiple IoT devices can be the same. Therefore, it is impossible to distinguish all data points in the field of IoT big data federation governance only by relying on the identifiers of the data points themselves.

[0058] In order to be able to distinguish all the above data points, the embodiment of the present application can pre-define a globally unique identifier for each data point so that the data point can be uniquely distinguished by the globally unique identifier.

[0059] Optionally, a globally unique identifier is defined using the Uniform Resource Identifier (URI) standard specification.

[0060] Optionally, the globally unique identifier may be composed of a correlation factor identifier and an identifier of the data point itself, where the correlation factor identifier refers to an identifier of a correlation factor of the data point.

[0061] The above correlation factors are used to define the space of data points. Optionally, the correlation factors include one or more of the following factors: the source of the data points, the collection protocol of the data points and the scope of the data points.

[0062] Of course, the correlation factor can also be other, which is not specifically limited in this application. For example, this embodiment can also customize logical grouping according to the abstract relationship of the business, and describe the logical correlation between data points through customized logical grouping. Then the correlation factor can also include customized logical grouping, for example, the customized logical grouping can be geographic location.

[0063] In the embodiment of the present application, data points with logical correlation can be distinguished by correlation factor identification (for the convenience of subsequent description, data points with logical correlation are defined as associated data points). For example, by identifying the source of the data point, data points from the same source (for example, from the same IoT device) are used as associated data points; for another example, by identifying the scope of action of the data point, data points with the same scope of action (for example, acting on the same IoT device) are used as associated data points, etc. That is, in the present embodiment, the correlation factor identification of associated data points with logical correlation is the same.

[0064] It should be noted that the above-mentioned globally unique identifier consisting of the correlation factor identifier and the identifier of the data point itself is only an example. In a specific implementation, other identifiers may be added on the basis of the correlation factor identifier included in the globally unique identifier, and this application does not make specific limitations.

[0065] Optionally, in addition to defining a globally unique identifier for the above data points, the value type of the data point can also be recorded. Optionally, the value type includes: Boolean type (also known as switch type), numeric type and text type; Optionally, the value type includes: integer type and floating point type.

[0066] For example, in some business scenarios, the above-mentioned customized logical grouping can also be a numerical type, and so on.

[0067] After globally unique identifiers are defined for all data points in the field of federated governance of big data in the Internet of Things, this embodiment can collect real-time values ​​of multiple data points through a cluster collection device in a distributed management system. The multiple data points here may refer to all data points in the field of federated governance of big data in the Internet of Things, or the multiple data points may refer to some data points in the field of federated governance of big data in the Internet of Things.

[0068] Optionally, the real-time value includes one or more of the following values: a real-time value, a timestamp, and a quality code. Of course, the real-time value may also include other values, which are not specifically limited in this application.

[0069] Step S102: aggregate the real-time values ​​of the associated data points in the multiple data points to the same group of network switches, so that the federal database nodes in the distributed management system receive the real-time values ​​of the associated data points from the corresponding group of network switches, and store the received real-time values ​​of the associated data points in the database of the federal database nodes.

[0070] As described above, the related data points among the multiple data points can be distinguished by the correlation factor identification, and then the clustering collection device can aggregate the real-time values ​​of the related data points among the multiple data points to the same group of network switches.

[0071] For example, multiple data points include data point 1 to data point 10. After being distinguished by correlation factor identifiers, data point 1 to data point 3 are a group of related data points, data point 4 to data point 8 are a group of related data points, and data point 9 and data point 10 are each a group of related data points. The clustering collection device can aggregate the real-time values ​​of data point 1 to data point 3 to the first group of network switches, aggregate the real-time values ​​of data point 4 to data point 8 to the second group of network switches, aggregate the real-time value of data point 9 to the third group of network switches, and aggregate the real-time value of data point 10 to the fourth group of network switches.

[0072] Of course, the above example is only an optional method. In addition to this, there may be other aggregation methods. For example, the real-time value of data point 9 and the real-time value of data point 10 may also be aggregated to the same group of network switches. This application does not make specific limitations on this.

[0073] Optionally, the process of "aggregating the real-time values ​​of associated data points among multiple data points to the same group of network switches" may include: using a wired network communication method to aggregate the real-time values ​​of associated data points to the same group of network switches, and / or using a wireless network communication method to aggregate the real-time values ​​of associated data points to the same group of network switches.

[0074] Optionally, the same group of network switches includes at least one network switch. In the case where the same group of network switches includes multiple network switches, optionally, the multiple network switches may adopt a stacked structure.

[0075] The above-mentioned cascading structure refers to connecting multiple network switches together through stacking cables (or other connection methods) to form a logical switch, and ensuring that when communicating within this logical switch, higher transmission speeds and better network management functions can be obtained. Cascading technology can improve network capacity, reliability, flexibility and scalability.

[0076] In this embodiment, the federated database modeling may be performed in advance. Specifically, the number of federated database nodes and the identification of the correlation factors to be managed by each federated database node may be determined in advance.

[0077] Optionally, the number of federated database nodes may be determined based on the scale of data points and the load capacity of a single node. For example, the number of federated database nodes = the total number of data points / the number of data points loaded by a single node.

[0078] Preferably, in addition to referring to the data point scale and single-node load capacity, correlation factors can also be referred to, and data points in different business areas can be processed separately to isolate the impact, for example, data points of the refining business and laboratory analysis business of the refinery can be processed separately.

[0079] In this embodiment, after the cluster collection device aggregates the real-time values ​​of associated data points among multiple data points to the same group of network switches, each federal database node can receive the real-time value of the associated data point from the corresponding group of network switches, and store the received real-time value of the associated data point in the database of the federal database node.

[0080] Preferably, the same federated database node can be used as both a federated node of the real-time database and a federated node of the historical database, that is, the database of each federated database node includes a real-time database and a historical database. Then the above process of "storing the real-time value of the associated data point into the database of the federated database node" includes: storing the real-time value of the associated data point into the real-time database, comparing it with the data stored in the real-time database during the storage process, obtaining the changed data, and storing the changed data into the historical database.

[0081] The above-mentioned "storing the real-time values ​​of the associated data points into the real-time database" adopts an overwrite writing method, that is, the real-time database writes the real-time values ​​of the associated data points to overwrite the old data. When overwriting, it is easy to compare whether the new and old data are different. If different, it is changed data, then the real-time database can output the changed data, and then store the changed data into the historical database.

[0082] Here, the real-time database may be an in-memory database, which stores only the latest data of a data point; the historical database may be a time series database, which is used to store data of a data point at each historical moment.

[0083] The reason why this embodiment distinguishes between real-time databases and historical databases is that: first, all data points require real-time data; the business layer needs to use real-time data to display the current status of each IoT device. Real-time data emphasizes real-time performance, so a memory database is used for storage; second, only key data points require historical data. The business layer needs to view the historical data of key data points to analyze problems and assist in decision-making. Historical data emphasizes time series, so a time series database is used for storage.

[0084] The data management method provided by the embodiment of the present application can collect the real-time values ​​of multiple data points through the cluster collection device in the distributed management system, and aggregate the real-time values ​​of the related data points in the multiple data points to the same group of network switches, so that the federal database node in the distributed management system receives the real-time values ​​of the related data points from the corresponding group of network switches, and stores the received real-time values ​​of the related data points in the database of the federal database node. The present application can transmit the real-time values ​​of the related data points collected by the cluster collection device to the same federal database node through the same group of network switches for storage, thereby making the related data points affected by the network delay and node performance to the same extent, so that there will be no real-time consistency problems between the related data points. At the same time, since the federal database node can be autonomous, its online and offline will only affect the autonomous business, and will not affect the business of other federal database nodes, so that if a federal database node is abnormal, only the federal database node with the abnormality can be maintained, reducing operating costs.

[0085] The present application also provides a data management method for a federated service node in a distributed management system. Here, the federated service node is used to interact with a federated database node to access a data point to be accessed in the federated database node. In order to enable those skilled in the art to better understand the access process, the following embodiments are used to introduce it.

[0086] See also Figure 2 , which is a flow chart of a data management method for a federated service node in a distributed management system provided by the present application. The method may include:

[0087] Step S201: Obtain a globally unique identifier of a data point to be accessed.

[0088] As introduced in the previous embodiment, each data point has a globally unique identifier. Therefore, when the federated service node receives an access request for a data point to be accessed, it can obtain the globally unique identifier of the data point to be accessed.

[0089] Step S202: extracting a correlation factor identifier from the global unique identifier of the data point to be accessed as a target correlation factor identifier.

[0090] As described above, the global unique identifier includes a correlation factor identifier. Therefore, this embodiment can extract the correlation factor identifier from the global unique identifier of the data point to be accessed. For the convenience of subsequent introduction, the correlation factor identifier extracted in this step is defined as the target correlation factor identifier.

[0091] Step S203: query the federal database node corresponding to the target correlation factor identifier from the pre-configured correlation factor identifier directory, so as to process the to-be-accessed data point based on the queried federal database node.

[0092] In this embodiment, the distributed management system includes a federated service cluster, which is composed of at least one federated service node. In a specific implementation, each federated service node in the federated service cluster is pre-configured with a correlation factor identification directory, which is an index relationship between correlation factor identifications and federated database nodes. Therefore, after the target correlation factor identification is obtained in the present text, the federated database node corresponding to the target correlation factor identification can be queried from the pre-configured correlation factor identification directory, and the queried federated database node is the federated node storing the data point to be accessed.

[0093] In this embodiment, the queried federated database node is used to receive the real-time value of the associated data point collected by the cluster collection device from the corresponding group network switch and store it in its own database. The associated data point refers to a data point with logical correlation and the same correlation factor identifier among multiple data points collected by the cluster collection device. The specific process of storing data in the target federated database node can be referred to the above description and will not be repeated here.

[0094] After the federated database node corresponding to the target correlation factor identifier is found, this embodiment may process the data point to be accessed based on the found federated database node.

[0095] Here, the specific processing process is related to the access request received by the federal service node. For example, if the access request is a read-write request, the read-write operation can be performed on the access data point in the federal database node; if the access request is a query request, the historical data can be queried and the results can be merged on the access data point in the federal database node.

[0096] In summary, in the data management method provided by the embodiment of the present application, when the cluster collection device transmits the real-time values ​​of the collected multiple data points to the federated database node through the network switch for storage, the federated service node can respond to the user's access request and access the data stored in the federated database node. Since different federated database nodes store data of different associated data points, in order to accurately find the federated database node storing the data of the data point to be accessed, the embodiment pre-configures a correlation factor identification directory in each federated service node, and the required federated database node can be indexed through the correlation factor identification directory, and then the access operation can be performed.

[0097] This embodiment adopts a federal autonomy approach, which can prevent an abnormality of a federal database node and / or a federal service node from affecting other federal nodes. When an abnormality occurs, only the abnormal federal database node and / or federal service node needs to be maintained, thereby reducing operating costs.

[0098] The above method of storing the real-time values ​​of associated data points in the same federal database node enables, when the federal service node accesses the data points to be accessed in the federal database node, if the data points to be accessed include associated data points with logical correlation, the data of the associated data points accessed by the federal service node have real-time consistency, and the user experience is better.

[0099] See also Figure 3 , is a structural diagram of a distributed management system provided by this application, such as Figure 3 The distributed management system includes a cluster collection layer 300, a federated data middle platform 400 and a federated service layer 500.

[0100] The cluster collection layer 300 includes at least one cluster collection device, each of which is used to collect real-time values ​​of multiple data points and aggregate the real-time values ​​of related data points among the multiple data points to the same group of network switches.

[0101] The federated data center 400 includes at least one federated database node, each of which receives real-time values ​​of associated data points from a corresponding group of network switches and stores the received real-time values ​​of the associated data points in a database of the federated database node.

[0102] The federated service layer 500 includes at least one federated service node. Each federated service node obtains a globally unique identifier of a data point to be accessed, extracts a correlation factor identifier from the globally unique identifier of the data point to be accessed as a target correlation factor identifier, queries a federal database node corresponding to the target correlation factor identifier from a preconfigured correlation factor identifier directory, and processes the data point to be accessed based on the queried federal database node.

[0103] See also Figure 4 , is a structural diagram of another distributed management system provided in an embodiment of the present application.

[0104] The cluster collection layer 300 includes a plurality of cluster collection devices 310 and a plurality of network switches 320. Optionally, the cluster collection devices 310 may be IoT instruments and / or other IoT devices capable of collecting data.

[0105] The above-mentioned federated data middle platform 400 includes multiple federated database nodes 410, each federated database node 410 corresponds to multiple redundant memory databases 411 and multiple redundant time series databases 412, wherein the memory database 411 can be used as a real-time database to store the real-time values ​​of data points collected by the cluster collection device 310 in the memory and identify whether there is any changed data; the time series database 412 can be used as a historical database to persistently store changed data.

[0106] The above-mentioned federal service layer 500 includes multiple federal service nodes 510 and multiple load balancers 520, wherein the federal service node 510 provides services for reading and writing real-time data and querying historical data, and the load balancer 520 provides request distribution and load balancing capabilities.

[0107] It should be noted that Figure 4 The numbers of the cluster collection devices 310, network switches 320, federated database nodes 410, and federated service nodes 510 shown above are only examples. In actual applications, they can be horizontally expanded according to actual scenarios, and this application does not limit them.

[0108] Based on the above structure, the overall process of data management in this application can include the following:

[0109] Step 1: Modeling based on correlation factors, including data point modeling and federated database modeling.

[0110] (1) The data point modeling process may include: modeling a globally unique identifier and a numerical type of the data point, wherein the globally unique identifier is composed of a correlation factor identifier and a data point identifier.

[0111] Preferably, the globally unique identifier of the data point is defined using the URI standard specification.

[0112] Preferably, the source of the data point is used as the relevance factor.

[0113] Optionally, the acquisition protocol of the data points is used as a correlation factor.

[0114] Optionally, use the range of the data point as a relevance factor.

[0115] Optionally, use custom logical groupings as relevancy factors.

[0116] Optionally, multiple correlation factors mentioned above are used in the globally unique identifier, for example, the scope of the data point and the collection protocol are used as correlation factors when constituting the globally unique identifier of the data point.

[0117] (2) The federated database modeling process may include: determining the number of federated database nodes 410 and the identification of correlation factors to be managed by each federated database node 410 .

[0118] Preferably, the same federated database node can be used as both a federated node for the real-time database and a federated node for the historical database, so that data can be stored nearby, reducing operating costs and communication overhead.

[0119] Step 2: Collecting data in the cluster collection layer 300 , the specific process may include: the cluster collection device 310 collects real-time values ​​of multiple data points, identifies related data points among the multiple data points according to correlation factors, and aggregates the real-time values ​​of the related data points to the same group of network switches 320 .

[0120] Optionally, wired and / or wireless network communication methods may be used to converge into the same group of network switches 320 .

[0121] Optionally, network switches can be cascaded.

[0122] Step 3: Data is stored in the federated data center 400. The specific process may include: each federated database node 410 receives the real-time value of the associated data point from a corresponding group of network switches 320, writes the received real-time value of the associated data point into its own real-time database (i.e., the memory database 411), and writes the changed data compared when writing into the real-time database into the historical database (i.e., the time series database 412).

[0123] Step 4: Accessing data within the federal service layer 500. The specific process may include: when there is an access demand, an access request for accessing the data point to be accessed may be generated. The access request is processed by the load balancer 520 and distributed to a certain federal service node 510. The federal service node 510 then parses out the globally unique identifier of the data point to be accessed in the access request, extracts the correlation factor identifier of the data point to be accessed, and then queries from its own pre-configured correlation factor identifier directory to obtain the federal database node 410 corresponding to the correlation factor identifier of the data point to be accessed, and then executes read and write data and / or queries historical data on the data point to be accessed based on the queried federal database node 410, and merges the results.

[0124] In summary, in the above implementation process, the related data points among the multiple data points are divided into the same federated database node for storage, so that the data of these related data points can be accessed only in the same federated database node, thereby ensuring real-time consistency.

[0125] Furthermore, each federated database node can be autonomous, that is, each federated database node can manage and control its own data without being affected by other nodes, and each federated database node can query data from other nodes and can independently decide whether to share its own data with other nodes. Due to autonomy, the online and offline of each federated database node only affects the autonomous business and will not affect other businesses. This divide-and-conquer approach allows the application to only maintain the local nodes where the problem occurs, reducing the scope of impact and thus reducing operating costs.

[0126] A data management method for a clustered data collection device in a distributed management system provided by an embodiment of the present application is introduced above. A device for executing the data management method for a clustered data collection device in a distributed management system will be introduced below.

[0127] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a data management device for a cluster collection device in a distributed management system provided by an embodiment of the present application. Figure 5 As shown, the device may include:

[0128] The data collection unit 601 is used to collect real-time values ​​of multiple data points, wherein each of the multiple data points has a globally unique identifier, the globally unique identifier includes a correlation factor identifier, and the correlation factor identifiers of the associated data points with logical correlation among the multiple data points are the same;

[0129] The data storage unit 602 is used to aggregate the real-time values ​​of associated data points among multiple data points to the same group of network switches, so that the federal database nodes in the distributed management system receive the real-time values ​​of the associated data points from the corresponding group of network switches and store the received real-time values ​​of the associated data points in the database of the federal database nodes.

[0130] In a possible implementation, the above correlation factor identification refers to the identification of the correlation factor, and the correlation factor includes one or more of the following factors: the source of the data point, the collection protocol of the data point, and the scope of the data point.

[0131] In a possible implementation, the above-mentioned globally unique identifier is composed of a correlation factor identifier and a data point identifier, and the globally unique identifier is defined using the uniform resource identifier standard specification.

[0132] In a possible implementation, the process in which the data storage unit aggregates the real-time values ​​of associated data points among multiple data points to the same group of network switches may specifically include: using a wired network communication method to aggregate the real-time values ​​of associated data points to the same group of network switches; and / or using a wireless network communication method to aggregate the real-time values ​​of associated data points to the same group of network switches.

[0133] In a possible implementation, the same group of network switches includes multiple network switches, and the multiple network switches are in a stacked structure.

[0134] The data management device for clustered collection devices in a distributed management system provided by the present application corresponds to the data management method for clustered collection devices in a distributed management system described above. For details, please refer to the above description and will not be repeated here.

[0135] The above also introduces a data management method for a federated service node in a distributed management system provided by an embodiment of the present application. The following will introduce a device for executing the above-mentioned data management method for a federated service node in a distributed management system.

[0136] See also Figure 6 , Figure 6 The present invention provides a structural diagram of a data management device for a federated service node in a distributed management system. Figure 6 As shown, the device may include:

[0137] An identification acquisition unit 701 is used to acquire a globally unique identification of a data point to be accessed;

[0138] An identifier extraction unit 702 is used to extract a correlation factor identifier from the global unique identifier of the data point to be accessed as a target correlation factor identifier;

[0139] The data processing unit 703 is used to query the federal database node corresponding to the target correlation factor identifier from the pre-configured correlation factor identifier directory, so as to process the data point to be accessed based on the queried federal database node, wherein the queried federal database node is used to receive the real-time value of the associated data point collected by the cluster collection device from the corresponding group network switch and store it in its own database, and the associated data point refers to a data point with logical correlation and the same correlation factor identifier among multiple data points collected by the cluster collection device.

[0140] The data management device for a federated service node in a distributed management system provided in the present application corresponds to the data management method for a federated service node in a distributed management system described above. For details, please refer to the above introduction and will not be repeated here.

[0141] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0142] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0143] The programs in the above ROM802 and RAM803 are the relevant programs of the data management method for the cluster collection device applied in the distributed management system and / or the relevant programs of the data management method for the federated service node applied in the distributed management system provided by the present application.

[0144] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a memory card, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0145] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any data management method for a cluster collection device applied in a distributed management system provided in an embodiment of the present application, or implements any data management method for a federated service node applied in a distributed management system provided in an embodiment of the present application.

[0146] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any data management method for a cluster collection device applied in a distributed management system provided in an embodiment of the present application, or implement any data management method for a federated service node applied in a distributed management system provided in an embodiment of the present application.

[0147] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0148] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0150] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data management method, characterized in that: A cluster collection device applied in a distributed management system, characterized by comprising: Collecting real-time values ​​of a plurality of data points, wherein each of the plurality of data points has a globally unique identifier, the globally unique identifier includes a correlation factor identifier, and the correlation factor identifiers of associated data points having logical correlation among the plurality of data points are the same; The real-time values ​​of the associated data points among the multiple data points are aggregated to the same group of network switches, so that the federal database nodes in the distributed management system receive the real-time values ​​of the associated data points from the corresponding group of network switches, and store the received real-time values ​​of the associated data points in the database of the federal database nodes.

2. The data management method according to claim 1, characterized in that: The correlation factor identifier refers to an identifier of a correlation factor, and the correlation factor includes one or more of the following factors: a source of a data point, a collection protocol of a data point, and a scope of action of a data point.

3. The data management method according to claim 2, characterized in that: The globally unique identifier is composed of a correlation factor identifier and a data point identifier, and the globally unique identifier is defined using the uniform resource identifier standard specification.

4. The data management method according to claim 1, characterized in that: The step of aggregating the real-time values ​​of the associated data points among the plurality of data points to the same group of network switches includes: A wired network communication method is used to aggregate the real-time values ​​of the associated data points to the same group of network switches; And / or, the real-time values ​​of the associated data points are aggregated to the same group of network switches using a wireless network communication method.

5. The data management method according to claim 1, characterized in that: The same group of network switches includes multiple network switches, and the multiple network switches are in a stacked structure.

6. A data management method, characterized in that: The federated service nodes used in the distributed management system include: Get the globally unique identifier of the data point to be accessed; Extracting a correlation factor identifier from the global unique identifier of the data point to be accessed as a target correlation factor identifier; The federal database node corresponding to the target correlation factor identifier is queried from a preconfigured correlation factor identifier directory, so as to process the data point to be accessed based on the queried federal database node, wherein the queried federal database node is used to receive the real-time value of the associated data point collected by the cluster collection device from the corresponding group network switch and store it in its own database, and the associated data point refers to a data point with logical correlation and the same correlation factor identifier among multiple data points collected by the cluster collection device.

7. A data management device, characterized in that: A cluster collection device applied in a distributed management system, characterized by comprising: A data collection unit, used for collecting real-time values ​​of a plurality of data points, wherein each of the plurality of data points has a globally unique identifier, the globally unique identifier includes a correlation factor identifier, and the correlation factor identifiers of associated data points having logical correlation among the plurality of data points are the same; A data storage unit is used to aggregate the real-time values ​​of the associated data points among the multiple data points to the same group of network switches, so that the federal database nodes in the distributed management system receive the real-time values ​​of the associated data points from the corresponding group of network switches and store the received real-time values ​​of the associated data points in the database of the federal database nodes.

8. A data management device, characterized in that: The federated service nodes used in the distributed management system include: An identification acquisition unit, used to acquire a globally unique identification of a data point to be accessed; An identifier extraction unit, used to extract a correlation factor identifier from the global unique identifier of the data point to be accessed as a target correlation factor identifier; A data processing unit, used for querying the federal database node corresponding to the target correlation factor identifier from a preconfigured correlation factor identifier directory, so as to process the data point to be accessed based on the queried federal database node, wherein the queried federal database node is used to receive the real-time value of the associated data point collected by the cluster collection device from the corresponding group network switch and store it in its own database, and the associated data point refers to a data point with logical correlation and the same correlation factor identifier among multiple data points collected by the cluster collection device.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data management method according to any one of claims 1 to 5, or the data management method according to claim 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the data management method as described in any one of claims 1 to 5, or the data management method as described in claim 6.

Images