Data storage system and method for multiple data nodes

Abstract

The application relates to the technical field of distributed storage, and discloses a data storage system and method for multiple data nodes. According to the application, a target node allocates a storage node to corresponding original data according to an original label, and meanwhile, a data structure body composed of a data storage queue is constructed in the data node based on the original label, so that each dimension data corresponding to the same target data is respectively stored in a corresponding data storage queue with the same queue serial number. On the one hand, the data of different data categories is distributedly stored, and the storage computing power is balanced. On the other hand, each dimension data corresponding to the same target data is respectively stored in a corresponding data storage queue with the same queue serial number, so that each dimension data in the same target data is one-to-one corresponding. The data caching mode based on the key-value pair is changed, the data reading efficiency caused by the expansion of the number of value keys is avoided, and the data utilization efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a data storage system and method for multiple data nodes. Background Technology

[0002] In process industry production lines, real-time data generated from monitoring the operating status of production equipment and collecting process parameters is characterized by high throughput, high timeliness, and strong time sequence. Traditional technical solutions mainly rely on relational databases for time-series data storage. However, in high-concurrency query scenarios, frequent system interactions and database I / O operations can easily lead to performance bottlenecks, resulting in insufficient real-time performance of the control model and a degraded user experience. Existing industrial scenarios are attempting to address these issues by using caching technology to improve data access performance.

[0003] However, since caching technology typically uses key-value pair storage, if fine-grained key-value pairs are built based on timestamps, the number of numerical keys will expand exponentially, leading to performance degradation. If a block storage strategy with fixed time periods is adopted, although the storage order can be maintained, the strong coupling of time intervals makes it impossible to flexibly support fast data retrieval in dynamic ranges. Therefore, key-value pair data caching results in low data reading efficiency and cannot meet the storage needs of industrial time-series data. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0005] In view of the shortcomings of the prior art described above, this application provides a data storage system and method for multiple data nodes to improve data utilization efficiency by changing the data caching method.

[0006] This application provides a data storage system for multiple data nodes, where any data node is used as a target node. The target node includes: an acquisition module, used to acquire raw data and raw tags corresponding to the raw data, and to match the raw tags from a preset node matching table to determine the storage node corresponding to the raw data from each of the data nodes, and to use the raw data as the data to be stored corresponding to the storage node, wherein the node matching table stores the correspondence between data nodes and node tags; a configuration module, used to acquire a data structure corresponding to the target tag, the data structure including multiple data storage queues corresponding to multiple data dimensions, wherein the data to be stored corresponding to the target node is used as the target data, and the raw tags corresponding to the target data are used as the target tags; and a processing module, used to extract data from the target data according to each data dimension corresponding to the target tag to obtain multiple dimension data, and to store the data of each dimension corresponding to the same target data in the corresponding data storage queues with the same queue number.

[0007] In one embodiment of this application, the acquisition module is further configured to: if the storage node includes a target node, use the data to be stored corresponding to the target node as target data and the original tag corresponding to the target data as target tag; if the storage node does not include a target node, send the original data and the original tag to the storage node.

[0008] In one embodiment of this application, the configuration module obtains the data structure corresponding to the target tag in the following ways: if the target node does not have a data structure corresponding to the storage tag, then the target tag is used as a new storage tag, and a data structure corresponding to the target tag is generated; if the target node has a data structure corresponding to the storage tag, and the storage tag is the same as the target tag, then the data structure corresponding to the storage tag is used as the data structure corresponding to the target tag; if the target node has a data structure corresponding to the storage tag, and the storage tag is different from the target tag, then the target tag is used as a new storage tag, and a data structure corresponding to the target tag is generated.

[0009] In one embodiment of this application, the data structure further includes at least one of the following: extreme values ​​of acquisition time corresponding to the acquisition time queue, wherein the extreme values ​​of acquisition time include a maximum acquisition time and / or a minimum acquisition time, the acquisition time queue belongs to the data storage queue, and the acquisition time queue is used to store data acquisition time; data storage period, wherein the data storage period includes at least one of period base duration, period start time, and period end time; valid data interval corresponding to the data value queue, wherein the data value queue belongs to the data storage queue, and the data value queue is used to store data values; and global data variables, wherein the global data variables include one or more of queue length, queue sequence number extreme value, data quantity, and statistical data, wherein the queue sequence number extreme value includes a maximum queue sequence number and / or a minimum queue sequence number, and the statistical data is obtained by calculating the data values ​​in the data value queue.

[0010] In one embodiment of this application, the configuration module is further configured to: if the data acquisition time corresponding to the target data is greater than the cycle end time, calculate the cycle end time according to the cycle reference duration to update the cycle end time; and update the data global variable in response to the data structure storing new target data.

[0011] In one embodiment of this application, the processing module is further configured to: delete the target data if the data acquisition time corresponding to the target data is less than or equal to the maximum acquisition time; delete the target data if the data value corresponding to the target data is outside the valid data range.

[0012] In one embodiment of this application, the target node further includes at least one of the following: a caching module, used to cache the data structure corresponding to the target tag using a preset caching technology; and a storage module, used to merge the data structure corresponding to the target tag to obtain merged data if the data structure corresponding to the target tag meets a preset storage trigger condition, and to persist the merged data, wherein the storage trigger condition includes the data volume corresponding to the data structure being greater than or equal to a preset data volume threshold, and / or the data acquisition duration of the data structure being greater than or equal to a preset acquisition duration threshold.

[0013] In one embodiment of this application, the acquisition module acquires the node configuration table in the following manner: determining the storage efficiency index corresponding to each data node based on the node storage space and / or node computing power, and determining the transmission efficiency index between each data node based on the data transmission rate, wherein the storage efficiency index is positively correlated with the node storage space and node computing power, and the transmission efficiency index is positively correlated with the data transmission rate; acquiring a data tag and calculating the data acquisition probability of the data tag at each data node, wherein at least some data nodes obtain tag data corresponding to the data tag through data acquisition, and the data acquisition probability is the probability that the data node acquires the tag data; taking any data node as the current node, calculating the node score corresponding to the current node based on the storage efficiency index corresponding to the current node, the transmission efficiency index between each data node and the current node, and the data acquisition probability corresponding to each data node; determining high-efficiency nodes from each data node based on the node scores corresponding to each data node, and using the data tag as the node tag corresponding to the high-efficiency node; storing the correspondence between the high-efficiency nodes and the node tags in the node configuration table.

[0014] In one embodiment of this application, the acquisition module calculates the node score corresponding to the current node in the following manner: In the formula, the current node is the m-th data node, Score(m) is the node score corresponding to the current node, N is the total number of data nodes, P(i) is the data collection probability corresponding to the i-th data node, S(m) is the storage efficiency index corresponding to the current node, and T(i,m) is the transmission efficiency index from the i-th data node to the current node.

[0015] This application provides a data storage method for multiple data nodes, which takes any data node as a target node and applies it to the target node. The method includes: obtaining raw data and raw tags corresponding to the raw data, and matching the raw tags from a preset node matching table to determine the storage node corresponding to the raw data from each of the data nodes, and using the raw data as the data to be stored corresponding to the storage node, wherein the node matching table stores the correspondence between data nodes and node tags; obtaining a data structure corresponding to the target tag, wherein the data structure includes multiple data storage queues corresponding to multiple data dimensions, wherein the data to be stored corresponding to the target node is used as the target data, and the raw tags corresponding to the target data are used as the target tags; extracting data from the target data according to each data dimension corresponding to the target tags to obtain multiple dimension data, and storing the data of each dimension corresponding to the same target data in the corresponding data storage queues with the same queue number.

[0016] This application provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described method.

[0017] The beneficial effects of this application are:

[0018] By allocating storage nodes to the corresponding raw data according to the original labels at the target nodes, and simultaneously constructing a data structure based on data storage queues within the data nodes, each dimension of the same target data is stored in its corresponding data storage queue with the same queue number. This approach achieves distributed storage of different data categories, balancing storage power, by allocating storage nodes to the corresponding raw data according to the original labels and constructing data structures based on data storage queues within the data nodes. Furthermore, storing each dimension of the same target data in its corresponding data storage queue with the same queue number ensures a one-to-one correspondence between the dimensions within the same target data. This changes the key-value pair-based data caching method, avoiding low data retrieval efficiency due to the proliferation of numerical keys, thereby improving data utilization efficiency. Attached Figure Description

[0019] Figure 1 This is a structural diagram of an application scenario in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the structure of a data storage system for multiple data nodes in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of a data storage system for multiple data nodes in an embodiment of this application;

[0022] Figure 4 This is a flowchart illustrating an industrial data processing method in an embodiment of this application;

[0023] Figure 5 This is a flowchart illustrating a data structure generation method in an embodiment of this application;

[0024] Figure 6 This is a flowchart illustrating a target data processing method in an embodiment of this application;

[0025] Figure 7 This is a flowchart illustrating a data storage method for multiple data nodes in an embodiment of this application.

[0026] Figure 8 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0027] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and sub-samples in the embodiments can be combined with each other.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0029] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0030] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0031] Unless otherwise stated, the term "multiple" means two or more.

[0032] In this application, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0033] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0034] Combination Figure 1 As shown, this application provides an application scenario for deploying a data storage system for multiple data nodes, wherein the application scenario includes multiple data nodes, and the data nodes transmit data to each other through a network.

[0035] In some embodiments, a data node consists of one or more data servers.

[0036] In some embodiments, the data storage system for multiple data nodes is suitable for high-throughput industrial scenarios, and improves data node performance and reduces database load by classifying or retrieving industrial production data through real-time and efficient time-series data caching.

[0037] Combination Figure 2 As shown, this application provides a data storage system for multiple data nodes, where any data node is taken as the target node. The target node includes an acquisition module 201, a configuration module 202, and a processing module 203.

[0038] The acquisition module 201 is used to acquire the original data and the original tags corresponding to the original data, and to match them from the preset node matching table according to the original tags in order to determine the storage node corresponding to the original data from each data node, and to use the original data as the data to be stored corresponding to the storage node. The node matching table stores the correspondence between data nodes and node tags.

[0039] The configuration module 202 is used to obtain the data structure corresponding to the target label. The data structure includes multiple data dimensions and corresponding data storage queues. The data to be stored corresponding to the target node is used as the target data, and the original label corresponding to the target data is used as the target label.

[0040] The processing module 203 is used to extract data from the target data according to the data dimensions corresponding to the target label, obtain multiple dimension data, and store the data of each dimension corresponding to the same target data in the corresponding data storage queue with the same queue number.

[0041] The data storage system for multiple data nodes provided in this application allocates storage nodes to the corresponding original data based on the original labels of the target nodes. Simultaneously, a data structure composed of data storage queues is constructed within each data node based on the original labels. Data of each dimension corresponding to the same target data is stored in its respective data storage queue using the same queue number. This achieves distributed storage of data of different categories, balancing storage computing power, by allocating storage nodes to the corresponding original data according to the original labels and constructing data structures based on data storage queues within each data node. Furthermore, storing data of each dimension corresponding to the same target data in its respective data storage queue using the same queue number ensures a one-to-one correspondence between the dimensions within the same target data. This changes the key-value pair-based data caching method, avoiding low data retrieval efficiency due to the proliferation of numerical keys, thereby improving data utilization efficiency.

[0042] Combination Figure 3As shown, this application provides a data storage system for multiple data nodes, where any data node is taken as the target node. The target node includes an acquisition module 201, a configuration module 202, a processing module 203, a cache module 204, and a storage module 205.

[0043] In some embodiments, the data storage system for multiple data nodes is implemented using one or more programming languages ​​such as Java, C#, C++, and SQL (Structured Query Language).

[0044] The acquisition module 201 is used to acquire the original data and the original tags corresponding to the original data, and to match them from the preset node matching table according to the original tags in order to determine the storage node corresponding to the original data from each data node, and to use the original data as the data to be stored corresponding to the storage node. The node matching table stores the correspondence between data nodes and node tags.

[0045] Optionally, the acquisition module 201 is further configured to: if the storage node includes the target node, use the data to be stored corresponding to the target node as the target data and the original label corresponding to the target data as the target label; if the storage node does not include the target node, send the original data and the original label to the storage node.

[0046] Combination Figure 4 As shown, this application provides an industrial data processing method, including:

[0047] Step S401: The data node uses the received real-time industrial data as raw data.

[0048] Step S402: Obtain the original labels corresponding to the original data;

[0049] Step S403: Calculate the hash value corresponding to the original label to obtain the label hash value;

[0050] Step S404: Match the hash values ​​corresponding to the tags of each node according to the tag hash values, so as to determine the storage node corresponding to the original data from each data node based on the matching results;

[0051] Step S405: Determine whether the storage node includes this data node. If yes, proceed to step S406; otherwise, proceed to step S407.

[0052] Step S406: Use the original data as the target data corresponding to this data node.

[0053] Step S407: Forward the raw data to the storage node.

[0054] Optionally, the acquisition module 201 acquires the node configuration table in the following ways: determining the storage efficiency index corresponding to each data node based on the node storage space and / or node computing power, and determining the transmission efficiency index between each data node based on the data transmission rate, wherein the storage efficiency index is positively correlated with the node storage space and node computing power, and the transmission efficiency index is positively correlated with the data transmission rate; acquiring a data tag and calculating the data acquisition probability of the data tag corresponding to each data node, wherein at least some data nodes obtain the tag data corresponding to the data tag through data acquisition, and the data acquisition probability is the probability that the data node acquires the tag data; taking any data node as the current node, calculating the node score corresponding to the current node based on the storage efficiency index corresponding to the current node, the transmission efficiency index between each data node and the current node, and the data acquisition probability corresponding to each data node; determining the high-efficiency node from each data node based on the node score corresponding to each data node, and using the data tag as the node tag corresponding to the high-efficiency node; storing the correspondence between the high-efficiency node and the node tag in the node configuration table.

[0055] In some embodiments, data nodes include Node-1, Node-2, and Node-3. A storage efficiency index for each data node is obtained by weighting the node's storage space and computing power. Nodes with higher storage efficiency indices are assigned more node scores to maximize storage efficiency. The calculation formula for the storage efficiency index is as follows: The unit of measurement for node storage space is GB (Gigabyte), and the weight of node storage space is β = 0.7. The unit of measurement for node computing power is TFLOPS (Tera Floating-point Operations Per Second), and the weight of node computing power is γ = 0.3. Data transmission rate is calculated based on a baseline transmission rate to obtain the data node's transmission efficiency index. Data nodes with higher transmission efficiency are assigned more node scores to maximize transmission efficiency. The calculation formula for the transmission efficiency index is as follows: The unit of measurement for the reference transmission rate is Mbps (Megabits per second), the reference transmission rate is 120Mbps, and the adjustment factor for the data transmission rate is α = 0.5. The storage efficiency index and transmission efficiency index of Node-1, Node-2 and Node-3 are shown in Table 1 after calculation.

[0056] Table 1

[0057] Node storage space (GB) 500 800 300 Node computing power (TFLOPS) 8 6 10 Storage efficiency metrics 0.35 0.56 0.21 Transmission efficiency metrics to Node-1 1 0.34 0.88 Transmission efficiency metrics to Node-2 0.42 1 0.23 Transmission efficiency metrics to Node-3 0.33 0.52 1

[0058] In some embodiments, different data nodes collect different data categories. The probability of data nodes collecting data for different data labels is determined based on the historical statistical results of the data nodes, thereby reducing the impact of data nodes with lower data collection probabilities on node scores.

[0059] Optionally, the acquisition module 201 calculates the node score corresponding to the current node using formula (1):

[0060]

[0061] In formula (1), the current node is the m-th data node, Score(m) is the node score corresponding to the current node, N is the total number of data nodes, P(i) is the data collection probability corresponding to the i-th data node, S(m) is the storage efficiency index corresponding to the current node, and T(i,m) is the transmission efficiency index from the i-th data node to the current node.

[0062] In some embodiments, the node score of Node-1 is calculated to be 0.31, the node score of Node-2 is 0.58, and the node score of Node-3 is 0.39. Therefore, the data label is used as the node label of Node-2. When other data nodes extract the original data corresponding to the node label, they send the original data to Node-2 for storage.

[0063] The configuration module 202 is used to obtain the data structure corresponding to the target label. The data structure includes multiple data dimensions and corresponding data storage queues. The data to be stored corresponding to the target node is used as the target data, and the original label corresponding to the target data is used as the target label.

[0064] Optionally, the configuration module 202 obtains the data structure corresponding to the target label in the following ways: if the target node does not have a data structure corresponding to the storage label, the target label is used as a new storage label, and a data structure corresponding to the target label is generated; if the target node has a data structure corresponding to the storage label, and the storage label and the target label are the same, the data structure corresponding to the storage label is used as the data structure corresponding to the target label; if the target node has a data structure corresponding to the storage label, and the storage label and the target label are different, the target label is used as a new storage label, and a data structure corresponding to the target label is generated.

[0065] Combination Figure 5 As shown, this application provides a method for generating a data structure, including:

[0066] Step S501: Obtain the storage tag and the data structure corresponding to the storage tag in the target node;

[0067] Step S502: Obtain the target label;

[0068] Step S503: Determine whether the stored tag includes the target tag. If yes, proceed to step S504; otherwise, proceed to step S506.

[0069] Step S504: Load the data configuration information corresponding to the target label;

[0070] The data configuration information includes at least one of the following: data storage period, data validity period, and global data variables;

[0071] Step S505: Use the target tag as the new storage tag, initialize the data structure corresponding to the target tag according to the data configuration information, and allocate cache space to the data structure.

[0072] Step S506: Use the data structure corresponding to the storage tag as the data structure corresponding to the target tag.

[0073] Optionally, the data structure further includes at least one of the following: extreme values ​​of acquisition time corresponding to the acquisition time queue, wherein the extreme values ​​of acquisition time include the maximum and / or minimum acquisition time values, the acquisition time queue belongs to the data storage queue, and the acquisition time queue is used to store data acquisition time; data storage period, wherein the data storage period includes at least one of period base duration, period start time, and period end time; valid data interval corresponding to the data value queue, wherein the data value queue belongs to the data storage queue, and the data value queue is used to store data values; global data variables, wherein the global data variables include one or more of queue length, queue sequence number extreme values, data quantity, and statistical data, the queue sequence number extreme values ​​include the maximum and / or minimum queue sequence number values, and the statistical data is obtained by calculating the data values ​​in the data value queue.

[0074] In some embodiments, at least a portion of the data structures are shown in Table 2.

[0075] Table 2

[0076]

[0077] Optionally, the configuration module 202 is also used to: if the data acquisition time corresponding to the target data is greater than the cycle end time, calculate the cycle end time according to the cycle reference duration to update the cycle end time.

[0078] In some embodiments, if the data acquisition time corresponding to the target data is greater than the end time of the cycle, it indicates that the data acquisition time corresponding to the target data is later than the end time of the cycle, and the end time of the cycle is incremented by the cycle base duration.

[0079] Optionally, the configuration module 202 is also used to: update the global data variable in response to the data structure storing new target data.

[0080] In some embodiments, if the data acquisition time corresponding to the target data is longer than the end time of the cycle, the extreme values ​​of the data, the statistical data, and the number of data are updated.

[0081] The processing module 203 is used to extract data from the target data according to the data dimensions corresponding to the target label, obtain multiple dimension data, and store the data of each dimension corresponding to the same target data in the corresponding data storage queue with the same queue number.

[0082] In some embodiments, the data storage queue is a circular data queue.

[0083] Optionally, the processing module 203 is further configured to: delete the target data if the data acquisition time corresponding to the target data is less than or equal to the maximum acquisition time.

[0084] In some embodiments, if the data acquisition time corresponding to the target data is less than or equal to the maximum acquisition time, it means that the data acquisition time corresponding to the target data is earlier than the latest data acquisition time in the acquisition time queue, and the target data is not the latest real-time data, so the target data is deleted.

[0085] Optionally, the processing module 203 is further configured to: delete the target data if the data value corresponding to the target data is outside the valid data range.

[0086] In some embodiments, if the data value corresponding to the target data is outside the valid data range, it indicates that the target data is abnormal data and the abnormal data is deleted.

[0087] Combination Figure 6 As shown, this application provides a target data processing method, including:

[0088] Step S601: Extract the data acquisition time and data value from the target data;

[0089] Step S602: Determine whether the data acquisition time is greater than the maximum acquisition time. If yes, proceed to step S603; otherwise, proceed to step S610.

[0090] Step S603: Determine whether the data acquisition time is greater than the cycle end time. If yes, proceed to step S604; otherwise, proceed to step S605.

[0091] Step S604: The cycle start time is incremented by the cycle reference duration, and the cycle end time is incremented by the cycle reference duration.

[0092] Step S605: Reset the extreme values, statistical data, and number of data points;

[0093] Step S606: Determine whether the data value is within the valid data range. If yes, proceed to step S607; otherwise, proceed to step S610.

[0094] Step S607: Update the extreme values ​​of the collection time, the extreme values ​​of the data, the statistical data, and the number of data.

[0095] Step S608: Calculate the maximum value of the queue sequence number to obtain the storage sequence number corresponding to the target data;

[0096] Step S609: Store the data acquisition time and data value in the corresponding data storage queues according to the storage sequence number.

[0097] Step S610: Delete the target data.

[0098] The caching module 204 is used to cache the data structure corresponding to the target tag using a preset caching technology.

[0099] In some embodiments, the preset caching technology includes one or more of Caffeine, Guava Cache, Ehcache, Memcache, Redis, etc.

[0100] In this way, multiple data are integrated into the same data structure for storage, and the correspondence of the same data is represented by the queue number. The data structure is then saved in the form of key-value pairs, thereby reducing the number of key-value pairs and ensuring data utilization efficiency.

[0101] The storage module 205 is used to merge the data structure corresponding to the target tag if the data structure corresponding to the target tag meets the preset storage trigger conditions, to obtain merged data, and to persist the merged data. The storage trigger conditions include the data volume corresponding to the data structure being greater than or equal to a preset data volume threshold, and / or the data acquisition duration of the data structure being greater than or equal to a preset acquisition duration threshold.

[0102] In some embodiments, if the queue length of the data storage queue reaches the queue length requirement, the entire data structure is stored on the hard disk.

[0103] In some embodiments, if the data acquisition duration between the start time and the end time of the cycle reaches the acquisition duration requirement, the entire data structure is stored on the hard disk to persist the data structure.

[0104] Combination Figure 7 As shown, this application provides a data storage method for multiple data nodes, which takes any data node as the target node and applies it to the target node. The method includes:

[0105] Step S701: Obtain the original data and the original labels corresponding to the original data;

[0106] Step S702: Match the original tags from the preset node matching table to determine the storage node corresponding to the original data from each data node, and use the original data as the data to be stored corresponding to the storage node.

[0107] The node matching table stores the correspondence between data nodes and node labels.

[0108] Step S703: Obtain the data structure corresponding to the target label. The data structure includes data storage queues corresponding to multiple data dimensions.

[0109] Specifically, the data to be stored corresponding to the target node is taken as the target data, and the original label corresponding to the target data is taken as the target label;

[0110] Step S704: Extract data from the target data according to the data dimensions corresponding to the target label to obtain multiple dimension data, and store the data of each dimension corresponding to the same target data in the corresponding data storage queue with the same queue number.

[0111] The data storage method for multiple data nodes provided in this application allocates storage nodes to the corresponding original data according to the original labels of the target nodes. Simultaneously, a data structure based on data storage queues is constructed within each data node based on the original labels. Data of each dimension corresponding to the same target data is stored in its respective data storage queue using the same queue number. This achieves distributed storage of data of different categories, balancing storage computing power, by allocating storage nodes to the corresponding original data according to the original labels and constructing data structures based on data storage queues within each data node. Furthermore, storing data of each dimension corresponding to the same target data in its respective data storage queue using the same queue number ensures a one-to-one correspondence between the dimensions within the same target data. This changes the key-value pair-based data caching method, avoiding low data retrieval efficiency due to the proliferation of numerical keys, thereby improving data utilization efficiency.

[0112] This application also provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described method.

[0113] Figure 8 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0114] like Figure 8 As shown, the computer system 800 includes a Central Processing Unit (CPU) 801, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on a program stored in Read-Only Memory (ROM) 802 or a program loaded from storage portion 808 into Random Access Memory (RAM) 803. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An Input / Output (I / O) interface 805 is also connected to the bus 804.

[0115] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0116] The electronic device disclosed in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic device performs the various steps of the above method.

[0117] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated subsamples, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other subsamples, wholes, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes the element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0118] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0119] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some sub-samples may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. Furthermore, the functional units in this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than those disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A data storage system for multiple data nodes, characterized in that, Any data node can be used as the target node, and the target node includes: The acquisition module is used to acquire raw data and the raw tags corresponding to the raw data, and to match them from a preset node matching table according to the raw tags in order to determine the storage node corresponding to the raw data from each data node, and to use the raw data as the data to be stored corresponding to the storage node. The node matching table stores the correspondence between data nodes and node tags. The configuration module is used to obtain the data structure corresponding to the target label. The data structure includes multiple data storage queues corresponding to multiple data dimensions. The data to be stored corresponding to the target node is used as the target data, and the original label corresponding to the target data is used as the target label. The processing module is used to extract data from the target data according to each data dimension corresponding to the target label, obtain multiple dimension data, and store the dimension data corresponding to the same target data in the corresponding data storage queue with the same queue number.

2. The system according to claim 1, characterized in that, The acquisition module is also used for: If the storage node includes a target node, the data to be stored corresponding to the target node shall be used as the target data, and the original label corresponding to the target data shall be used as the target label; If the storage node does not include the target node, the original data and the original tag are sent to the storage node.

3. The system according to claim 1, characterized in that, The configuration module obtains the data structure corresponding to the target label in the following way: If the target node does not have a data structure corresponding to the storage label, then the target label is used as a new storage label, and a data structure corresponding to the target label is generated. If the target node has a data structure corresponding to a storage label, and the storage label is the same as the target label, then the data structure corresponding to the storage label is used as the data structure corresponding to the target label; If the target node has a data structure corresponding to a storage label, and the storage label and the target label are different, then the target label is used as the new storage label, and a data structure corresponding to the target label is generated.

4. The system according to claim 1, characterized in that, The data structure also includes at least one of the following: The acquisition time extreme values ​​corresponding to the acquisition time queue, wherein the acquisition time extreme values ​​include the maximum acquisition time and / or the minimum acquisition time, the acquisition time queue belongs to the data storage queue, and the acquisition time queue is used to store data acquisition time; Data storage cycle, wherein the data storage cycle includes at least one of cycle base duration, cycle start time, and cycle end time; The valid data range corresponding to the data value queue, wherein the data value queue belongs to the data storage queue and is used to store data values; Global data variables include one or more of the following: queue length, queue index extreme value, data quantity, and statistical data. The queue index extreme value includes the maximum queue index and / or the minimum queue index. The statistical data is calculated from the data values ​​in the data value queue.

5. The system according to claim 4, characterized in that, The configuration module is also used for at least one of the following: If the data acquisition time corresponding to the target data is greater than the end time of the cycle, the end time of the cycle is calculated according to the baseline duration of the cycle to update the end time of the cycle. In response to the data structure storing new target data, the global data variable is updated.

6. The system according to claim 4, characterized in that, The processing module is also used for at least one of the following: If the data acquisition time corresponding to the target data is less than or equal to the maximum acquisition time, then the target data is deleted. If the data value corresponding to the target data is outside the valid data range, then the target data is deleted.

7. The system according to any one of claims 1 to 6, characterized in that, The target node also includes at least one of the following: The caching module is used to cache the data structure corresponding to the target tag using a preset caching technology; The storage module is used to merge the data structure corresponding to the target tag to obtain merged data if the data structure corresponding to the target tag meets the preset storage trigger conditions, and to persist the merged data. The storage trigger conditions include the data volume corresponding to the data structure being greater than or equal to a preset data volume threshold, and / or the data acquisition duration of the data structure being greater than or equal to a preset acquisition duration threshold.

8. The system according to any one of claims 1 to 6, characterized in that, The acquisition module obtains the node configuration table in the following way: The storage efficiency index corresponding to each data node is determined based on the node storage space and / or node computing power, and the transmission efficiency index between each data node is determined based on the data transmission rate. The storage efficiency index is positively correlated with the node storage space and node computing power, and the transmission efficiency index is positively correlated with the data transmission rate. A data tag is obtained, and the data collection probability of the data tag at each of the data nodes is calculated. At least some data nodes obtain the tag data corresponding to the data tag through data collection, and the data collection probability is the probability that the data node collects the tag data. Take any data node as the current node, and calculate the node score corresponding to the current node based on the storage efficiency index corresponding to the current node, the transmission efficiency index between each data node and the current node, and the data collection probability corresponding to each data node. Based on the node scores corresponding to each data node, high-efficiency nodes are determined from each data node, and the data tags are used as the node tags corresponding to the high-efficiency nodes. The correspondence between the high-efficiency nodes and the node labels is stored in the node configuration table.

9. The system according to claim 8, characterized in that, The acquisition module calculates the node score corresponding to the current node in the following way: In the formula, the current node is the m-th data node, Score(m) is the node score corresponding to the current node, N is the total number of data nodes, P(i) is the data collection probability corresponding to the i-th data node, S(m) is the storage efficiency index corresponding to the current node, and T(i,m) is the transmission efficiency index from the i-th data node to the current node.

10. A data storage method for multiple data nodes, characterized in that, The method involves using any data node as the target node and applying it to the target node. Obtain raw data and the raw tags corresponding to the raw data, and match them from a preset node matching table according to the raw tags to determine the storage node corresponding to the raw data from each data node, and use the raw data as the data to be stored corresponding to the storage node, wherein the node matching table stores the correspondence between data nodes and node tags; Obtain the data structure corresponding to the target label. The data structure includes multiple data storage queues corresponding to different data dimensions. The data to be stored corresponding to the target node is taken as the target data, and the original label corresponding to the target data is taken as the target label. Data is extracted from the target data according to the data dimensions corresponding to the target label to obtain multiple dimension data, and the dimension data corresponding to the same target data are stored in the corresponding data storage queue with the same queue number.

Images