Data storage method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose a data storage method, device and equipment, and a storage medium. The method comprises the following steps: in response to receiving data with label information sent by a client, obtaining cluster load information; based on the label information and the cluster load information, screening a target node from a preset data relationship network; wherein the data relationship network represents a mapping relationship between each node of a rack in a cluster and a data label contained by each node; and forwarding the data to the target node for storage.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of data storage technology, and in particular to a data storage method, apparatus, device, and storage medium. Background Technology

[0002] Existing distributed storage location technologies primarily store different copies of data through predetermined rack location rules, or allocate data write locations based on the load status of server nodes. Both of these methods are based on the cluster server side and do not take into account the correlation between business data, thus failing to achieve more efficient data storage and retrieval. Summary of the Invention

[0003] In view of the above, embodiments of this application provide at least one data storage method, apparatus, device, and storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a data storage method, the method comprising:

[0006] In response to receiving data with tagged information sent by the client, cluster load information is obtained; based on the tagged information and the cluster load information, target nodes are selected from a preset data relationship network; wherein, the data relationship network represents the mapping relationship between each node in the rack of the cluster and the data tags contained in each node; the data is forwarded to the target node for storage.

[0007] In some implementations, the method further includes: constructing a data relationship network using racks in the cluster, nodes of the racks, and data labels contained in each node as points, and the distance between adjacent racks as edges; wherein the length of the distance characterizes the correlation between nodes under different racks.

[0008] In some implementations, the step of selecting target nodes from a preset data relationship network based on the tag information and the cluster load information includes: querying the location information of the N racks or nodes most associated with the tag information from the data relationship network based on the tag information; wherein N is the number of replicas set in the cluster; and using the cluster load information, selecting the target nodes from the nodes corresponding to the N location information according to preset rules.

[0009] In some implementations, querying the location information of the N racks or nodes most associated with the tag information from the data relationship network based on the tag information includes: determining whether there are candidate nodes containing the tag information in the data relationship network; if the candidate nodes exist, determining the first candidate rack to which the candidate nodes belong; and if the number of nodes M included in the first candidate rack is greater than or equal to N, selecting the location information of N nodes from the M nodes as the most associated N location information.

[0010] In some implementations, the step of querying the location information of the N racks or nodes most associated with the tag information from the data relationship network based on the tag information further includes: when M is less than N, filtering the second candidate rack that is closest to the first candidate rack from the data relationship network; and taking the location information of the M nodes included in the first candidate rack and the location information of the NM nodes included in the second candidate rack as the most associated N location information.

[0011] In some implementations, the method further includes: in the absence of the candidate node, determining the similarity between the tag information and the data tags contained in each node under each rack in the cluster; determining the location information of the first node where the data tag whose similarity satisfies a first threshold is stored; and based on the location information of the first node, filtering N location information that are the same as or similar to the first node from the data relationship network as the N most related location information.

[0012] In some implementations, the step of using the cluster load information to filter the target node from the nodes corresponding to the N location information according to a preset rule includes: determining the load value of each node corresponding to the N location information based on the cluster load information; and selecting the corresponding node with the lowest load value as the target node.

[0013] Secondly, embodiments of this application provide a data storage device, including:

[0014] The information acquisition module is used to obtain cluster load information in response to data with tagged information sent by the client;

[0015] The node filtering module is used to filter target nodes from a preset data relationship network based on the tag information and the cluster load information; wherein, the data relationship network represents the mapping relationship between each node in the rack of the cluster and the data tag contained in each node;

[0016] The data storage module is used to forward the data to the target node for storage.

[0017] Thirdly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0019] In this embodiment, firstly, in response to receiving data with tagged information sent by the client, cluster load information is obtained; secondly, based on the tagged information and the cluster load information, target nodes are selected from a preset data relationship network; finally, the data is forwarded to the target node for storage. Thus, by constructing a data relationship network of data tags and node relationships in metadata management, the relationships between data and the location of storage nodes can be clearly understood. Determining the location of storage nodes based on the size of data associations can achieve a clustering effect for data queries; simultaneously, combining cluster load information to select the most suitable storage node location improves storage efficiency.

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0022] Figure 1A A diagram illustrating the selection of replica nodes for lower versions of Hadoop in related technologies;

[0023] Figure 1B A diagram illustrating the selection of replica nodes for higher versions of Hadoop in related technologies;

[0024] Figure 1C This is a schematic diagram of the TDengine distributed architecture provided in related technologies;

[0025] Figure 2 A schematic diagram of an optional data storage method provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of an optional data storage method provided in an embodiment of this application;

[0027] Figure 4 A schematic diagram of an optional data storage method provided in an embodiment of this application;

[0028] Figure 5 A schematic diagram of an optional data storage method provided in an embodiment of this application;

[0029] Figure 6 A flowchart illustrating a data storage method provided in an embodiment of this application;

[0030] Figure 7 A schematic diagram illustrating the construction of a data relationship network provided in an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the composition structure of a data storage device provided in an embodiment of this application;

[0032] Figure 9 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0037] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0038] Metadata, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0039] Hadoop is a distributed processing software framework that enables the processing of massive amounts of data and distributed computation across clusters of computers. Due to its advantages such as scalability, low cost, high efficiency, and reliability, Hadoop is widely used in the field of distributed computing.

[0040] Hadoop is currently the mainstream enterprise big data analytics platform. Hadoop uses HDFS (Hadoop Distributed File System) for data storage. HDFS adopts a master / slave architecture, with a NameNode and several DataNodes forming the HDFS cluster. HDFS employs a three-replica redundancy mechanism to ensure data security. The default replica placement strategy of HDFS is to store two data blocks on one rack and the other on a different rack, achieving a good balance between bandwidth resources and reliability.

[0041] In an HDFS cluster, to improve the availability of data blocks when storing files, the HDFS cluster adopts a multi-replica approach. First, the file is divided into a series of data blocks, and then the data blocks are stored on DataNodes.

[0042] Regarding the selection of data replica storage nodes for time-series database metadata, relevant technologies include Hadoop-based distributed architecture (including selection of replica nodes for low-version Hadoop and high-version Hadoop) and TDengine-based distributed architecture.

[0043] Figure 1A A diagram illustrating the selection of replica nodes for lower versions of Hadoop in related technologies, such as... Figure 1A As shown, the cluster where core switch 11 resides includes three racks: switch 1, switch 2, and switch 3. Each switch includes six DateNodes. The first replica 101 is located on the node where client 12 is located. If client 12 is outside the cluster, a rack is randomly selected, such as switch 1. The second replica 102 and the first replica 101 are located on random nodes in different racks, for example, respectively located on... Figure 1AThe diagram shows the first node of switch 1 and the first node of switch 3; the third replica 103 and the second replica 102 are located on the same rack, and the nodes are randomized, for example, located in... Figure 1A The first and sixth nodes of switch 3 are shown in the diagram.

[0044] Figure 1B A diagram illustrating the selection of replica nodes for higher versions of Hadoop in related technologies, such as... Figure 1B As shown, the cluster where core switch 21 resides has three racks: switch 1, switch 2, and switch 3. Each switch includes six DateNodes. The first replica 111 is located on the node where client 22 is located. If client 22 is outside the cluster, a rack is randomly selected, such as switch 1. The second replica 112 and the first replica 111 are located on the same rack, for example... Figure 1B The first and sixth nodes are randomly located on switch 1; the third replica 113 is located on random nodes on a different rack, for example... Figure 1B The sixth node on switch 2 is shown.

[0045] Figure 1C The diagram shows the distributed architecture of TDengine provided in related technologies, such as... Figure 1C As shown, a complete TDengine system runs on one or more physical nodes. Logically, the TDengine system includes data nodes, TDengine client 32 (taosc), and application 31 (app). Figure 1C The Tdengine cluster shown includes six data nodes: node 0, node 1, node 2, node 3, node 4, and node 5. Nodes 2, 3, and 4 are management nodes, responsible for monitoring and maintaining the operational status of all data nodes, as well as load balancing among them. The management nodes are also responsible for storing and managing metadata (including users, databases, tables, static tags, etc.). In other words, the TDengine solution allocates database write locations based on the node load status.

[0046] Existing distributed storage location technologies primarily store different copies of data through predetermined rack location rules, or allocate data write locations based on the load status of server nodes. Both of these approaches are based on the cluster server side.

[0047] This application provides a data storage method, which can be executed by a processor of a computer device. The computer device refers to a device with data storage capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 2 The following is a schematic diagram of an optional process for the data storage method provided in the embodiments of this application, such as... Figure 2 As shown, the method includes the following steps S210 to S230:

[0048] Step S210: In response to receiving data with tag information sent by the client, obtain cluster load information;

[0049] Here, the tag information refers to the location information tag of the data, such as the physical location of the data center or the location information of the computing device where the client is located. In the Internet of Things and device monitoring, data usually carries obvious location information tags, and the location information of most business data is correlated with each other. The embodiments of this application aim to store similar data on the same or nearby nodes based on data tag information.

[0050] The cluster load information characterizes the CPU usage status and memory usage status of each node in the cluster, including cluster resource information, cluster-allocated resources for processing job tasks, and cluster resource information required for processing job tasks. The resource information includes the number of CPU cores of the nodes in the cluster, the memory capacity of the nodes in the cluster, the number of FPGA resources of the nodes in the cluster, and the specifications and capacity of the network processing resources of the nodes in the cluster.

[0051] Step S220: Based on the tag information and the cluster load information, select target nodes from the preset data relationship network;

[0052] Here, the data relationship network represents the mapping relationship between each node in the rack of the cluster and the data tags contained in each node. This application embodiment, by constructing the data tag-node relationship in metadata management, clearly shows the relationships between data and the location of storage nodes.

[0053] In implementation, the data can be tagged with labels to perform correlation queries within the data relationship network. Combined with cluster load information, the storage location of data replicas can be determined, achieving a clustering effect in data queries and improving query efficiency and time. The relationship between data labels and nodes can be measured by the weight of the correlation between the label information. Based on the correlation between the data to be stored and existing data in the data relationship network, as well as the load information of the nodes containing the existing data with high correlation, the most suitable storage node location is determined.

[0054] Step S230: Forward the data to the target node for storage.

[0055] Here, after the target node is determined, the data is directly written to the target node to complete the data storage.

[0056] In this embodiment, firstly, in response to receiving data with tagged information sent by the client, cluster load information is obtained; secondly, based on the tagged information and the cluster load information, target nodes are selected from a preset data relationship network; finally, the data is forwarded to the target node for storage. Thus, by constructing a data relationship network of data tags and node relationships in metadata management, the relationships between data and the location of storage nodes can be clearly understood. Determining the location of storage nodes based on the size of data associations can achieve a clustering effect for data queries; simultaneously, combining cluster load information to select the most suitable storage node location improves storage efficiency.

[0057] In some embodiments, the method further includes the following step S240:

[0058] Step S240: Construct the data relationship network using racks in the cluster, nodes of the racks, and data tags contained in each node as points, and the distance between adjacent racks as edges.

[0059] Here, the length of the distance represents the correlation between nodes under different racks, thus establishing a relationship between data tags and storage nodes. The longer the distance, the weaker the relationship; the shorter the distance, the stronger the relationship.

[0060] For example, if the distance between rack 1 and rack 2 is less than the distance between rack 1 and rack 3, it indicates that the correlation between the nodes under rack 1 and the nodes under rack 2 is greater, thus storing data with similar tag information in the nodes of rack 1 and rack 2.

[0061] In a data relationship network, the relationship between data tags and nodes can be represented by the weight of the degree of association between the data tags. Data with more similar tags are more likely to exist on the same or nearby nodes. Therefore, data with greater correlation are more likely to be found together.

[0062] In the above embodiments, a data relationship network is pre-constructed using racks, nodes within racks, and data tags contained in nodes as points, and the distance between racks as edges. The distance length represents the correlation between nodes, thus establishing a relationship between data tags and storage nodes. In this way, when selecting storage nodes for subsequent new data, nodes with similar or identical tags can be directly queried from the data relationship network as target nodes.

[0063] Figure 3 The following is a schematic diagram of an optional process for the data storage method provided in the embodiments of this application, such as... Figure 3 As shown, the method includes the following steps S310 to S340:

[0064] Step S310: In response to receiving data with tag information sent by the client, obtain cluster load information;

[0065] Here, step S310 corresponds to step S210 mentioned above, and the specific implementation of step S210 can be referred to during implementation.

[0066] Step S320: Based on the tag information, query the location information of the N racks or nodes most associated with the tag information from the data relationship network;

[0067] Here, N is the number of replicas set in the cluster; for example, a natural number such as 2 or 3. The specific value depends on the settings of different distributed systems, and this application embodiment does not limit this.

[0068] If the data's tag information is already stored in the data relationship network, the query returns the location information of the node containing the corresponding data tag. If the data's tag information does not exist in the data relationship network, the query returns the location information of all racks in the cluster and the nodes under each rack.

[0069] When this application embodiment receives new data sent by the client that needs to be stored, it first compares the tag information of the new data with the data tags of the data already stored in the data relationship network, and selects the rack or node location of the data with the most similar data tags. This ensures that data with high correlation, i.e. data tags that are more similar, are stored on the same or nearby nodes, achieving a clustering effect and improving the efficiency and time of subsequent data queries.

[0070] Step S330: Using the cluster load information, the target node is selected from the nodes corresponding to the N location information according to preset rules.

[0071] In some implementations, step S330 is achieved through the following process: based on the cluster load information, the load value of each node corresponding to the N location information is determined; the node with the lowest load value is selected as the target node. This ensures that similar data is stored on the same or nearby nodes while simultaneously considering the node load status to allocate the final storage node for the data to be stored, thereby improving data storage performance.

[0072] Step S340: Forward the data to the target node for storage.

[0073] In the above embodiments, after receiving data carrying tag information sent by the client, the system first queries the established data relationship network based on the tag information to find the location of the metadata storage rack or node most associated with the tag information, and then determines the most suitable storage node by combining the cluster load information. In this way, while ensuring that similar data are stored on the same or nearby nodes, the system also considers the load status of the nodes to allocate the final storage node for the data to be stored, thereby improving data storage performance.

[0074] based on Figure 3 , Figure 4 The following is a schematic diagram of an optional process for the data storage method provided in the embodiments of this application, such as... Figure 4 As shown, step S320 above, "based on the tag information, query the location information of the N racks or nodes most associated with the tag information from the data relationship network," includes the following steps S410 to S450:

[0075] Step S410: Determine whether there are candidate nodes containing the tag information in the data relationship network;

[0076] Step S420: If the candidate node exists, determine the first candidate rack to which the candidate node belongs;

[0077] Here, if there are candidate nodes in the data relationship network that contain tag information of the data to be stored, the data tags are directly stored on other nodes under the rack of the candidate node according to their correlation. In this way, multiple data with high correlation can be queried together, improving storage and query performance.

[0078] Step S430: If the number of nodes M included in the first candidate rack is greater than or equal to N, select the location information of N nodes from the M nodes as the most associated N location information.

[0079] Here, if there are candidate nodes containing tag information of the data to be stored and the rack where the candidate node is located still has remaining storage space, the location information of N nodes on the same rack can be selected as the most relevant location information. This ensures that the metadata of N copies of the data is stored on the candidate node and its nearby nodes, achieving a clustering effect and improving query efficiency and time.

[0080] Step S440: If M is less than N, select the second candidate rack that is closest to the first candidate rack from the data relationship network.

[0081] Step S450: The location information of the M nodes included in the first candidate rack and the location information of the NM nodes included in the second candidate rack are taken as the most associated N location information.

[0082] Here, N location information are selected from the first and second candidate racks that are close to each other.

[0083] In the above embodiments, if there are candidate nodes containing tag information of the data to be stored and the remaining storage space in the rack where the candidate node is located is insufficient, the location information of other nodes can be selected from nearby racks to supplement the data, and the most relevant N location information can be determined. This ensures that the metadata of N copies of the data is stored on the candidate node and nodes in its nearby racks, achieving a clustering effect and improving query efficiency and time.

[0084] based on Figure 3 , Figure 5 The following is a schematic diagram of an optional process for the data storage method provided in the embodiments of this application, such as... Figure 5 As shown, step S320 above, "based on the tag information, query the location information of the N racks or nodes most associated with the tag information from the data relationship network," includes the following steps S510 to S540:

[0085] Step S510: Determine whether there are candidate nodes containing the tag information in the data relationship network;

[0086] Step S520: In the absence of the candidate node, determine the similarity between the label information and the data label contained in each node under each rack in the cluster;

[0087] Here, if there are no candidate nodes containing corresponding data label information in the data relationship network, the nodes containing similar data labels can be selected by comparing the data labels of each node under each rack, and the data with the new label can be stored.

[0088] Step S530: Determine the location information of the first node where the data label whose similarity satisfies the first threshold is stored.

[0089] Here, the first threshold is a preset value, which is adjusted according to the actual situation. The first node is any node in the data relationship network that stores similar data tags, and there can be one or more nodes.

[0090] Step S540: Based on the location information of the first node, select N location information that is the same as or similar to the first node from the data relationship network, and use them as the N most related location information.

[0091] Here, multiple data tags can be stored on the first node. If the storage space on the first node is insufficient, the location information of other nodes under the same rack as the first node can be selected.

[0092] In the above embodiments, when there is no tag information containing the data to be stored in the data relationship network, the location information of the first node containing data with similar data tags is determined, and then N location information related to the first node are selected to store copies of the data. In this way, the selection of the data storage node for the new tag is related to the location of the existing data tag storage nodes in the data relationship network. The more similar the data tags are, the more likely they are to be stored on the same or nearby nodes.

[0093] The above data storage method will be described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0094] Since most business data is interconnected, selecting nearby nodes to store this related data based on their relationships will improve data locality and significantly enhance query performance. Therefore, this application proposes a method for selecting data replica storage nodes based on data tags. This method primarily involves using a data relationship processing device with racks as nodes to form a relationship network between data tags and nodes based on stored data tag information. When data storage with tags occurs, a relationship degree query is performed in the data relationship processing device based on the data's tag information to determine the storage location of the data replica, achieving a clustering effect where data with stronger correlations are stored together.

[0095] Figure 6 A flowchart illustrating a data storage method provided in an embodiment of this application is shown below. Figure 6 As shown, the distributed cluster 61 includes a cluster management node 600, rack 601, rack 602, rack 603, data location selection device 604, and data relationship processing device 605 (the device where the aforementioned data relationship network is located). Rack 601 includes nodes 1-1, 1-2, and 1-3; rack 602 includes nodes 2-1, 2-2, and 2-3; and rack 603 includes nodes 3-1, 3-2, and 3-3. The data location selection device 604 and the data relationship processing device 605 are both located in the cluster management node 600. The data relationship processing device 605 can be constructed from a relational database such as a graph database. When the client 62 needs to store data, it is implemented through the following steps S1 to S6:

[0096] Step S1: Client 62 sends data with tag information to distributed cluster 62.

[0097] In step S2, the cluster management node 600 sends the data tag information and cluster load information in the data to the data location selection device 604, so that the data location selection device 604 can determine the node location of the data storage.

[0098] In step S3, the data location selection device 604 sends the data tag information to the data relationship processing device 605.

[0099] In step S4, the data relationship processing device 605 queries and returns N locations of metadata storage racks (or nodes) that are most correlated with the data tag information based on the data tag information (N is determined by the number of replicas set in the cluster).

[0100] In step S5, the data location selection device 604 calculates the node location of the data storage according to the preset rules and returns it to the cluster management node.

[0101] In some implementations, if the data tag information already exists in the data relationship processing device 605, the data location selection device 604 receives node information and returns the most suitable node location to the cluster management node 600.

[0102] In some implementations, if the data tag information is not present in the data relationship processing device 605, the received information is the rack information within the cluster. The data location selection device 604 then selects the node location with the least load in the rack based on the cluster load information and returns the node location to the cluster management node 600. Simultaneously, the metadata is stored in the data relationship processing device 605.

[0103] In step S6, after receiving the response, the cluster management node 600 forwards the data to the corresponding node for storage, thus completing the data writing process.

[0104] Figure 7 This is a schematic diagram illustrating the construction of the data relationship network provided in the embodiments of this application, such as... Figure 7 As shown, racks, server nodes, and data tags are used as nodes in a data relationship network. The distance between racks is used as an edge, and the value on the edge, i.e., the distance weight, represents the degree of relationship. The longer the distance, the weaker the relationship; the shorter the distance, the stronger the relationship. The distance weight between rack 1 and rack 2 is 30, the distance weight between rack 2 and rack 3 is 10, and the distance weight between rack 1 and rack 3 is 25. Nodes within the same rack have the same degree of association. Each server node belongs to a rack and contains multiple data tag information.

[0105] This application embodiment addresses metadata management for clusters. It uses a data relationship processing device to construct a data relationship network with racks, server nodes, and data tags as nodes, and edges representing distances between racks, server nodes belonging to racks, and server nodes containing data tags. The distance weights represent the relationship values ​​between nodes, establishing relationships between data tags and storage nodes. Furthermore, the selection of storage nodes for new tags is related to the existing storage nodes for data tags in the data relationship processing device; more similar data tags are more likely to be stored on the same or nearby nodes.

[0106] The embodiments of this application have at least the following technical effects: On the one hand, the management node uses a data relationship processing device to maintain the relationships between metadata, which clearly shows the relationships between data and the location of storage nodes. On the other hand, the characteristics of modern data queries are often related to the degree of data correlation; data with high correlation are more likely to be queried together. Determining the location of storage nodes based on the degree of data correlation can achieve a clustering effect, improving query efficiency and time.

[0107] It is worth noting that the key point of this application's embodiments lies in constructing the relationship between data tags and nodes in metadata management, and the relationship between data tags and nodes is measured by the weight of the relevance of tag data. In other embodiments, the method for constructing the data relationship network can be optimized, for example, by adding dimensions such as data center and network bandwidth information to construct the data relationship network, thereby providing richer and more accurate data retrieval.

[0108] The data storage method provided in this application pertains to a method for selecting the storage location of data files in a distributed system. This is particularly relevant in the Internet of Things (IoT) and device monitoring, where data often carries clear location information tags, and there are numerous scenarios involving related data queries. Using the data storage method provided in this application to store data copies can significantly improve query performance.

[0109] Based on the foregoing embodiments, this application provides a data storage device, which includes the included modules, as well as the sub-modules and units included in each module. It can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0110] Figure 8 This is a schematic diagram of the composition structure of a data storage device provided in an embodiment of this application, as shown below. Figure 8 As shown, the device 800 includes: an information acquisition module 810, a node filtering module 820, and a data storage module 830, wherein:

[0111] The information acquisition module 810 is used to acquire cluster load information in response to receiving data with tagged information sent by the client;

[0112] The node filtering module 820 is used to filter target nodes from a preset data relationship network based on the tag information and the cluster load information; wherein, the data relationship network represents the mapping relationship between each node in the rack of the cluster and the data tag contained in each node;

[0113] The data storage module 830 is used to forward the data to the target node for storage.

[0114] In some possible embodiments, the apparatus further includes a relationship building module for constructing the data relationship network using racks in the cluster, nodes of the racks, and data tags contained in each node as points, and the distance between adjacent racks as edges; wherein the length of the distance characterizes the degree of association between nodes under different racks.

[0115] In some possible embodiments, the node filtering module 820 includes: a query submodule, used to query the location information of the N racks or nodes most associated with the tag information from the data relationship network based on the tag information; wherein N is the number of replicas set in the cluster; and a filtering submodule, used to filter the target node from the nodes corresponding to the N location information according to a preset rule using the cluster load information.

[0116] In some possible embodiments, the query submodule includes: a first determining unit, configured to determine whether there is a candidate node containing the tag information in the data relationship network; a second determining unit, configured to determine the first candidate rack to which the candidate node belongs if the candidate node exists; and a first selecting unit, configured to select the location information of N nodes from the M nodes as the most relevant N location information if the number M of nodes included in the first candidate rack is greater than or equal to N.

[0117] In some possible embodiments, the query submodule further includes: a second selection unit, configured to filter from the data relationship network a second candidate rack that is closest to the first candidate rack when M is less than N; and a third determination unit, configured to take the location information of the M nodes included in the first candidate rack and the location information of the NM nodes included in the second candidate rack as the most relevant N location information.

[0118] In some possible embodiments, the query submodule further includes: a fourth determining unit, configured to determine the similarity between the tag information and the data tags contained in each node under each rack in the cluster when the candidate node does not exist; a fifth determining unit, configured to determine the location information of the first node where the data tag whose similarity satisfies a first threshold is stored; and a second selecting unit, configured to filter N location information that is the same as or similar to the first node from the data relationship network based on the location information of the first node, as the N most related location information.

[0119] In some possible embodiments, the filtering submodule includes: a sixth determining unit, configured to determine the load value of each node corresponding to the N location information based on the cluster load information; and a third selecting unit, configured to select the corresponding node with the lowest load value as the target node.

[0120] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0121] It should be noted that, in the embodiments of this application, if the above-described data storage method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0122] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0123] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0124] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0125] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0126] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0127] It should be noted that, Figure 9 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 9 As shown, the hardware entity of the computer device 900 includes: a processor 901, a communication interface 902, and a memory 903, wherein:

[0128] Processor 901 typically controls the overall operation of computer device 900.

[0129] Communication interface 902 enables computer devices to communicate with other terminals or servers over a network.

[0130] The memory 903 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 901 and various modules in the computer device 900. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 901, the communication interface 902, and the memory 903 can be performed via bus 904.

[0131] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0132] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0134] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0136] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0137] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0138] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data storage method, characterized by, Applied to cluster management nodes, the method includes: In response to receiving data with tagged information sent by the client, cluster load information is obtained; the tagged information is the location information tag of the data; If a candidate node containing the tag information exists in the data relationship network, the first candidate rack to which the candidate node belongs is determined; wherein, the data relationship network represents the mapping relationship between each node of the rack in the cluster and the data tag contained in each node; If the number of nodes M included in the first candidate rack is greater than or equal to N, the location information of N nodes is selected from the M nodes as the most relevant N location information; N is the number of replicas set in the cluster. In the absence of the candidate node in the data relationship network, the similarity between the label information and the data label contained in each node under each rack in the cluster is determined; The location information of the first node where the data label whose similarity satisfies the first threshold is determined; Based on the location information of the first node, N location information that is the same as or similar to the first node is selected from the data relationship network as the N most related location information; wherein, the N location information that is similar to the first node refers to the location information of other nodes under the same rack as the first node. Using the cluster load information, target nodes are selected from the nodes corresponding to the N location information according to preset rules; The data is forwarded to the target node for storage.

2. The method according to claim 1, characterized in that, The method further includes: The data relationship network is constructed using racks in the cluster, nodes of the racks, and data tags contained in each node as points, and the distance between adjacent racks as edges. The length of the distance represents the degree of correlation between nodes under different racks.

3. The method according to claim 1, characterized in that, The method further includes: When M is less than N, a second candidate rack that is closest to the first candidate rack is selected from the data relationship network; The location information of the M nodes included in the first candidate rack and the location information of the NM nodes included in the second candidate rack are taken as the most associated N location information.

4. The method according to claim 1, characterized in that, The step of using the cluster load information to filter target nodes from the nodes corresponding to the N location information according to preset rules includes: Based on the cluster load information, the load value of each node corresponding to the N location information is determined respectively; The node with the lowest load value is selected as the target node.

5. A data storage device, characterized in that, The device includes: The information acquisition module is used to acquire cluster load information in response to receiving data with tagged information sent by the client; the tagged information is the location information tag of the data; The node filtering module is used to determine the first candidate rack to which the candidate node belongs when there is a candidate node containing the tag information in the data relationship network; when the number of nodes M included in the first candidate rack is greater than or equal to N, select the location information of N nodes from the M nodes as the most relevant N location information; when there is no candidate node in the data relationship network, determine the similarity between the tag information and the data tag contained in each node under each rack in the cluster. The location information of the first node where the data label whose similarity satisfies the first threshold is determined; Based on the location information of the first node, N location information that is the same as or similar to the first node is selected from the data relationship network as the N most relevant location information; wherein, the N location information that is similar to the first node refers to the location information of other nodes under the same rack as the first node; using the cluster load information, target nodes are selected from the nodes corresponding to the N location information according to a preset rule; wherein, the data relationship network represents the mapping relationship between each node in the rack of the cluster and the data tag contained in each node; N is the number of replicas set in the cluster; The data storage module is used to forward the data to the target node for storage.

6. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 4.

Images