A data storage method and related apparatus

Abstract

A data storage method and related device, the data storage method comprising: obtaining historical transaction information; determining the closeness between a plurality of data according to the historical transaction information, the closeness representing the degree of association between the plurality of data; and performing data migration on the plurality of data according to the closeness between the plurality of data, to adjust the distributed layout of the plurality of data in a plurality of storage nodes. The historical transaction information represents information about processing the plurality of data in a historical transaction, and the plurality of data is stored in the plurality of storage nodes in a distributed manner. In this application, the distributed storage location of the data is more in line with the law of reading and writing data in the historical transaction. Thus, when processing an actual transaction, the number of data migration is less, and the cost of data migration is lower. In addition, the distributed storage of the data will change with the change of the transaction content.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data storage method and related apparatus. Background Technology

[0002] With the advent of the big data era, data can no longer be stored on a single node; instead, it is typically stored in a distributed manner. In distributed storage technology, the processing of a single transaction often requires migrating data from different locations to the same node before processing. Therefore, in distributed storage technology, it is necessary to choose a suitable distribution method to address the high cost of data migration during transaction execution. Summary of the Invention

[0003] To address the aforementioned issues, this application provides a data storage method and related apparatus that reduces data migration costs during transaction execution by combining the data distribution pattern with the data access patterns in historical transactions.

[0004] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0005] Firstly, this application provides a data storage method, comprising: acquiring historical transaction information; determining the proximity between multiple data sets based on the historical transaction information, wherein the proximity characterizes the degree of correlation between the multiple data sets; and migrating the multiple data sets based on the proximity between them to adjust the distributed layout of the multiple data sets across multiple storage nodes. Here, the historical transaction information represents information about processing multiple data sets in a historical transaction, and the multiple data sets are distributed and stored across multiple storage nodes.

[0006] In this application, the proximity between data is determined based on historical transaction information; then, data migration is performed based on this proximity to adjust the distributed layout of multiple data sets. Therefore, the distributed storage location of the data better aligns with the read / write patterns in historical transactions. Consequently, fewer data migrations and lower data migration costs are achieved when processing actual transactions. Furthermore, transaction content changes over time. Distributing data storage based on the proximity between data in historical transaction information ensures that the data distribution changes in accordance with the transaction content, allowing the data storage to adapt promptly to these changes. This further reduces the number of data migrations and lowers data migration costs after transactions change.

[0007] In one possible implementation, determining the proximity between multiple data based on historical transaction information includes: constructing a graph structure of multiple data based on historical transaction information, the graph structure including multiple graph nodes and connecting edges, the multiple graph nodes representing multiple data respectively, and the connecting edges representing the proximity between the two graph nodes they connect; and determining the proximity between multiple data based on the graph structure of multiple data.

[0008] In this implementation, a graph structure is constructed to represent the affinity between multiple data points. In the graph structure, multiple graph nodes represent multiple data points, and connecting edges represent the affinity between the two graph nodes they connect. Graph structures are one optional implementation for determining affinity; other implementation methods, such as using tabular data, can also be used.

[0009] In one possible implementation, the historical transaction information includes information from multiple historical transactions. The aforementioned construction of a graph structure based on the historical transaction information includes: constructing multiple graph nodes in the graph structure based on the multiple data processed by the historical transaction information; establishing a unit affinity for the data processed by each historical transaction; accumulating the unit affinity of the data processed by multiple historical transactions to obtain the affinity between the multiple data; and constructing connecting edges in the graph structure based on the affinity between the multiple data.

[0010] This implementation provides a method for determining proximity. By iterating through multiple historical transactions, data appearing simultaneously within a historical transaction are assigned a unit of proximity. Furthermore, the proximity between multiple data points is related to the frequency with which they are accessed simultaneously in historical transactions. For example, if two data points are accessed frequently in multiple historical transactions, it indicates a higher proximity between them, and a higher probability that they will be accessed simultaneously in the next transaction. Therefore, storing data with high proximity on the same node reduces the number of data migrations and lowers the cost of data migration in the next transaction.

[0011] In one possible implementation, the method further includes deleting connection edges in the graph structure whose proximity is lower than a first threshold.

[0012] In this implementation, when the number of data association queries is relatively low and the migration cost is relatively high, in order to prevent the return from being negative due to excessive cost, it is necessary to perform "edge deletion" on the graph structure, deleting connection edges with a closeness lower than the first threshold.

[0013] In one possible implementation, the multiple data include first data, second data, and third data. The method further includes: determining the storage locations of the first data, second data, and third data based on routing information of the first data, second data, and third data; when the distance between the storage locations of the first data and second data is less than a second threshold, fusing the first data and second data to obtain fourth data, wherein the affinity between the fourth data and the third data is the sum of the affinity between the first data and the third data, and the affinity between the second data and the third data.

[0014] In this implementation, data fusion is performed based on the data routing information, which increases the "granularity" of the data segmentation and prevents excessively fine data segmentation during the data migration process, thus avoiding the problem of multiple migrations. In other words, the routing table information limits the data from being divided too finely, thereby preventing increased migration costs.

[0015] In one possible implementation, the above-mentioned data migration based on the proximity between multiple data sets to adjust the distributed layout of the multiple data sets across multiple storage nodes includes: splitting the data in each of the multiple storage nodes into multiple data groups based on the proximity between the multiple data sets; obtaining the proximity between the multiple data groups based on the proximity between the multiple data sets; and migrating the multiple data groups based on the proximity between the multiple data groups to adjust the distributed layout of the multiple data groups across multiple storage nodes.

[0016] In this implementation, data is split across multiple storage nodes based on the proximity of the data, resulting in multiple data groups. Data migration is then performed based on the proximity of these data groups. This approach ensures that data with high proximity is grouped into the same data group, minimizing the number of data migrations and migration costs in subsequent transactions. Furthermore, since each data group contains multiple data items, migrating data in groups prevents the data from being fragmented too finely, thus avoiding increased migration costs.

[0017] In one possible implementation, the above-mentioned method of dividing the data of each storage node in multiple storage nodes according to the proximity between multiple data to obtain multiple data groups includes: dividing the data of each storage node into at least one data group; adjusting the splitting position between every two adjacent data groups in the at least one data group with the goal of maximizing the difference between the first benefit and the second benefit, wherein the first benefit represents the sum of the proximity between the moved data and the data in the first data group, the second benefit represents the sum of the proximity between the moved data and the data in the second data group, the moved data represents one or more data whose data group changed before and after the splitting position was moved, the first data group represents the data group to which the moved data belonged after the splitting position was moved, the second data group represents the data group to which the moved data belonged before the splitting position was moved, and the first data group and the second data group are respectively data groups in at least one data group; and obtaining multiple data groups based on the data groups adjusted according to the splitting position in each of the multiple storage nodes.

[0018] This implementation provides a method for determining the data splitting position. By adjusting the splitting position with the goal of maximizing the difference between the first and second benefits, it ensures that data with high affinity is grouped into the same data group, minimizing the number of data migrations in subsequent transactions.

[0019] In one possible implementation, the above-mentioned data migration of multiple data groups based on their proximity to adjust the distributed layout of the multiple data groups across multiple storage nodes includes: clustering the multiple data groups based on their proximity to divide them into multiple categories; and migrating the data groups in each category to one of the multiple storage nodes to adjust the distributed layout of the multiple data groups across multiple storage nodes.

[0020] In this implementation, multiple data groups are clustered into multiple categories based on their proximity; data groups within each category are then migrated to a storage node. This ensures that data with high proximity is migrated to the same storage node, minimizing the number of data migrations in subsequent transactions and reducing data migration costs.

[0021] Secondly, this application provides a data storage device, comprising: an acquisition module for acquiring historical transaction information; a processing module for determining the proximity between multiple data sets based on the historical transaction information, wherein proximity characterizes the degree of correlation between the multiple data sets; and a data migration module for adjusting the distributed layout of the multiple data sets across multiple storage nodes based on the proximity between the multiple data sets. The historical transaction information represents information about processing multiple data sets in a historical transaction, and the multiple data sets are distributed and stored across multiple storage nodes.

[0022] In one possible implementation, the above processing module is specifically used to: construct a graph structure of multiple data based on historical transaction information, the graph structure including multiple graph nodes and connecting edges, the multiple graph nodes respectively representing multiple data, and the connecting edges representing the proximity between the two graph nodes they connect; and determine the proximity between the multiple data based on the graph structure of the multiple data.

[0023] In one possible implementation, the historical transaction information includes information from multiple historical transactions. Specifically, the aforementioned processing module is used to: construct multiple graph nodes in a graph structure based on the multiple data processed by the historical transaction information; establish unit proximity for the data processed by each historical transaction; accumulate the unit proximity of the data processed by multiple historical transactions to obtain the proximity between the multiple data; and construct connecting edges in the graph structure based on the proximity between the multiple data.

[0024] In one possible implementation, the above processing module is also used to: delete connection edges in the graph structure whose proximity is lower than a first threshold.

[0025] In one possible implementation, the multiple data includes first data, second data, and third data. The aforementioned processing module is further configured to: determine the storage locations of the first data, second data, and third data based on routing information of the first data, second data, and third data; when the distance between the storage locations of the first data and second data is less than a second threshold, merge the first data and second data to obtain fourth data, wherein the affinity between the fourth data and the third data is the sum of the affinity between the first data and the third data, and the affinity between the second data and the third data.

[0026] In one possible implementation, the above processing module is specifically used to: split the data of each storage node in multiple storage nodes according to the proximity between multiple data, to obtain multiple data groups; obtain the proximity between multiple data groups according to the proximity between multiple data; and perform data migration on multiple data groups according to the proximity between multiple data groups to adjust the distributed layout of multiple data groups in multiple storage nodes.

[0027] In one possible implementation, the above processing module is specifically used to: divide the data of each storage node into at least one data group; adjust the splitting position between every two adjacent data groups in the at least one data group with the goal of maximizing the difference between the first benefit and the second benefit, where the first benefit represents the sum of the proximity between the moved data and the data in the first data group, the second benefit represents the sum of the proximity between the moved data and the data in the second data group, the moved data represents one or more data whose data group changed before and after the splitting position was moved, the first data group represents the data group to which the moved data belonged after the splitting position was moved, the second data group represents the data group to which the moved data belonged before the splitting position was moved, and the first data group and the second data group are respectively data groups in at least one data group; and obtain multiple data groups based on the data groups adjusted by the splitting position in each of the multiple storage nodes.

[0028] In one possible implementation, the above processing module is specifically used to: cluster multiple data groups according to the closeness between them, dividing the multiple data groups into multiple categories; and migrate the data groups in each category to one of the multiple storage nodes to adjust the distributed layout of the multiple data groups in the multiple storage nodes.

[0029] Thirdly, this application provides a computing device, including a memory and a processor. The memory stores instructions that, when executed by the processor, cause the aforementioned method to be implemented.

[0030] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-described method is implemented.

[0031] Fifthly, this application provides a computer program product including program instructions that, when executed by a computer, cause the computer to perform the above-described method. Attached Figure Description

[0032] The accompanying drawings used in the embodiments or technical description are briefly introduced below.

[0033] Figure 1a A schematic diagram illustrating the first example of a data processing transaction process as a solution to a related technology;

[0034] Figure 1b A schematic diagram illustrating a second example of a data processing transaction process as a solution to a related technology;

[0035] Figure 2a A schematic diagram illustrating the changes in data distribution and storage location before and after a historical transaction in another solution provided in this application embodiment;

[0036] Figure 2b A schematic diagram of a second example of a data processing transaction process, which is another solution provided in the embodiments of this application;

[0037] Figure 3 A flowchart illustrating a data storage method provided in an embodiment of this application;

[0038] Figure 4a A schematic diagram illustrating the graph structure for constructing the affinity between multiple data points in the first example, provided as an embodiment of this application;

[0039] Figure 4b A schematic diagram illustrating the graph structure for constructing the affinity between multiple data points in the second example provided in this application embodiment;

[0040] Figure 5a A schematic diagram illustrating a first example of deleting connection edges with an affinity lower than a first threshold, as provided in an embodiment of this application;

[0041] Figure 5b A schematic diagram illustrating a second example of deleting connection edges with an affinity lower than a first threshold, provided in an embodiment of this application;

[0042] Figure 6 A schematic diagram illustrating the data storage process provided in an embodiment of this application;

[0043] Figure 7a A schematic diagram of the data fusion process in the first example provided for the embodiments of this application;

[0044] Figure 7b A schematic diagram of the data fusion process in the second example provided for the embodiments of this application;

[0045] Figure 8 A flowchart illustrating a method for determining data segmentation positions provided in an embodiment of this application;

[0046] Figure 9a A schematic diagram illustrating the movement of the segmentation position as provided in an embodiment of this application;

[0047] Figure 9b A schematic diagram illustrating an example of how shifting the segmentation location generates revenue, as provided in an embodiment of this application.

[0048] Figure 10 A schematic diagram illustrating the relationship between the segmentation position and the revenue provided in the embodiments of this application;

[0049] Figure 11 A schematic diagram illustrating the partitioning results of the stored data across multiple nodes provided in an embodiment of this application;

[0050] Figure 12 A schematic diagram illustrating the clustering process of the segmented data groups provided in this application embodiment;

[0051] Figure 13 A schematic diagram illustrating an example of a data storage method provided in an embodiment of this application;

[0052] Figure 14 This is a schematic diagram of the composition of a data storage scheduling system provided in an embodiment of this application;

[0053] Figure 15 This is a schematic diagram illustrating the composition of a data storage device provided in an embodiment of this application;

[0054] Figure 16 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0056] In this article, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects have an "or" relationship; for example, A / B means A or B.

[0057] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0058] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0059] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0060] To facilitate understanding of the solutions provided in the embodiments of this application, a brief introduction to some of the terms involved in this solution will be given first.

[0061] A node is a hardware unit that can refer to a single hard drive or a single server. Typically, data exchange within a node is much faster than data exchange between nodes.

[0062] Distributed transaction: refers to a business process that involves reading and storing data across different nodes.

[0063] Closeness: refers to the degree of correlation between different data. Closeness is also known as the degree of association. Closeness can refer to the relationship between data itself. For example, data such as student "Zhang San's" name and student ID are data with high closeness; student "Zhang San's" name information and student "Li Si's" parent information are data with low closeness. Closeness can also refer to the relationship between data in a transaction. For example, if student "Zhang San's" score and student "Li Si's" score frequently appear together in multiple transactions, then the relationship between student "Zhang San's" score and student "Li Si's" score is high, and the closeness is high; if student "Zhang San's" score and student "Li Si's" score almost never appear together in multiple transactions, then the relationship between student "Zhang San's" score and student "Li Si's" score is low, and the closeness is low. In the embodiments of this application, "closeness" refers to the latter, that is, closeness refers to the relationship between data in a transaction.

[0064] With the advent of the big data era, data can no longer be stored on a single node; instead, it is typically stored in a distributed manner. During a transaction, data distributed across different locations needs to be migrated to the same node for processing. If the original data changes and needs updating, it is necessary to migrate the data back to its original storage location after the transaction is completed. Therefore, distributed storage technology requires selecting a suitable distribution method to address the high cost of data migration during transaction execution.

[0065] In one solution of related technologies, the affinity between different data is determined based on the correlation between the data itself; and the data is distributed and stored in a distributed manner according to the affinity between different data, that is, data with high affinity are stored on the same node as much as possible.

[0066] The first solution will be explained below with two examples.

[0067] Figure 1a This is a schematic diagram of a first example of a data processing transaction process as a solution to a related technology. For example, such as... Figure 1aAs shown, taking a server cluster as an example, the server cluster includes multiple servers, including master servers and slave servers. The master server is responsible for reading, writing, and storing data, while the slave servers are responsible for storing data. In the first example, a data processing transaction mainly includes: the user sending a user request to the execution module (such as a processor); the execution module generating a data query instruction based on the user request and sending the data query instruction to the master server; the master server responding to the data query instruction, retrieving data from both the master server and the slave servers, and sending the data to the execution module; and the execution module processing the data and sending the processing result back to the user.

[0068] For example, a user request might be "What is the distribution of student grades in this class?" The user sends the request to the execution module (e.g., a processor). Based on the user request, the execution module generates a data query instruction and sends it to the main server. In response to the data query instruction, the main server retrieves student A's grade 1 from slave server 1, student B's grade 2 from the main server, and student C's grade 3 from slave server 2, and sends the data "grade 1, grade 2, grade 3" to the execution module. The execution module processes the data and sends the processing results back to the user. The results can be displayed in a visual format such as a chart or table.

[0069] Figure 1b This is a schematic diagram of a second example of a data processing transaction process as a solution to a related technology. For example, as shown... Figure 1b As shown, unlike the first example, the execution module also needs to send the processed new data to the master server; the master server writes the processed data to the corresponding master server or slave server to update the data.

[0070] For example, a user request might be "What are the average scores of each student in this class across all subjects?" Since each student's average score is part of their overall grade, the master server adds each student's average score to the existing grade, resulting in new grade data. The master server then sends each student's new grade data to the corresponding master or slave server. For instance, it might write student A's new grade 1 to slave server 1, student B's new grade 2 to the master server, and student C's new grade 3 to slave server 2.

[0071] In the first solution, data is distributed and stored across multiple nodes based on the data's inherent relationships. The data required for data processing transactions may or may not be related to the data's inherent relationships. For example, the data required for data processing transactions may be a part of the data on different storage nodes.

[0072] For example, consider a distributed storage system containing a list of students and their related information (such as grades, height, and home address). In the first solution, because each student's information is linked to a single student, the relationship between each student and their associated information is very close. Therefore, during distributed storage, information is differentiated and distributed according to the student. For instance, a student's information might be stored on one node. However, in real-world transactions, it's common to need information on one or two students (such as grades). Therefore, each transaction involves data migration, resulting in high data migration costs.

[0073] In view of this, embodiments of this application provide another solution. A data storage method determines the proximity between different data based on historical transaction information; and stores the data based on the proximity between the different data.

[0074] Figure 2a This diagram illustrates the changes in data distribution and storage location before and after a historical transaction in another solution provided in this application embodiment. For example,... Figure 2a As shown, before a historical transaction occurs, data is stored on various storage nodes based on the data's proximity. Let's assume the historical transaction is... Figure 1a and Figure 1b The processing of transactions, or historical transaction information, includes transaction information for Transaction 1 and Transaction 2. Transaction 1's information includes phrases like "Transaction 1: What is the distribution of student scores in this class?" and "Data involved in Transaction 1: Score 1, Score 2, Score 3". Transaction 2's information includes phrases like "Transaction 2: What are the average scores of each student in each subject in this class?" and "Data involved in Transaction 2: Score 1, Score 2, Score 3". Based on historical transaction information, data with high affinity is designated as "Score 1, Score 2, Score 3". This data with high affinity is stored on a single node. For example, "Score 1, Score 2, Score 3" can be stored on a storage node on slave server 2. Alternatively, to differentiate between different students' scores, names and scores can be stored together on a storage node, such as "Name 1 and Score 1, Name 2 and Score 2, Name 3 and Score 3" stored on a storage node on slave server 2. This way, when a transaction related to "scores" occurs again, the master server directly reads data from a single storage node (e.g., slave server 2), reducing the number of data migrations and associated costs.

[0075] Figure 2b This is a schematic diagram of a second example of a data processing transaction process, providing another solution for an embodiment of this application. For example, as shown... Figure 2b As shown, based on Figure 2aThe data storage method allows the main server to retrieve the necessary data from server 2 when a transaction related to "grades" occurs again, such as "what is the ranking of the students' grades in this class?". This eliminates the need for data migration across multiple storage nodes, reducing the number of data migrations. For example, [the method could be described as follows:] Figure 1a and Figure 1b The number of data migrations was reduced from 3 to 1.

[0076] In this embodiment, the proximity between data is determined based on historical transaction information; then, the data is stored in a distributed manner based on this proximity. Therefore, the distributed storage location of the data better reflects the patterns of reading and writing data in historical transactions. Consequently, when processing actual transactions, fewer data migrations are required, and the cost of data migration is lower.

[0077] In this embodiment, the content of a transaction changes over time. Based on the proximity of data within historical transaction information, data is stored in a distributed manner. This ensures that the data distribution changes in accordance with the transaction content, making the data storage more consistent with the transaction content. This reduces the number of data migrations in actual transactions and lowers data migration costs.

[0078] Figure 3 This is a flowchart illustrating a data storage method provided in an embodiment of this application. Figure 3 As shown in the figure, this application provides a data storage method, which mainly includes the following steps:

[0079] Step S310: Obtain historical transaction information. This historical transaction information represents information about multiple data points processed in a historical transaction, with these data points distributed and stored across multiple storage nodes.

[0080] Step S320: Determine the affinity between multiple data points based on historical transaction information. Affinity represents the degree of correlation between any two data points.

[0081] Optionally, historical transaction information refers to information from multiple historical transactions. Based on the historical transaction information, the data processed by each historical transaction in the multiple historical transactions is determined; a unit affinity is established between every two data points processed by each historical transaction; the unit affinity between the data points processed by each historical transaction is accumulated to obtain the affinity between multiple data points.

[0082] In this context, the unit affinity between two data points can be understood as the affinity corresponding to the two data points appearing simultaneously in a single transaction; the unit affinity between two data points can be understood as the affinity corresponding to the total number of times the two data points appear. For example, the unit affinity of two data points appearing simultaneously in a single transaction is set to 1; the affinity of two data points appearing simultaneously in 5 transactions is 5; the unit affinity of two data points appearing simultaneously in a single transaction is set to 0.1; and the affinity of two data points appearing simultaneously in 5 transactions is 0.5.

[0083] The following two examples illustrate the process of determining intimacy based on historical transaction information.

[0084] In the first example, the historical transactions include the four historical transactions shown in Table 1.

[0085] Table 1 shows the data processed by the four historical transactions and each historical transaction office.

[0086] Historical Affairs Historical offices process data Transaction A Data 0, Data 5 Transaction B Data 1, Data 2, Data 3 Transaction C Data 2, Data 3, Data 4 Transaction D Data 0, Data 1, Data 5

[0087] Figure 4a This is a schematic diagram illustrating the graph structure for constructing the affinity between multiple data points in the first example, as provided in the embodiments of this application. Figure 4a As shown, based on the data processed by each historical transaction, a graph structure is established for each historical transaction. The graph structure includes multiple nodes (also called graph nodes) and connecting edges between every two nodes, such as the graph structure of transactions A to D. In the graph structure, each node represents a piece of data, and each connecting edge represents the unit affinity between the two connected nodes, such as 1.

[0088] Furthermore, based on the graph structure of each historical transaction, the graph structure of multiple data points within multiple transactions is determined. The graph structure of multiple transactions includes multiple nodes and connecting edges between every two nodes. Each node represents one data point from the multiple data points, and the connecting edge represents the proximity between the two connected nodes. For example, nodes (also called graph nodes) are obtained by combining nodes, and connecting edges are obtained by accumulating connecting edges, thus obtaining the graph structure of multiple historical transactions.

[0089] Alternatively, a graph structure can be constructed directly based on historical transaction information. For example, multiple graph nodes can be built based on the multiple data processed by historical transactions. A unit affinity score is established for the data processed by each historical transaction. The unit affinity scores of the data processed by multiple historical transactions are summed to obtain the affinity score between the multiple data points. Connecting edges are then constructed in the graph structure based on the affinity scores between the multiple data points.

[0090] It should be noted that graph structure is one possible way to implement the process of determining intimacy. The process of determining intimacy can also be implemented in other ways, such as using tabular data. For n data, an n*n table is used, and the data in the i-th row and j-th column represents the intimacy between the i-th data and the j-th data.

[0091] Weights can be set in the connecting edges to represent the closeness between the two nodes connected by the edge. The closeness represented by the weight of the connecting edge can be obtained by summing the unit closeness of the connecting edges. For example, the closeness or weight "2" between data 0 and data 5 is obtained by summing the unit closeness "1" between data 0 and data 5 in transaction A and the unit closeness "1" between data 0 and data 5 in transaction D.

[0092] For example, in Figure 4a In the diagram, the edge connecting node 0 and node 1 is represented as "1", the edge connecting node 0 and node 2 is represented as "0", the edge connecting node 0 and node 3 is represented as "0", the edge connecting node 0 and node 4 is represented as "0", and the edge connecting node 0 and node 5 is represented as "2". Therefore, the node representation of node 0 is [1,0,0,0,2]. By analyzing the node representation of a node, we can obtain the affinity between that node and other nodes.

[0093] To further illustrate the method for determining intimacy, this application also provides a second example. In the second example, in addition to the historical transactions AD shown in Table 1, the historical transactions also include transaction EI.

[0094] Table 2 shows the data processed by the nine historical transactions and each historical transaction office.

[0095]

[0096] Figure 4b This is a schematic diagram illustrating the graph structure for constructing the affinity between multiple data points in the second example, as provided in an embodiment of this application. Figure 4b As shown, based on the data processed by each historical transaction, a graph structure is established for each historical transaction. The graph structure includes multiple nodes and connecting edges between every two nodes, such as the graph structure of transactions A to I. In the graph structure, each node represents a piece of data, and each connecting edge represents the unit proximity between the two connected nodes, with unit proximity being, for example, 1.

[0097] Furthermore, based on the graph structure of each historical transaction, the graph structure is determined. The graph structure includes multiple nodes and connecting edges between every two nodes. Each node represents one data point from multiple data sets, and the connecting edges represent the proximity between the two connected nodes. For example, nodes are obtained by accumulating nodes, and connecting edges are obtained by accumulating connecting edges, thereby obtaining the graph structure of multiple historical transactions.

[0098] The closeness represented by the weight of a connecting edge can be obtained by summing the unit closeness of the connecting edges. For example, the closeness or weight between data 0 and data 5 is "3", which is obtained by summing the unit closeness "1" between data 0 and data 5 in transaction A, transaction D, and transaction E. Similarly, the closeness or weight between data 0 and data 1 is "2", which is obtained by summing the unit closeness "1" between data 0 and data 1 in transaction D and transaction F.

[0099] In historical transaction information, data with a closeness level below a first threshold can be deleted. This low closeness indicates a lower number of data queries within the historical transaction. Deleting low-closeness data reduces the number of data migrations during data storage, lowering migration costs. Taking a graph structure as an example, when the number of queries related to data relationships is low, but migration costs are high, to prevent negative returns due to excessive costs, it's necessary to perform "edge deletion" on the graph structure, removing connections with a closeness level below the first threshold.

[0100] Figure 5a This is a schematic diagram illustrating a first example of deleting connection edges with a proximity level below a first threshold, as provided in an embodiment of this application. Figure 5a As shown, in the first example above, the second structural graph in Figure 4 is obtained based on Table 1. The first threshold is 2, and connection edges with a closeness lower than 2 are deleted, resulting in... Figure 5a The graph structure on the right. Optionally, isolated nodes (such as data 1 and data 4) can be deleted.

[0101] For example, the affinity between data 3 and data 4 is 1, indicating that data 3 and data 4 have only been processed once in multiple historical transactions. Since the affinity between data 3 and data 4 is too low, the benefit of migrating data 3 and data 4 is not high, so it is necessary to remove the affinity (i.e., the connection edge) between data 3 and data 4.

[0102] Figure 5b This is a schematic diagram illustrating a second example of deleting connection edges with a proximity level below a first threshold, as provided in an embodiment of this application. Figure 5bAs shown, in the second example, the data between 0 and 5 may have more complex affinity; assuming the first threshold is 2, removing connection edges with affinity lower than 2 yields... Figure 5b The diagram structure on the right.

[0103] Step S330: Based on the proximity between multiple data points, perform data migration on the multiple data points to adjust the distributed layout of the multiple data points across multiple storage nodes.

[0104] Figure 6 This is a schematic diagram illustrating the data storage process provided in an embodiment of this application. For example... Figure 6 As shown, the data storage process involves storing data from different nodes to a target node based on the proximity of the data. These different nodes are, for example, node 1, node 2, and node 3. These nodes are also called storage nodes. In this way, when a data processing transaction occurs, data can be read from the target node for processing without needing to perform multiple data migrations across multiple nodes.

[0105] For example, taking nodes 1, 2, and 3 as an example, assume that node 1 stores data 0-29, node 2 stores data 30-89, and node 3 stores data 90-149. Then, assuming that the data with high affinity determined based on historical transaction information are data 0-9, data 45-59, data 60-74, and data 110-129, data 0-9 is read from node 1, data 45-59 and data 60-74 are read from node 2, and data 110-129 is read from node 3. The data involved in this transaction is then migrated to the target node. When a data processing transaction occurs, the data involved in this transaction includes some or all of data 0-9, data 45-59, data 60-74, and data 110-129. During the execution of this transaction, the migrated data is directly read from the target node for processing, without needing to perform multiple data migrations from multiple nodes.

[0106] It should be noted that multiple data migrations are required during the process of storing data from different nodes to the target node. To reduce the number of data migrations, this application embodiment divides the data into multiple data groups and performs data migration in this group manner. During the data grouping process, this application embodiment provides two optional optimization methods: First, data fusion is performed based on the physical location of the data to increase the "granularity" of the segmentation and prevent excessive migrations caused by overly fine segmentation; Second, the data segmentation position is determined based on the proximity of the data, ensuring that data with high proximity is grouped into the same data group, minimizing the number of data migrations in subsequent transactions and maximizing the benefits of the data storage method in this application embodiment.

[0107] The following is an exemplary description of the data fusion process.

[0108]

[0109] Optionally, when some data have similar physical addresses, these data can be treated as a single entity, without further internal subdivision. This avoids the increased migration frequency and costs associated with overly granular data segmentation. In other words, by determining the locational relationships between multiple data points based on their physical addresses, data with these relationships are treated as a single entity for migration, increasing the granularity of the segmentation, preventing excessive data splitting, and reducing the number of migrations and costs.

[0110] For example, data routing information reveals the proximity of data physical addresses; that is, data located close to each other have more similar information in the routing table. From the perspective of controlling migration costs, it is necessary to limit the data from being segmented too finely based on routing table information, thereby preventing increased migration costs.

[0111] For example, data routing table information or routing information can be used as part of the node representation in a graph structure. Graph nodes can be pre-fused based on the routing information to ensure that data that is close in location is not split.

[0112] Figure 7a This is a schematic diagram of the data fusion process in the first example provided for an embodiment of this application. For example... Figure 7a As shown, assuming data 0 and data 1 have similar physical addresses, they can be merged into data 0'; data 2 and data 5 have similar physical addresses, so they can be merged into data 2'; data 3 and data 4 have similar physical addresses, so they can be merged into data 3'. After merging, there is a proximity between data 0' and data 2', and a proximity between data 2' and data 3'. Here, data merging refers to combining the indices of two data sets into the index of a single data set, without changing the content of the data itself.

[0113] A routing table is a table stored on routers or other internet network devices that contains paths to specific network terminals. In some cases, it also includes metrics related to these paths. Generally, routing table information includes: destination network address, related network nodes, satisfaction level with a particular path, and expected path information. The destination network address indicates the target address for data transmission, which is also the storage address after the data is sent. Therefore, the destination network address in the routing information indicates the distance relationship between the physical addresses of the data. As shown in Table 3, based on the destination network addresses in the routing information, data 0 and data 1 have relatively close physical addresses and can be merged into data 0′; data 2 and data 5 have relatively close physical addresses and can be merged into data 2′; data 3 and data 4 have relatively close physical addresses and can be merged into data 3′.

[0114] Table 3. Destination network address of routing table information for different data.

[0115] data Destination network address Data 0 192.168.123.254 Data 1 192.168.123.259 Data 2 192.168.424.334 Data 3 192.168.714.252 Data 4 192.168.714.211 Data 5 192.168.424.358

[0116] Before data fusion, data 0 and data 5 had a similar affinity, requiring them to be stored on the same storage node, necessitating two data migrations. Similarly, data 2 and data 3 also had a similar affinity, requiring them to be stored on the same storage node, again necessitating two data migrations. The total number of data migrations before fusion was four.

[0117] After data fusion, there is a closeness between data 0′ and data 2′, and a closeness between data 2′ and data 3′. Therefore, data 0′, data 2′, and data 3′ need to be stored on the same storage node. After data fusion, the total number of data migrations is 3. Thus, after data fusion, the number of data migrations is reduced from 4 to 3, reducing the number of migrations and costs.

[0118] It should be noted that, taking data 2 and data 5 as an example, data 2 and data 5 are physically close. During the process of merging data 2 and data 5 into data 2′, if there is other data between data 2 and data 5, then that other data will be merged into data 2′ as well. This is to ensure that data 2 and data 5 can be migrated as a whole, reducing the number of migrations and costs.

[0119] Figure 7b This is a schematic diagram of the data fusion process in the second example provided for an embodiment of this application. For example... Figure 7b As shown, assuming the data storage location is... Figure 7aThe first example is the same, so it will not be repeated here. After data fusion, there is a closeness between each pair of data points 0′, 2′, and 3′. The closeness between the fused data and other data is the sum of the closenesses between the corresponding data points before fusion and other data.

[0120] In the second example, data 0-5 all have a degree of affinity, so they need to be stored on the same storage node. Before data fusion, the total number of data migrations was 6. After data fusion, data 0′, data 2′, and data 3′ need to be stored on the same storage node, reducing the total number of data migrations to 3. Thus, after data fusion, the number of data migrations is reduced from 6 to 3, decreasing both the number of migrations and the cost.

[0121] The following is an exemplary description of the process for determining the data segmentation location.

[0122] It's important to note that real-world scenarios are far more complex than the first two examples, involving a large number of nodes and a vast amount of data within each node. To more accurately analyze the patterns in historical transaction processing, a large volume of historical transactions needs to be analyzed. Consequently, the affinity between data points becomes complex; in a graph structure, for example, the affinity between data points forms a complex network. Therefore, when determining the data partitioning points, it's crucial to consider the impact of different partitioning locations on the affinity between data points.

[0123] Figure 8 This is a flowchart illustrating a method for determining data segmentation positions provided in an embodiment of this application. Figure 8 As shown in the figure, this application provides a method for determining the data segmentation position, which mainly includes the following steps:

[0124] Step S810: Divide the data into p equal parts to obtain p data groups, where p is a positive integer. For example, for the data 0-29, the data 0-29 can be divided into 3 equal parts.

[0125] For example, data partitioning can employ graph partitioning to obtain multiple data "slices". When requests need to access multiple slices, these requests can be split and distributed to the correct slices, and the returns from each slice can be "assembled" into a complete result. This distribution process is figuratively called "graph slicing," which is cutting a large graph into many smaller graphs.

[0126] Step S820: Traverse all data and determine whether two adjacent data belong to the same data group.

[0127] For example, when data is stored in the form of pages, all page cutting positions are traversed to determine whether two adjacent pages belong to the same data group. For instance, after dividing the data 0-29 into three equal parts, starting from data 0, it is determined whether every two adjacent data items belong to the same data group. A page in a database is a logical storage format for data and is the basic unit for data management by the database storage engine.

[0128] In step S830, when two adjacent data points belong to the same data group, no operation is performed. That is, the two adjacent data points are not split into two data groups.

[0129] Step S840: When two adjacent data do not belong to the same data group, determine the data splitting position between the two adjacent data.

[0130] In step S850, based on the cutting position obtained in step S840, the cutting position is slid left and right, and the optimal cutting position is determined according to the closeness of different positions.

[0131] Steps S810-S840 are equivalent to a "method for finding the data segmentation position". This method is exemplary. In other examples, other methods can also be used, such as first calculating the data corresponding to the segmentation position and then determining the position of the data page.

[0132] Step S850 indicates that after finding the data splitting position, the splitting position can be moved. The optimal splitting position is determined based on the proximity of the data at different positions.

[0133] The following is an example of how to determine the optimal segmentation position.

[0134] Figure 9a This is a schematic diagram illustrating the movement of the segmentation position as provided in an embodiment of this application. Figure 9a As shown, for example, data 0-29 can be divided into three equal parts: data group 1 consists of data 0-9; data group 2 consists of data 10-19; and data group 3 consists of data 20-29. However, this division may be unreasonable. For example, two data points with high affinity may be divided into different data groups, which is equivalent to changing one data migration into two data migrations. The division position between data group 2 and data group 3 is used as an example for illustration. The formula for calculating the benefit generated by moving the division position is shown in formula (1), where the benefit represents the increase in affinity when the division position is moved one data point to the left.

[0135] E = E out -E in (1)

[0136] In formula (1), E represents the benefit generated by the shift in the cutting position. out E represents the sum of the affinity of the moved data after the split position is moved, and the data in the current data group are data 1 to data n. out The calculation formula is shown in formula (2). The moved data refers to the data whose data group has changed due to the movement of the split position. e(i) represents the proximity between the moved data and the i-th data in the current data group.

[0137]

[0138] E in The calculation formula for E is shown in formula (3). in This represents the sum of the affinity scores of the data to be moved before the split position was moved, and all data in the current data group, where the data ranges from data 1 to data m. e(j) represents the affinity score between the data to be moved and the j-th data in the current data group.

[0139]

[0140] Figure 9b This is a schematic diagram illustrating an example of how shifting the segmentation location generates revenue, as provided in an embodiment of this application. Figure 9b As shown, in one example, with Figure 6 Taking the leftward shift of the splitting position between data group 2 and data group 3 as an example, the data being shifted is data 19; E out This represents the sum of the affinity between data 19 after the split position is moved and data 20-29 in data group 3; E in This represents the sum of the affinity between data 19 and data 10-18 in data group 2.

[0141] Figure 10 This is a schematic diagram illustrating the relationship between the segmentation position and the revenue provided in an embodiment of this application. For example... Figure 10 As shown, a larger benefit E indicates a greater increase in the affinity after moving the splitting position, suggesting that the current splitting position is better. Therefore, the splitting position corresponding to the maximum benefit E can be determined as the optimal data splitting position. Determining the optimal data splitting position based on benefit E through sliding splitting can be considered a sliding annealing technique that satisfies continuity requirements. Sliding annealing fully utilizes the consideration of data coherence requirements, ensures a low number of splitting operations, and achieves extremely fast data splitting and allocation.

[0142] It should be noted that after the data on the storage node is divided into multiple data groups or data blocks, the data groups on different nodes need to be classified into different categories according to the closeness between the data groups, and the data groups in each category are stored on the same storage node.

[0143] The following is an exemplary description of the process of storing data based on intimacy.

[0144] Figure 11 This is a schematic diagram illustrating the partitioning result of the stored data across multiple nodes provided in an embodiment of this application. For example... Figure 11 As shown, taking nodes 1, 2, and 3 as examples, assume that node 1 stores data 0-29, node 2 stores data 30-89, and node 1 stores data 90-149. According to the data partitioning method described above, the partitioning result of node 1's stored data 0-29 is data 0-9, data 10-19, and data 20-29; the partitioning result of node 2's stored data 30-89 is data 30-44, data 45-59, data 60-74, and data 75-89; and the partitioning result of node 1's stored data 90-149 is data 90-109, data 110-129, and data 130-149.

[0145] Figure 12 This is a schematic diagram illustrating the clustering process of the segmented data groups provided in an embodiment of this application. For example... Figure 12 As shown, the data in the storage node is divided into... Figure 11 After processing multiple data groups, it is necessary to cluster the data groups on different nodes into category A and category B based on the closeness between the data groups. The data groups of category A are stored on one storage node, and the data groups of category B are stored on another storage node, thereby adjusting the distributed layout of multiple data groups.

[0146] For example, the clustering method can be the "balanced minimum cut in graph partitioning". This method partitions an undirected graph into two subgraphs that satisfy the following two requirements: the number of nodes in the two subgraphs remains balanced; and the number of edges connecting the two subgraphs is minimized, or the sum of the affinity of the edges connecting the two subgraphs is minimized.

[0147] For example, a clustering method can also be to use the K-Means algorithm, which represents the nodes in the graph structure as vectors, and by solving the similarity of different vectors, the nodes in the graph structure are clustered into N categories to obtain N subgraphs; the data groups in each of the N categories are stored in the same storage node; where N is a positive integer.

[0148] Figure 13 This is a schematic diagram illustrating an example of a data storage method provided in an embodiment of this application. For example... Figure 13As shown in the example, the data storage method mainly includes the following steps:

[0149] Step S1310: Input data. The input data can be any data.

[0150] Step S1320: Obtain the index corresponding to the data. For example, an index is a common method in data storage. Generally, an index is a sorted data structure in a database management system used to assist in quickly querying and updating data in database tables. In addition to the actual data used, the database system also maintains data structures that satisfy specific search algorithms. These data structures reference (point to) the actual data used in the database in some way, allowing advanced search algorithms to be implemented on this data. This data structure is called an index.

[0151] Step S1330: Distribute and store data according to the index.

[0152] Step S1340: Perform a transaction query. A transaction refers to any transaction related to the input data; this embodiment of the application does not impose any limitations on this. These transactions, once they occur, become historical transactions, which are transactions used to analyze the closeness between data.

[0153] Step S1350, log generation. A log is an example of recording information related to a transaction.

[0154] Steps S1310 to S1350 are the standard data storage process, which involves first distributing the data and then performing related transaction processing by reading and writing the data.

[0155] Step S1360: Determine the closeness relationship between data based on the transaction log. This step is similar to the content of "Step S320" above and will not be repeated here.

[0156] Step S1370: Based on the proximity between data points, perform a graph partitioning algorithm to obtain new partition indices for the data. This step is described in detail in step S330 above and will not be repeated here.

[0157] Step S1380: Based on the new partition index of the data, execute step S1330.

[0158] In steps S1360 to S1380, the proximity relationship between data is determined based on the transaction log; a new partition index is determined based on the proximity relationship; and the data is re-stored based on the new partition index. This results in a data distribution after re-storage that better conforms to the transaction's pattern. When the transaction is queried again, the number of data migrations is reduced, lowering data migration costs. When the transaction pattern changes, the initial execution of this type of transaction involves more data migrations and incurs higher costs. After executing steps S1360 to S1380, the re-stored data distribution better conforms to the pattern of this type of transaction, thus reducing the number of data migrations and lowering data migration costs when this type of transaction is executed again.

[0159] Figure 14 This is a schematic diagram illustrating the composition of a data storage scheduling system provided in an embodiment of this application. Figure 14 As shown, a data storage scheduling system includes an execution module. The execution module can be any processor, computing device, etc. The data storage method provided in this application embodiment is applicable to database systems in private clouds or public clouds.

[0160] The system stores transaction-related data on the server or server cluster. Firstly, the execution module determines a data movement strategy based on the query transaction and sends data movement instructions to the server or server cluster. Secondly, the execution module determines data layout optimization instructions based on the data allocation plan and sends these instructions to the server or server cluster. In the second aspect, the execution module includes a solution module and a scheduling module. The solution module analyzes historical transaction query records in the logs, mines the data affinity, and issues a rearrangement plan to the scheduling module. The scheduling module performs data layout optimization, generates data layout optimization instructions, and sends these instructions to the server or server cluster.

[0161] Optionally, the execution module includes a routing decision module, a scheduling module, a solution module, and a log module.

[0162] For example, in a single transaction, the routing decision module obtains the user request; based on the user request, it determines the distribution of the data corresponding to the request across the server cluster; based on the data distribution, it determines a data migration strategy and sends the data migration strategy instruction to the master server in the server cluster. The master server executes the data migration strategy instruction to perform data migration and sends the migrated data to the routing decision module. The routing decision module then executes the transaction based on the migrated data. After the transaction is completed, the log module generates and stores the transaction log.

[0163] Optionally, the solution module includes a data acquisition module, a proximity mining module, and a data partitioning module. The data acquisition module is used to read historical transaction information; the proximity mining module is used to mine the proximity between data based on historical transaction information; and the data partitioning module is used to partition the data index based on the proximity between data.

[0164] For example, after one or more transactions occur, the data acquisition module retrieves data from the log module to query and read historical transaction information. The proximity mining module mines the proximity between data based on historical transaction information; the data partitioning module partitions the data index based on the proximity between data, obtaining a data allocation plan. The data allocation plan is essentially a plan for storing data groups within each clustered category based on the storage space of each storage node. The scheduling module performs data migration based on the data allocation plan (partitioning results).

[0165] Optionally, the data storage scheduling system can perform a shutdown refresh, and then re-execute data storage according to the data allocation plan. For example, at intervals, the solver module can recalculate the affinity relationship between processed data to determine the data allocation plan; the scheduling module then re-executes data storage according to the data allocation plan. This embodiment of the application requires reading historical data and then refreshing the data storage location.

[0166] This application provides a method for determining data affinity with transactions and a distributed storage strategy method, which can adapt to changes in transaction query information. Based on historical transaction data, the affinity between data is determined to adapt to changes in actual transactions.

[0167] This application provides a method for determining affinity. Specifically, based on statistically obtained historical transaction records, a unit affinity is established for the data or indexes involved in a single historical transaction. The affinity of data from all historical transactions is then summed to obtain a graph structure with data or indexes as nodes.

[0168] This application provides an adaptive distributed data storage method. The method involves: reading historical transaction information of the distributed data; mining the data affinity based on the historical transaction information; partitioning the data index based on the data affinity; and migrating the data based on the partitioning results. Optionally, this process can be executed periodically to achieve adaptive changes in transaction focus in the distributed storage results.

[0169] Based on the same concept as the foregoing embodiments, this application also provides a data storage device.

[0170] Figure 15 This is a schematic diagram illustrating the composition of a data storage device provided in an embodiment of this application. Figure 15 As shown, this application embodiment provides a data storage device 1500, which mainly includes:

[0171] The acquisition module 1510 is used to acquire historical transaction information. This historical transaction information represents information about multiple data points processed in a historical transaction, with these data points distributed and stored across multiple storage nodes.

[0172] The processing module 1520 is used to determine the proximity between multiple data based on historical transaction information, where proximity represents the degree of correlation between multiple data; and to perform data migration on multiple data based on the proximity between multiple data to adjust the distributed layout of multiple data in multiple storage nodes.

[0173] In one possible implementation, the processing module 1520 is specifically used to: construct a graph structure of multiple data based on historical transaction information, the graph structure including multiple graph nodes and connecting edges, the multiple graph nodes respectively representing multiple data, and the connecting edges representing the proximity between the two graph nodes they connect; and determine the proximity between the multiple data based on the graph structure of the multiple data.

[0174] In one possible implementation, the historical transaction information includes information from multiple historical transactions. Specifically, the processing module 1520 is used to: construct multiple graph nodes in a graph structure based on the multiple data processed by the historical transaction information; establish unit proximity for the data processed by each historical transaction; accumulate the unit proximity of the data processed by multiple historical transactions to obtain the proximity between the multiple data; and construct connecting edges in the graph structure based on the proximity between the multiple data.

[0175] In one possible implementation, the processing module 1520 is further configured to: delete connection edges in the graph structure whose proximity is lower than a first threshold.

[0176] In one possible implementation, the multiple data include first data, second data, and third data. The processing module 1520 is further configured to: determine the storage locations of the first data, second data, and third data based on routing information of the first data, second data, and third data; when the distance between the storage locations of the first data and second data is less than a second threshold, merge the first data and second data to obtain fourth data, wherein the affinity between the fourth data and the third data is the sum of the affinity between the first data and the third data and the affinity between the second data and the third data.

[0177] In one possible implementation, the processing module 1520 is specifically used to: divide the data of each storage node in multiple storage nodes according to the proximity between multiple data, to obtain multiple data groups; obtain the proximity between multiple data groups according to the proximity between multiple data; and perform data migration on multiple data groups according to the proximity between multiple data groups to adjust the distributed layout of multiple data groups in multiple storage nodes.

[0178] In one possible implementation, the processing module 1520 is specifically used to: divide the data of each storage node into at least one data group; adjust the splitting position between every two adjacent data groups in the at least one data group with the goal of maximizing the difference between the first benefit and the second benefit, wherein the first benefit represents the sum of the proximity between the moved data and the data in the first data group, the second benefit represents the sum of the proximity between the moved data and the data in the second data group, the moved data represents one or more data whose data group changed before and after the splitting position was moved, the first data group represents the data group to which the moved data belonged after the splitting position was moved, the second data group represents the data group to which the moved data belonged before the splitting position was moved, and the first data group and the second data group are respectively data groups in at least one data group; and obtain multiple data groups based on the data groups adjusted by the splitting position in each of the multiple storage nodes.

[0179] In one possible implementation, the processing module 1520 is specifically used to: cluster multiple data groups according to the closeness between them, dividing the multiple data groups into multiple categories; and migrate the data groups in each category to one of the multiple storage nodes to adjust the distributed layout of the multiple data groups in the multiple storage nodes.

[0180] The following is about... Figure 14 A data storage scheduling system is shown, such as Figure 15 The software and hardware implementation of the data storage device 1500 shown (hereinafter referred to as "the device in the embodiments of this application") needs further explanation.

[0181] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0182] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0183] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, a module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0184] The multiple computing devices included in the module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0185] It should be noted that, in other embodiments, the apparatus in this application embodiment may additionally provide one or more modules for performing any of the steps included in the above implementation. The steps implemented by one or more modules in the apparatus in this application embodiment can be specified as needed, and more or fewer modules can be obtained than in the embodiments of this application to implement different steps in the above method, thereby realizing all the functions of the apparatus in the embodiments of this application.

[0186] Based on the same concept as the foregoing embodiments, this application also provides a computing device, which includes at least a processor and a memory. The memory stores a program, and when the processor reads the program, it can implement the algorithmic functions embodied by the above-described methods or devices.

[0187] Figure 16 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 16 As shown, the computing device 1600 includes at least one processor 1601, a memory 1602, and a communication interface 1603. The processor 1601, memory 1602, and communication interface 1603 are communicatively connected, which can be achieved via a wired (e.g., bus) or wireless connection. The communication interface 1603 is used to receive data sent by other devices; the memory 1602 stores computer instructions, and the processor 1601 executes these computer instructions to perform the method described in the aforementioned method embodiments.

[0188] It should be understood that, in the embodiments of this application, the processor 1601 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0189] The memory 1602 may include read-only memory and random access memory, and provides instructions and data to the processor 1601. The memory 1602 may also include non-volatile random access memory.

[0190] The memory 1602 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM).

[0191] It should be understood that the computing device 1600 according to the embodiments of this application can execute the methods mentioned in the embodiments of this application. For a detailed description of the implementation of the method, please refer to the above text. For the sake of brevity, it will not be repeated here.

[0192] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned technical solutions are implemented.

[0193] An embodiment of this application provides a chip including at least one processor and an interface. The at least one processor determines program instructions or data through the interface. The at least one processor is used to execute the program instructions to implement the technical solutions mentioned above.

[0194] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the aforementioned technical solutions.

[0195] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0196] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, processor-executed software modules, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0197] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data storage method, characterized in that, include: Obtain historical transaction information, which represents information about multiple data points processed in a historical transaction, and the multiple data points are distributed and stored on multiple storage nodes; Based on the historical transaction information, the affinity between the multiple data sets is determined, whereby affinity represents the degree of correlation between the multiple data sets. Based on the proximity between the multiple data sets, data migration is performed on the multiple data sets to adjust their distributed layout across the multiple storage nodes.

2. The method according to claim 1, characterized in that, Determining the affinity between the multiple data sets based on the historical transaction information includes: Based on the historical transaction information, a graph structure of the multiple data is constructed. The graph structure includes multiple graph nodes and connecting edges. The multiple graph nodes respectively represent the multiple data, and the connecting edges represent the proximity between the two graph nodes they connect. Based on the graph structure of the multiple data sets, the affinity between the multiple data sets is determined.

3. The method according to claim 2, characterized in that, The historical transaction information includes information on multiple historical transactions; The step of constructing a graph structure for the multiple data sets based on the historical transaction information includes: Based on the processed data from the historical transaction information, construct multiple graph nodes in the graph structure; Establish a unit affinity for the data processed in each of the multiple historical transactions; The unit affinity of the data processed by the multiple historical offices is summed to obtain the affinity between the multiple data sets; Based on the proximity between the multiple data points, the connecting edges in the graph structure are constructed.

4. The method according to claim 2 or 3, characterized in that, The method further includes: Delete the connecting edges in the graph structure whose affinity is lower than the first threshold.

5. The method according to any one of claims 1-4, characterized in that, The plurality of data includes first data, second data, and third data; The method further includes: Based on the routing information of the first data, the second data, and the third data, determine the storage locations of the first data, the second data, and the third data; When the distance between the storage locations of the first data and the second data is less than a second threshold, the first data and the second data are merged to obtain the fourth data. The affinity between the fourth data and the third data is the sum of the affinity between the first data and the third data and the affinity between the second data and the third data.

6. The method according to any one of claims 1-5, characterized in that, The step of migrating the multiple data sets based on their proximity to each other, in order to adjust the distributed layout of the multiple data sets across the multiple storage nodes, includes: Based on the proximity between the multiple data, the data in each of the multiple storage nodes is divided to obtain multiple data groups; The affinity between the multiple data sets is obtained based on the affinity between the multiple data sets. Based on the proximity between the multiple data groups, data migration is performed on the multiple data groups to adjust the distributed layout of the multiple data groups in the multiple storage nodes.

7. The method according to claim 6, characterized in that, The data in each of the multiple storage nodes is divided according to the proximity between the multiple data, resulting in multiple data groups, including: Divide the data on each storage node into at least one data group; With the goal of maximizing the difference between the first benefit and the second benefit, the splitting position between every two adjacent data groups in the at least one data group is adjusted. The first benefit represents the sum of the proximity between the moved data and the data in the first data group, and the second benefit represents the sum of the proximity between the moved data and the data in the second data group. The moved data represents one or more data whose data group changed before and after the splitting position was moved. The first data group represents the data group to which the moved data belonged after the splitting position was moved, and the second data group represents the data group to which the moved data belonged before the splitting position was moved. The first data group and the second data group are data groups in the at least one data group. The plurality of data groups are obtained by adjusting the data groups based on the splitting positions in each of the plurality of storage nodes.

8. The method according to claim 6 or 7, characterized in that, The step of migrating data across multiple data groups based on their proximity to each other, in order to adjust the distributed layout of the multiple data groups across multiple storage nodes, includes: Based on the closeness between the multiple data groups, the multiple data groups are clustered and divided into multiple categories. The data groups in each of the multiple categories are migrated to one of the multiple storage nodes to adjust the distributed layout of the multiple data groups in the multiple storage nodes.

9. A data storage device, characterized in that, include: The acquisition module is used to acquire historical transaction information, which represents information about multiple data processed in a historical transaction, and the multiple data are distributed and stored on multiple storage nodes. The processing module is used to determine the affinity between the multiple data based on the historical transaction information, wherein the affinity characterizes the degree of correlation between the multiple data. Furthermore, based on the proximity between the multiple data sets, data migration is performed on the multiple data sets to adjust the distributed layout of the multiple data sets across the multiple storage nodes.

10. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-8 to be implemented.

Images