Data migration method, device, system and computer-readable storage medium

Abstract

The present disclosure relates to a data migration method, device, system and computer-readable storage medium, and relates to the field of computer technology. The method of the present disclosure includes: determining the position of the label of the newly added graph engine service in the hash ring; determining the graph shard data to be migrated belonging to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each allocated graph shard data in the hash ring; sending indication information to the newly added graph engine service and the original graph engine service of the graph shard data to be migrated, and migrating the graph shard data to be migrated from the original graph engine service to the newly added graph engine service.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data migration method, device, system and computer-readable storage medium. Background Art

[0002] In recent years, the popularity of mobile Internet has promoted the rapid development of e-commerce (E-Commerce), social networks (Social Network) and other fields, generating a massive amount of user and product data. How to efficiently use this data to optimize user experience and continuously create value has become a major challenge faced by such application providers. The graph data structure has a strong expressive ability and is very suitable for modeling the relationship between users in social networks and the relationship between users and products in e-commerce systems. Therefore, introducing the graph data structure into e-commerce systems and social networks to characterize the relationship between users and products and between users is conducive to better capturing the evolution of user interests, and can provide more guidance information for personalized recommendations in e-commerce systems, online advertising, and friend recommendations in social networks, thereby optimizing user experience and creating value.

[0003] In the actual application of graph modeling, an online graph engine service is usually built to load graph data. This service provides graph data query services for other services. In real industrial-level applications, the amount of graph data is very large and cannot be loaded by a single machine. When it is actually online, it is necessary to split the graph data to obtain graph shard data, build a distributed graph engine to load the graph data, and provide external graph data query services.

[0004] As the number of active users in e-commerce systems and social networks continues to grow, user behaviors are becoming more diverse. Therefore, the graph data obtained by graph modeling of e-commerce systems and social networks continues to grow. And because the number of online graph shard data N generally remains constant, the size of a single shard file will continue to increase, resulting in the graph engine service machine memory being unable to load. Therefore, it is necessary to increase the number of graph engine services M to re-allocate graph data shard files to graph engine services so that graph data shard files can be loaded smoothly. Summary of the invention

[0005] The inventors found that when the graph data of an online distributed graph engine continues to grow and the number of graph data engine services needs to be expanded, the existing methods will lead to excessive migration of graph shard data and even cause some queries to fail during the large-scale migration process.

[0006] A technical problem to be solved by the present disclosure is: how to reduce the migration amount of graph shard data while expanding the graph engine.

[0007] According to some embodiments of the present disclosure, a data migration method is provided, including: determining the position of a label of a newly added graph engine service in a hash ring; determining the graph shard data to be migrated belonging to the newly added graph engine service based on the distance between the position of the label in the hash ring and the positions of each allocated graph shard data in the hash ring; sending indication information to the newly added graph engine service and the original graph engine service of the graph shard data to be migrated, and migrating the graph shard data to be migrated from the original graph engine service to the newly added graph engine service.

[0008] In some embodiments, there are multiple labels for the newly added graph engine service, and determining the position of the label of the newly added graph engine service in the hash ring includes: obtaining multiple labels for the newly added graph engine service; determining the hash value of each label according to a preset hash function; and determining the position of each label in the hash ring according to the hash value of each label.

[0009] In some embodiments, determining the graph shard data to be migrated belonging to a newly added graph engine service based on the position of each label in a hash ring and the distance between the positions of each allocated graph shard data in the hash ring includes: determining, in the hash ring in accordance with a preset order, the labels that are closest to the positions of each graph shard data in the hash ring, as the labels corresponding to each graph shard data; regenerating a mapping table of graph shard data and graph engine services based on the labels corresponding to each graph shard data and the graph engine services to which each label belongs, wherein the graph shard data to be migrated corresponds to the newly added graph engine service in the mapping table.

[0010] In some embodiments, indication information is sent to a newly added graph engine service and the original graph engine service of the graph slice data to be migrated, and migrating the graph slice data to be migrated from the original graph engine service to the newly added graph engine service includes: sending indication information to each graph engine service, wherein the indication information includes a mapping table; receiving a loading completion message sent by the newly added graph engine service, wherein the loading completion message indicates that the newly added graph engine service has completed the loading of the graph slice data to be migrated according to the mapping table; sending a deletion indication to the original graph engine service; receiving a deletion completion message sent by the original graph engine service, wherein the deletion completion message indicates that the original graph engine service has completed the deletion of the graph slice data to be migrated according to the mapping table.

[0011] In some embodiments, the method also includes: when the graph shard data to be migrated is migrated from the original graph engine service to the newly added graph engine service, sending a mapping table of the graph shard data and the graph engine service to the graph data query service, wherein the graph shard data to be migrated corresponds to the newly added graph engine service in the mapping table.

[0012] In some embodiments, the method also includes: determining the hash value of the identification of each entity in the graph, wherein the entity includes the data of the node or the data of the edge of the graph; determining the identification of the graph shard data to which each entity belongs based on the hash value of the identification of each entity; determining the hash value of the identification of each graph shard data; and determining the position of each graph shard data in the hash ring based on the hash value of the identification of each graph shard data as the position of each allocated graph shard data in the hash ring.

[0013] In some embodiments, the number of tags for each graph engine service is determined based on the processing capability of the graph engine service, wherein the stronger the processing capability of the graph engine service, the greater the number of tags.

[0014] In some embodiments, the method also includes: a graph data query service receives a data query request, wherein the data query request includes: an entity to be queried, the entity including data of a node or edge of a graph; the graph data query service determines a hash value of an identifier of the entity to be queried; the graph data query service determines an identifier of the graph shard data to which the entity to be queried belongs based on the hash value of the identifier of the entity to be queried; the graph data query service searches a mapping table for a corresponding graph engine service as a target graph engine service based on the identifier of the graph shard data to which the entity to be queried belongs; the graph data query service queries the target graph engine service for the entity to be queried.

[0015] According to some other embodiments of the present disclosure, a data migration device is provided, including: a label position determination module, used to determine the position of the label of a newly added graph engine service in a hash ring; a migration information determination module, used to determine the graph shard data to be migrated belonging to the newly added graph engine service based on the distance between the position of the label in the hash ring and the position of each allocated graph shard data in the hash ring; a migration indication module, used to send indication information to the newly added graph engine service and the original graph engine service of the graph shard data to be migrated, so as to migrate the graph shard data to be migrated from the original graph engine service to the newly added graph engine service.

[0016] According to some further embodiments of the present disclosure, a data migration device is provided, comprising: a processor; and a memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the data migration method as described in any of the aforementioned embodiments.

[0017] According to some further embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the data migration method of any of the aforementioned embodiments.

[0018] According to some other embodiments of the present disclosure, a data migration system is provided, comprising: a data migration device of any of the aforementioned embodiments, and multiple graph engine services; wherein, when the graph engine service is a newly added graph service engine or is the original graph engine service of the graph slice data to be migrated, the graph engine service is used to receive indication information sent by the data migration device to realize the migration of the graph slice data to be migrated.

[0019] In some embodiments, the indication information includes a mapping table; when the graph engine service is a newly added graph service engine, it is used to complete the loading of the graph shard data to be migrated according to the mapping table, and send a loading completion message to the data migration device; when the graph engine service is the original graph service engine, it is used to receive a deletion indication sent by the data migration device, complete the deletion of the graph shard data to be migrated according to the mapping table, and send a deletion completion message to the data migration device.

[0020] In some embodiments, the system further includes: a graph data query service, configured to receive a mapping table sent by the migration device and provide a query service according to the mapping table.

[0021] In some embodiments, a graph data query service is used to receive a data query request, determine a hash value of an identifier of an entity to be queried, determine an identifier of the graph shard data to which the entity to be queried belongs based on the hash value of the identifier of the entity to be queried, and in a mapping table, search for a corresponding graph engine service based on the identifier of the graph shard data to which the entity to be queried belongs, and query the target graph engine service for the entity to be queried as a target graph engine service; wherein the data query request includes: an entity to be queried, and the entity includes data of a node or edge of a graph.

[0022] The present disclosure adopts a hash ring method to determine the relationship between graph shard data and graph engine services, and then determines the graph shard data to be migrated that belongs to the newly added graph engine service, thereby migrating the graph shard data to be migrated from the original graph engine service to the newly added graph engine service. When the number of graph engine services is expanded in the present disclosure, there is no data migration between the original graph engine services, and there is only data migration from the original graph engine service to the newly added graph engine service, which reduces the migration amount of graph shard data and improves the migration efficiency of graph shard data, which can improve the real-time performance and query effect of graph data query, optimize user experience and improve graph modeling benefits.

[0023] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram showing some existing mapping methods of graph shard data to graph engine services.

[0026] Figure 2 A schematic diagram showing a flow chart of a data migration method according to some embodiments of the present disclosure.

[0027] Figure 3 A schematic diagram illustrating a method for mapping graph shard data to a graph engine service according to some embodiments of the present disclosure.

[0028] Figure 4 A schematic diagram showing a flow chart of a data migration method according to other embodiments of the present disclosure.

[0029] Figure 5 A schematic flowchart showing a data migration method according to some other embodiments of the present disclosure is shown.

[0030] Figure 6 A schematic diagram showing the structure of a data migration device according to some embodiments of the present disclosure is shown.

[0031] Figure 7 A schematic diagram showing the structure of a data migration device according to some other embodiments of the present disclosure.

[0032] Figure 8 A schematic diagram showing the structure of a data migration device according to some other embodiments of the present disclosure.

[0033] Fig. 9 A schematic diagram showing the structure of a data migration system according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] Figure 1The figure shows a method for migrating graph shard data known to the inventor. Since the current graph shard data is mapped to the graph engine service mainly through modular operation as the routing rule, when the number of graph engine services is increased, it is equivalent to re-customizing the routing rules. For example, Figure 1 As shown in the figure, it is assumed that the current number of graph data shard files N = 12, that is, there is a graph data shard file set {P0, P1, P2, P3, P4, P5, P6, P7, P8, P9, P 10 ,P 11}. The number of graph engine services M = 3, that is, there is a graph engine service set {S0, S1, S2}. When M increases by 1, a graph engine service is added, and the graph engine set is {S0, S1, S2, S3}. The distribution relationship between graph shard data and graph engine services is as follows Figure 1 As shown in the figure, before adding the graph engine service S3, the graph shard data corresponding to S0 is P0, P3, P6, P9, and the graph shard data corresponding to S1 is P1, P4, P7, P8. 10 , the graph shard data corresponding to S2 are P2, P5, P8, P 11 After adding the graph engine service S3, the graph shard data corresponding to S0 is P0, P4, P8, the graph shard data corresponding to S1 is P1, P5, P9, and the graph shard data corresponding to S2 is P2, P6, P7, P8, P9, P10, P11, P12, P13, P14, P15, P16, P17, P18, P19, P20, P21, P22 10 , the graph shard data corresponding to S3 are P3, P7, P 11 It can be seen that there is a lot of data migration between graph engine services.

[0036] In view of the above problems, the present disclosure proposes a data migration method. Figure 2 to Figure 5 Give a description.

[0037] Figure 2 Flowcharts of some embodiments of the data migration method disclosed in the present invention. Figure 2 As shown, the method of this embodiment includes: steps S202 to S206.

[0038] In step S202, the position of the label of the newly added graph engine service in the hash ring is determined.

[0039] In some embodiments, the newly added graph engine service has one or more labels. The label of the graph engine service refers to the specific identifier of the graph engine service. It can be composed of the domain name + index (index) or ip + index of the graph engine service. Assuming that a graph engine service domain name is graph_service_1, and there are three labels, each label can be represented as graph_service_1_0, graph_service_1_1, and graph_service_1_2.

[0040] The label of the graph engine service can be understood as the virtual node of the graph engine service. In the consistent hashing algorithm, the number of labels for each graph engine service can be set to 1, that is, each graph engine service has only one unique (physical) node in the hash ring, and the graph shard data mapped to the node is loaded by the graph engine service. However, when the label of the graph engine service is set to 1, it is easy to cause uneven graph shard data in different graph engine services during capacity expansion, and even cause service avalanche. Therefore, setting the label of the graph engine service to multiple can make the distribution of graph shard data more even and reduce the occurrence of avalanche.

[0041] In some embodiments, the number of tags of each graph engine service is determined according to the processing capability of the graph engine service, wherein the stronger the processing capability of the graph engine service, the greater the number of tags. The more tags a graph engine service has, the more graph sharding data can be allocated to the graph engine service. A graph engine service with stronger processing capability allocates more graph sharding data, which can fully utilize the resources of the graph engine service, reduce the load of graph engine services with poor capabilities, and improve the processing efficiency of graph sharding data.

[0042] In some embodiments, multiple tags of a newly added graph engine service are obtained; the hash value of each tag is determined according to a preset hash function; and the position of each tag in the hash ring is determined according to the hash value of each tag. The preset hash function is, for example, an existing hash function such as murmurhash3 or md5. A hash ring is constructed, in which the maximum value of the hash function corresponds to a radian value of 2π, and the minimum value corresponds to a radian value of 0. Different hash function values ​​correspond to different positions in the hash ring. The position of each tag in the hash ring can be determined in a preset order (clockwise or counterclockwise). The method for calculating the position in the hash ring can refer to the prior art and will not be repeated here.

[0043] In some embodiments, in response to the increase in the amount of graph shard data reaching a preset amount, the number of newly added graph engine services is determined according to the increase in the amount of graph shard data. For each newly added graph engine service, the number of corresponding tags and each tag are determined according to the processing capacity of the graph engine service. Then, step S102 and subsequent methods are executed.

[0044] In step S204, the graph shard data to be migrated belonging to the newly added graph engine service is determined according to the distance between the position of the label in the hash ring and the position of each allocated graph shard data in the hash ring.

[0045] The position of each graph shard data in the hash ring has been determined before the graph engine service is added. In some embodiments, in the hash ring, the labels that are closest to the positions of each graph shard data in the hash ring are determined in sequence in a preset order (clockwise or counterclockwise) as the labels corresponding to each graph shard data; according to the labels corresponding to each graph shard data and the graph engine service to which each label belongs, a mapping table of graph shard data and graph engine service is regenerated. The mapping table includes the correspondence between the identifiers of each graph shard data and the identifiers of each graph engine service. The identifier of the graph engine service is different from the label of the graph engine service, and is the unique identifier of the graph engine service. The graph shard data to be migrated corresponds to the newly added graph engine service in the mapping table.

[0046] like Figure 3 As shown, for example, a new engine service S3 is added, which has three labels S3.0, S3.1, and S3.2. By calculating the position of each label in the hash ring, three labels S3.0, S3.1, and S3.2 are added to the consistent hash ring. In a counterclockwise direction, it is determined that the graph shard data P6 corresponds to label S3.1, P9 corresponds to label S3.2, and P1 corresponds to label S3.0.

[0047] In step S206, instruction information is sent to the newly added graph engine service and the original graph engine service of the graph slice data to be migrated, so that the graph slice data to be migrated is migrated from the original graph engine service to the newly added graph engine service.

[0048] The identifier of the graph slice data to be migrated can be directly sent to the original graph engine service and the newly added graph engine service, and the newly added graph engine service loads the graph slice data to be migrated according to the identifier of the graph slice data to be migrated (for example, loads the graph slice data to be migrated from the network disk). After the loading is completed, the newly added graph engine service sends a loading completion message to the data migration device (the device that executes the method of the present invention). The data migration device sends a deletion indication to the original graph engine service, and the original graph engine service deletes the graph slice data to be migrated. The data migration device can notify the newly added graph engine service to go online and provide services.

[0049] In some embodiments, an indication message is sent to each graph engine service, wherein the indication message includes a mapping table; a loading completion message is received from a newly added graph engine service, wherein the loading completion message indicates that the newly added graph engine service has completed loading the graph slice data to be migrated according to the mapping table; a deletion indication is sent to the original graph engine service; a deletion completion message is received from the original graph engine service, wherein the deletion completion message indicates that the original graph engine service has completed deleting the graph slice data to be migrated according to the mapping table. The mapping table is sent to each graph engine service, and each graph engine service can determine whether it is necessary to load or delete certain graph slice data based on the mapping table and the stored graph slice data.

[0050] For example, a deletion instruction may be sent to the original graph engine service through a heartbeat mechanism. The heartbeat mechanism may be used to determine whether each graph engine service is working properly. Sending a deletion instruction to the original graph engine service through a message of the heartbeat mechanism may reduce the sending of additional deletion instructions.

[0051] The method of the above embodiment uses a hash ring method to determine the relationship between the graph shard data and the graph engine service, and then determines the graph shard data to be migrated that belongs to the newly added graph engine service, thereby migrating the graph shard data to be migrated from the original graph engine service to the newly added graph engine service. When the number of graph engine services is expanded in the method of the above embodiment, there is no data migration between the original graph engine services, and there is only data migration from the original graph engine service to the newly added graph engine service, which reduces the migration amount of graph shard data and improves the migration efficiency of graph shard data, which can improve the real-time performance and query effect of graph data query, optimize user experience and improve graph modeling benefits.

[0052] Combine the following Figure 4 Other embodiments of the data migration method disclosed herein are described.

[0053] Figure 4 Flowcharts of other embodiments of the data migration method disclosed herein. Figure 4 As shown, the method of this embodiment includes: steps S402 to S408.

[0054] In step S402, the hash value of the identifier of each entity in the graph is determined.

[0055] Entities include, for example, data of nodes or edges of a graph. Hash the entities in the full graph data.

[0056] In step S404, the identifier of the graph slice data to which each entity belongs is determined according to the hash value of the identifier of each entity.

[0057] For example, the hash value is modulo the number of graph shard data N to determine the graph shard data to which each entity belongs.

[0058] In step S406, the hash value of the identifier of each graph fragment data is determined.

[0059] In step S408, the position of each graph slice data in the hash ring is determined according to the hash value of the identifier of each graph slice data, as the position of each allocated graph slice data in the hash ring.

[0060] The method of the above embodiment can shard the full amount of graph data and determine the position of each graph shard data in the hash ring.

[0061] The data migration device maintains a state machine, which is responsible for communicating with the graph engine service through a heartbeat mechanism. The data migration device can provide a registration function for the graph engine service, and can obtain the graph shard data information loaded by each graph engine service. The data migration device includes a consistent hashing module, which is responsible for mapping the graph data shards to different graph engine services, and maintaining a mapping table from graph data shards to graph engines. When the number of graph engine services M increases, the consistent hashing algorithm redetermines the mapping relationship between the graph data shards and the graph engine services, generates a new mapping table, and sends it to other modules of the graph engine service. After the graph engine service obtains the graph data shards from the data migration device to the mapping table, it loads the corresponding graph data shard data and goes online after loading is complete.

[0062] Combine the following Figure 5 Some embodiments of the query of graph shard data of the present disclosure are described.

[0063] Figure 5 Flowcharts of some other embodiments of the data migration method disclosed herein. Figure 5 As shown, the method of this embodiment includes: steps S502 to S510.

[0064] In step S502, when the graph shard data to be migrated is migrated from the original graph engine service to the newly added graph engine service, the data migration device sends a mapping table of the graph shard data and the graph engine service to the graph data query service.

[0065] In step S504, the graph data query service receives a data query request.

[0066] The data query request includes, for example, an entity to be queried, where the entity includes, for example, data of a node or edge of a graph.

[0067] In step S506, the graph data query service determines the identifier of the graph shard data to which the entity to be queried belongs according to the hash value of the identifier of the entity to be queried.

[0068] For example, the hash value is modulo the number of graph shard data N to determine the graph shard data to which the entity to be queried belongs.

[0069] In step S508, the graph data query service searches the mapping table for the corresponding graph engine service according to the identifier of the graph shard data to which the entity to be queried belongs, and uses the service as the target graph engine service.

[0070] In step S510, the graph data query service queries the target graph engine service for the entity to be queried.

[0071] The method of the above embodiment sends the mapping table to the data query service, and the data query service provides query services according to the new mapping table, thereby reducing the probability of query errors.

[0072] The present disclosure also provides a data migration device. Figure 6 Give a description.

[0073] Figure 6 FIG. 1 is a structural diagram of some embodiments of the data migration device disclosed in the present invention. Figure 6 As shown, the device 60 of this embodiment includes: a tag position determination module 610 , a migration information determination module 620 , and a migration indication module 630 .

[0074] The label position determination module 610 is used to determine the position of the label of the newly added graph engine service in the hash ring.

[0075] In some embodiments, there are multiple labels for the newly added graph engine service, and the label position determination module 610 is used to obtain multiple labels for the newly added graph engine service; determine the hash value of each label according to a preset hash function; and determine the position of each label in the hash ring according to the hash value of each label.

[0076] The migration information determination module 620 is used to determine the graph shard data to be migrated belonging to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each allocated graph shard data in the hash ring.

[0077] In some embodiments, the migration information determination module 620 is used to determine, in a preset order in the hash ring, the labels that are closest to the positions of the respective graph shard data in the hash ring, as the labels corresponding to the respective graph shard data; and according to the labels corresponding to the respective graph shard data and the graph engine services to which the respective labels belong, regenerate a mapping table of the graph shard data and the graph engine services, wherein the graph shard data to be migrated corresponds to the newly added graph engine services in the mapping table.

[0078] The migration instruction module 630 is used to send instruction information to the newly added graph engine service and the original graph engine service of the graph slice data to be migrated, so as to migrate the graph slice data to be migrated from the original graph engine service to the newly added graph engine service.

[0079] In some embodiments, the migration indication module 630 is used to send indication information to each graph engine service, wherein the indication information includes a mapping table; receive a loading completion message sent by a newly added graph engine service, wherein the loading completion message indicates that the newly added graph engine service completes the loading of the graph slice data to be migrated according to the mapping table; send a deletion indication to the original graph engine service; receive a deletion completion message sent by the original graph engine service, wherein the deletion completion message indicates that the original graph engine service completes the deletion of the graph slice data to be migrated according to the mapping table.

[0080] In some embodiments, the migration indication module 630 is also used to send a mapping table of graph shard data and graph engine service to the graph data query service when the graph shard data to be migrated is migrated from the original graph engine service to the newly added graph engine service, wherein the graph shard data to be migrated corresponds to the newly added graph engine service in the mapping table.

[0081] In some embodiments, the device 60 also includes: a data position determination module 640, which is used to determine the hash value of the identifier of each entity in the graph, wherein the entity includes the data of the node or the data of the edge of the graph; determine the identifier of the graph shard data to which each entity belongs based on the hash value of the identifier of each entity; determine the hash value of the identifier of each graph shard data; and determine the position of each graph shard data in the hash ring based on the hash value of the identifier of each graph shard data as the position of each allocated graph shard data in the hash ring.

[0082] In some embodiments, the number of tags for each graph engine service is determined based on the processing capability of the graph engine service, wherein the stronger the processing capability of the graph engine service, the greater the number of tags.

[0083] The data migration device in the embodiments of the present disclosure can be implemented by various computing devices or computer systems. Figure 7 as well as Figure 8 Give a description.

[0084] Figure 7 FIG. 1 is a structural diagram of some embodiments of the data migration device disclosed in the present invention. Figure 7 As shown, the device 70 of this embodiment includes: a memory 710 and a processor 720 coupled to the memory 710, and the processor 720 is configured to execute the data migration method in any of some embodiments of the present disclosure based on instructions stored in the memory 710.

[0085] The memory 710 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0086] Figure 8 FIG. 1 is a structural diagram of some other embodiments of the data migration device disclosed in the present invention. Figure 8As shown, the device 80 of this embodiment includes: a memory 810 and a processor 820, which are similar to the memory 710 and the processor 720 respectively. It can also include an input and output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850 and the memory 810 and the processor 820 can be connected, for example, through a bus 860. Among them, the input and output interface 830 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 840 provides a connection interface for various networked devices, for example, it can be connected to a database server or a cloud storage server. The storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0087] The present disclosure also provides a data migration system. Fig. 9 Give a description.

[0088] Fig. 9 FIG. 1 is a structural diagram of some embodiments of the data migration system disclosed in the present invention. Fig. 9 As shown, the system 9 of this embodiment includes: the data migration device 60 / 70 / 80 of any of the aforementioned embodiments, and multiple graph engine services 92.

[0089] When the graph engine service 92 is a newly added graph service engine or the original graph engine service of the graph slice data to be migrated, it is used to receive the instruction information sent by the data migration device to achieve the migration of the graph slice data to be migrated. The graph engine service 92 can be a server, PC or other equipment.

[0090] In some embodiments, the indication information includes a mapping table; when the graph engine service 92 is a newly added graph service engine, it is used to complete the loading of the graph shard data to be migrated according to the mapping table, and send a loading completion message to the data migration device 60 / 70 / 80; when the graph engine service 92 is the original graph service engine, it is used to receive the deletion indication sent by the data migration device 60 / 70 / 80, complete the deletion of the graph shard data to be migrated according to the mapping table, and send a deletion completion message to the data migration device 60 / 70 / 80.

[0091] In some embodiments, the system 9 further includes: a graph data query service 94 for receiving a mapping table sent by the migration device 60 / 70 / 80 and providing a query service according to the mapping table.

[0092] In some embodiments, the graph data query service 94 is used to receive a data query request, determine the hash value of the identifier of the entity to be queried, determine the identifier of the graph shard data to which the entity to be queried belongs based on the hash value of the identifier of the entity to be queried, and in the mapping table, find the corresponding graph engine service based on the identifier of the graph shard data to which the entity to be queried belongs, and query the target graph engine service for the entity to be queried as the target graph engine service; wherein the data query request includes: the entity to be queried, and the entity includes the data of the nodes or edges of the graph.

[0093] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0097] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A data migration method, comprising: Determine the position of the label of the newly added graph engine service in the hash ring; Determine the graph shard data to be migrated belonging to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each allocated graph shard data in the hash ring; Instruction information is sent to the newly added graph engine service and the original graph engine service of the graph slice data to be migrated, so as to migrate the graph slice data to be migrated from the original graph engine service to the newly added graph engine service.

2. The data migration method according to claim 1, wherein: There are multiple labels for the newly added graph engine service, and determining the position of the label of the newly added graph engine service in the hash ring includes: Get multiple tags of the newly added graph engine service; Determine the hash value of each tag according to the preset hash function; The position of each label in the hash ring is determined according to the hash value of each label.

3. The data migration method according to claim 1, wherein: The determining, according to the distance between the position of the label in the hash ring and the position of each allocated graph shard data in the hash ring, the graph shard data to be migrated belonging to the newly added graph engine service comprises: In the hash ring, according to a preset order, the label closest to the position of each graph shard data in the hash ring is determined in turn as the label corresponding to each graph shard data; According to the labels corresponding to each graph shard data and the graph engine services to which each label belongs, a mapping table of graph shard data and graph engine services is regenerated, wherein the graph shard data to be migrated corresponds to the newly added graph engine services in the mapping table.

4. The data migration method according to claim 3, wherein: The sending of instruction information to the newly added graph engine service and the original graph engine service of the graph slice data to be migrated, and migrating the graph slice data to be migrated from the original graph engine service to the newly added graph engine service includes: Sending indication information to each graph engine service, wherein the indication information includes the mapping table; Receive a loading completion message sent by the newly added graph engine service, wherein the loading completion message indicates that the newly added graph engine service completes loading of the graph slice data to be migrated according to the mapping table; Sending a deletion instruction to the original image engine service; Receive a deletion completion message sent by the original image engine service, wherein the deletion completion message indicates that the original image engine service completes deletion of the to-be-migrated image slice data according to the mapping table.

5. The data migration method according to claim 1, further comprising: When the graph shard data to be migrated is migrated from the original graph engine service to the newly added graph engine service, a mapping table of graph shard data and graph engine service is sent to the graph data query service, and the graph shard data to be migrated corresponds to the newly added graph engine service in the mapping table.

6. The data migration method according to claim 1, further comprising: Determine a hash value of an identifier of each entity in the graph, wherein the entity includes data of a node or data of an edge of the graph; Determine the identifier of the graph shard data to which each entity belongs according to the hash value of the identifier of each entity; Determine the hash value of the identifier of each graph shard data; According to the hash value of the identifier of each graph shard data, the position of each graph shard data in the hash ring is determined as the position of each allocated graph shard data in the hash ring.

7. The data migration method according to claim 1, wherein: The number of tags of each graph engine service is determined according to the processing capability of the graph engine service, wherein the stronger the processing capability of the graph engine service is, the greater the number of tags is.

8. The data migration method according to claim 5, further comprising: The graph data query service receives a data query request, wherein the data query request includes: an entity to be queried, the entity including data of a node or edge of the graph; The graph data query service determines a hash value of the identifier of the entity to be queried; The graph data query service determines the identifier of the graph shard data to which the entity to be queried belongs according to the hash value of the identifier of the entity to be queried; The graph data query service searches for the corresponding graph engine service in the mapping table according to the identifier of the graph shard data to which the entity to be queried belongs, as the target graph engine service; The graph data query service queries the target graph engine service for the entity to be queried.

9. A data migration device, comprising: The label position determination module is used to determine the position of the label of the newly added graph engine service in the hash ring; A migration information determination module, used to determine the graph shard data to be migrated belonging to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each allocated graph shard data in the hash ring; The migration instruction module is used to send instruction information to the newly added graph engine service and the original graph engine service of the graph slice data to be migrated, so as to migrate the graph slice data to be migrated from the original graph engine service to the newly added graph engine service.

10. A data migration device, comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the data migration method according to any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

12. A data migration system, comprising: The data migration device according to claim 9 or 10, and a plurality of graph engine services; Wherein, when the graph engine service is a newly added graph engine service or the original graph engine service of the graph slice data to be migrated, it is used to receive the indication information sent by the data migration device to realize the migration of the graph slice data to be migrated.

13. The data migration system according to claim 12, wherein: The indication information includes a mapping table; when the graph engine service is a newly added graph engine service, it is used to complete the loading of the graph slice data to be migrated according to the mapping table, and send a loading completion message to the data migration device; when the graph engine service is the original graph engine service, it is used to receive a deletion indication sent by the data migration device, complete the deletion of the graph slice data to be migrated according to the mapping table, and send a deletion completion message to the data migration device.

14. The data migration system according to claim 12, further comprising: The graph data query service is used to receive the mapping table sent by the data migration device and provide query services according to the mapping table.

15. The data migration system according to claim 14, wherein: The graph data query service is used to receive a data query request, determine the hash value of the identifier of the entity to be queried, determine the identifier of the graph shard data to which the entity to be queried belongs based on the hash value of the identifier of the entity to be queried, and in the mapping table, search for the corresponding graph engine service based on the identifier of the graph shard data to which the entity to be queried belongs, and query the target graph engine service for the entity to be queried as the target graph engine service; wherein the data query request includes: the entity to be queried, and the entity includes the data of the nodes or edges of the graph.

Images