Object data migration method, device, and equipment

Abstract

The present invention provides an object data migration method, device and equipment, the method comprising: in response to a first data migration request received from a management node of a distributed database to be migrated, obtaining a target node identifier for storing data to be migrated and metadata to be migrated in the first data migration request; determining differential metadata based on metadata pre-stored in a target node corresponding to the metadata to be migrated and the target node identifier; in response to target business migration data corresponding to the received differential metadata, storing the target business migration data in the target business node and storing the differential metadata in the target node respectively based on the differential metadata and the target node; and storing a mapping relationship between the target node identifier and the differential metadata in the target distributed database.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically to a method, device and equipment for object data migration. Background Art

[0002] With the development of technology and the expansion of business, enterprises may consider migrating data from the original database to a new database with more comprehensive functions and larger data storage capacity. In related technologies, the migration of two distributed databases usually requires the sending and receiving of data through the management nodes in the two databases. In this way, after the data determines the storage node, it is still necessary to transmit the data to the corresponding node, which is inefficient. Summary of the invention

[0003] In view of the above problems, the present invention provides a method, apparatus and device for object data migration.

[0004] According to a first aspect of the present invention, there is provided an object data migration method, wherein the object data includes metadata and business data, and the method is applied to a target distributed database, comprising: in response to a first data migration request received from a management node of a distributed database to be migrated, obtaining a target node identifier for storing the data to be migrated and the metadata to be migrated in the first data migration request; determining difference metadata based on metadata pre-stored in the target node corresponding to the metadata to be migrated and the target node identifier; in response to target business migration data corresponding to the received difference metadata, storing the target business migration data to the target business node and storing the difference metadata to the target node respectively based on the difference metadata and the target node; and storing a mapping relationship between the target node identifier and the difference metadata to the target distributed database.

[0005] According to an embodiment of the present invention, the above-mentioned target business migration data includes multiple sub-target business migration data, and the above-mentioned multiple sub-target business migration data are arranged in order. The above-mentioned storing the above-mentioned target business migration data in the above-mentioned target business node includes: allocating a first cache space based on the first sub-target business migration data in the above-mentioned target business migration data; when the storage space required for the current sub-target business migration data is less than the above-mentioned first cache space, writing the above-mentioned current sub-target business migration data into the above-mentioned first cache space; and writing the above-mentioned current sub-target business migration data into the above-mentioned target business node, and clearing the above-mentioned first cache space.

[0006] According to an embodiment of the present invention, the above method also includes: when the storage space required for the current sub-target business migration data is greater than the above-mentioned first storage memory, releasing the above-mentioned first storage memory; and allocating a second storage memory based on the above-mentioned current sub-target business migration data; and writing the above-mentioned current sub-target business migration data into the above-mentioned target business node based on the above-mentioned second storage memory.

[0007] According to an embodiment of the present invention, the above-mentioned difference metadata and the above-mentioned target business migration data are stored based on an asynchronous storage mechanism, and the above-mentioned method also includes: in the event that the storage of the above-mentioned difference metadata and / or the above-mentioned target business migration data fails, the difference metadata in the above-mentioned target node and the target business migration data of the above-mentioned target business node are rolled back.

[0008] According to an embodiment of the present invention, before storing the above-mentioned difference metadata and the above-mentioned target business migration data, the above-mentioned method also includes: using a preset data consistency verification algorithm to verify the above-mentioned target business migration data to obtain a verification result; in response to the above-mentioned verification result being passed, performing a storage operation.

[0009] According to a second aspect of the present invention, there is provided an object data migration method, which is applied to a management node of a distributed database to be migrated, wherein the distributed database to be migrated further comprises a plurality of storage nodes for storing metadata, and the distributed database to be migrated further stores a first mapping relationship between object data and the storage nodes, wherein the method comprises: in response to a received second data migration request, obtaining an identifier of the object data to be migrated and a target node identifier in the second data migration request; determining the metadata to be migrated based on the identifier of the object data to be migrated and the first mapping relationship; sending the target node identifier and the metadata to be migrated to the management node of the target distributed database; in response to the received differential metadata, determining target business migration data corresponding to the differential metadata based on the differential metadata; and sending the target business migration data to the target node.

[0010] According to an embodiment of the present invention, the above-mentioned sending of the target business migration data to the target node includes: using multiple threads to read the above-mentioned target business migration data to obtain multiple sub-target business migration data, and the above-mentioned multiple sub-target business migration data include a reading sequence identifier; based on the above-mentioned reading sequence identifier, sorting the above-mentioned sub-target business migration data to obtain sorted sub-target business migration data; and sending the above-mentioned sorted sub-target business migration data to the target node of the above-mentioned target distributed database in sequence.

[0011] The third aspect of the present invention provides an object data migration device, wherein the object data includes metadata and business data, and the device is installed in a target distributed database, and the device includes: an acquisition module, which is used to respond to a first data migration request received from a management node of a distributed database to be migrated, and obtain a target node identifier for storing the data to be migrated and the metadata to be migrated in the first data migration request; a determination module, which is used to determine difference metadata based on the metadata to be migrated and the metadata pre-stored in the target node corresponding to the target node identifier; a first storage module, which is used to respond to the target business migration data corresponding to the received difference metadata, and store the target business migration data to the target business node based on the difference metadata and the target node, and store the difference metadata to the target node; a second storage module, which is used to store the mapping relationship between the target node identifier and the difference metadata to the target distributed database.

[0012] According to an embodiment of the present invention, the above-mentioned target business migration data includes multiple sub-target business migration data, and the above-mentioned multiple sub-target business migration data are arranged in order, and the above-mentioned first storage module includes: a first allocation sub-module, used to allocate a first cache space based on the first sub-target business migration data in the above-mentioned target business migration data; a first writing sub-module, used to write the above-mentioned current sub-target business migration data into the above-mentioned first cache space when the storage space required for the current sub-target business migration data is less than the above-mentioned first cache space; and a first clearing sub-module, used to write the above-mentioned current sub-target business migration data into the target business node and clear the above-mentioned first cache space.

[0013] According to an embodiment of the present invention, the above-mentioned first storage module also includes: a release sub-module, which is used to release the above-mentioned first cache space when the storage space required for the current sub-target business migration data is greater than the above-mentioned first cache space; and a second allocation sub-module, which is used to allocate the second cache space based on the above-mentioned current sub-target business migration data; a second writing sub-module, which is used to write the above-mentioned current sub-target business migration data into the above-mentioned second cache space; and a second clearing sub-module, which is used to write the above-mentioned current sub-target business migration data into the target business node and clear the above-mentioned second cache space.

[0014] According to an embodiment of the present invention, the above-mentioned difference metadata and the above-mentioned target business migration data are stored based on an asynchronous storage mechanism, and the above-mentioned device also includes: a rollback module, which is used to roll back the difference metadata in the above-mentioned target node and the target business migration data of the above-mentioned target business node when the storage of the above-mentioned difference metadata and / or the above-mentioned target business migration data fails.

[0015] According to an embodiment of the present invention, the above-mentioned device also includes: a consistency verification module, which is used to verify the above-mentioned target business migration data using a preset data consistency verification algorithm to obtain a verification result; in response to the above-mentioned verification result being passed, performing a storage operation.

[0016] A fourth aspect of the present invention provides an object data migration device, which is installed on the management node of a distributed database to be migrated. The distributed database to be migrated also includes multiple storage nodes. The distributed database to be migrated also stores a first mapping relationship between object data and the storage nodes. The device includes: a second acquisition module, which is used to respond to a received second data migration request and acquire the object data identifier and target node identifier in the second data migration request; a metadata to be migrated determination module, which is used to determine the metadata to be migrated based on the object data identifier and the first mapping relationship; a first sending module, which is used to send the target node identifier and the metadata to be migrated to the management node of the target distributed database; a target business migration data determination module, which is used to respond to the received difference metadata and determine the target business migration data corresponding to the difference metadata based on the difference metadata; a second sending module, which is used to send the target business migration data to the target node.

[0017] According to an embodiment of the present invention, the above-mentioned second sending module includes: a reading sub-module, which is used to use multi-threading to read the above-mentioned target business migration data to obtain multiple sub-target business migration data, and the above-mentioned multiple sub-target business migration data include a reading sequence identifier; a sorting sub-module, which is used to sort the above-mentioned sub-target business migration data based on the above-mentioned reading sequence identifier to obtain sorted sub-target business migration data; and a sorting and sending sub-module, which is used to send the above-mentioned sorted sub-target business migration data to the target node of the above-mentioned target distributed database in sequence.

[0018] A fifth aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0019] The sixth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0020] The seventh aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0021] According to an embodiment of the present invention, through the connection between the target distributed database and the management node of the database to be migrated, the storage location of the migration data in the target distributed database is determined in the database to be migrated and sent directly to the corresponding target node for storage, which saves time and improves efficiency. At the same time, the incremental data in the migration data is determined through the differential metadata for incremental migration, which further reduces network consumption and reduces system load. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0023] Figure 1 An application scenario diagram of an object data migration method and device according to an embodiment of the present invention is shown;

[0024] Figure 2 A flowchart of an object data migration method according to an embodiment of the present invention is shown;

[0025] Figure 3 A flow chart of an object data migration method applied to a management node of a distributed database to be migrated according to an embodiment of the present invention is shown;

[0026] Figure 4 It shows a structural block diagram of an object data migration device according to an embodiment of the present invention;

[0027] Figure 5 It shows a structural block diagram of an object data migration device installed on a distributed database management node to be migrated according to an embodiment of the present invention;

[0028] Figure 6 A block diagram of an electronic device suitable for implementing an object data migration method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0033] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, invention and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0035] With the continuous development of technology and the expansion of business, enterprises may consider migrating data from existing databases to new databases with more comprehensive functions and larger data storage capacity. In related technologies, data is usually sent and received through the management nodes of two distributed databases. Once the data storage node is determined, the data still needs to be transmitted to the corresponding node, which is inefficient.

[0036] In view of this, an embodiment of the present invention provides an object data migration method, wherein the object data includes metadata and business data, and the method is applied to a target distributed database, and the method includes: in response to a first data migration request received from a management node of a distributed database to be migrated, obtaining a target node identifier for storing the data to be migrated and the metadata to be migrated in the first data migration request; determining difference metadata based on metadata pre-stored in the target node corresponding to the metadata to be migrated and the target node identifier; in response to target business migration data corresponding to the received difference metadata, storing the target business migration data in the target business node and storing the difference metadata in the target node respectively based on the difference metadata and the target node; and storing the mapping relationship between the target node identifier and the difference metadata in the target distributed database.

[0037] Figure 1 An application scenario diagram of an object data migration method and device according to an embodiment of the present invention is shown.

[0038] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a first server 105, and a second server 106. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, the first server 105, and the second server 106. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0039] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the first server 105 and / or the second server 106 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0041] The first server 105 and the second server 106 may be servers that provide various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103, or a storage server (only as an example) that provides storage services for distributed databases. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the object data migration method provided in the embodiment of the present invention can generally be executed by the first server 105 or the second server 106. Accordingly, the object data migration device provided in the embodiment of the present invention can generally be set in the first server 105. The object data migration method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the first server 105 or the second server 106 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the first server 105. Accordingly, the object data migration device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the first server 105 or the second server 106 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the first server 105 or the second server 106.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0044] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The object data migration method according to the embodiment of the present invention is described in detail.

[0045] Figure 2 A flow chart of an object data migration method according to an embodiment of the present invention is shown.

[0046] like Figure 2 As shown, the object data migration method of this embodiment includes operations S210 to S240.

[0047] It should be noted that this method can be applied to the target distributed database.

[0048] In operation S210, in response to a first data migration request received from a management node of a distributed database to be migrated, a target node identifier for storing data to be migrated and metadata to be migrated in the first data migration request are obtained.

[0049] According to an embodiment of the present invention, the above-mentioned object data includes metadata and business data. The aforementioned metadata can be data index data in the database to be migrated. The aforementioned data index data records the index and characteristic information of the corresponding business data. The aforementioned characteristic information can be, for example, data size, maximum value, minimum value, null value, data type, etc., and a unique business data can be determined based on the characteristic information.

[0050] According to an embodiment of the present invention, the above-mentioned distributed database to be migrated can be a multi-distributed database in which both data nodes and computing nodes are deployed on the server, and the above-mentioned target distributed database can be a cloud-based distributed database in which computing nodes, scheduling nodes, metadata storage nodes, and business data storage nodes are deployed separately and can all be deployed on the cloud.

[0051] According to an embodiment of the present invention, the online status of the distributed database to be migrated can be checked by a script. The script can be connected to the distributed database to be migrated and the target distributed database by passing in or pre-setting parameters, and obtain the online status of the distribution nodes of the data table on the distributed database to be migrated by passing in or pre-setting the data table to be migrated. Because there may be spare data storage nodes, as long as the online distribution nodes can form a complete data table, it is considered that the distributed database to be migrated meets the migration conditions. Among them, the above script can be deployed on a personal terminal device or on a server, and the present invention does not limit this.

[0052] According to an embodiment of the present invention, a script can also be used to check the online status of the nodes of the target distributed database to decide which nodes to write the migration data to. When the distributed database to be migrated meets the migration conditions and there are writable nodes in the target distributed database, the script constructs a corresponding migration command and sends it to the target distributed database.

[0053] According to an embodiment of the present invention, in order to maintain data consistency during object data migration, the distributed database to be migrated and the target distributed database can be locked. That is, except for the writing of migration data, only data reading operations can be performed during the table lock period, and data deletion, modification, addition and other operations cannot be performed.

[0054] According to an embodiment of the present invention, after receiving the migration command, the above-mentioned target distributed database starts the migration tool client of the corresponding node based on the target node identifier in the migration command, and constructs a migration command and sends it to the management node of the distributed database to be migrated. A migration server is deployed in the aforementioned management node to obtain the metadata and business data of the object data to be migrated, and send it to the corresponding target node in the target distributed database.

[0055] In operation S220 , differential metadata is determined based on the metadata to be migrated and metadata pre-stored in the target node corresponding to the target node identifier.

[0056] According to an embodiment of the present invention, the target node may store metadata from the last migration, and can determine the difference metadata that needs to be migrated this time based on the metadata to be migrated and the pre-stored metadata, and send the difference metadata to the distributed database to be migrated, so that the distributed database to be migrated can search for the corresponding business data.

[0057] According to an embodiment of the present invention, during the object data migration process, the object data migration may be terminated due to network problems or a need to terminate the object data migration. At this time, the metadata that has been migrated can be used as pre-stored metadata for the next data migration process.

[0058] According to the embodiment of the present invention, when metadata is stored in the target distributed database, business data corresponding to the metadata must also be stored in the target distributed database.

[0059] In operation S230, in response to the received target service migration data corresponding to the differential metadata, the target service migration data is stored in the target service node based on the differential metadata and the target node, and the differential metadata is stored in the target node.

[0060] According to an embodiment of the present invention, there may be a situation where the amount of difference metadata and target business migration data is large. The server cluster stored in the target distributed database can write the storage of difference metadata and target business migration data through two threads to improve migration efficiency. If both threads return success, the data migration can be considered successful.

[0061] According to an embodiment of the present invention, the storage location of the target service migration data in the target distributed database may be determined according to the difference metadata.

[0062] In operation S240, a mapping relationship between the target node identifier and the difference metadata is stored in a target distributed database.

[0063] According to an embodiment of the present invention, through the connection between the target distributed database and the management node of the database to be migrated, the storage location of the migration data in the target distributed database is determined in the database to be migrated and sent directly to the corresponding target node for storage, which saves time and improves efficiency. At the same time, the incremental data in the migration data is determined through the differential metadata for incremental migration, which further reduces network consumption and reduces system load.

[0064] According to an embodiment of the present invention, before storing the difference metadata and the target business migration data, the target business migration data may be verified using a preset data consistency verification algorithm to obtain a verification result; in response to the verification result being passed, a storage operation is performed.

[0065] According to an embodiment of the present invention, the preset data consistency check algorithm may be, for example, a CRC (Cyclic Redundancy Check) algorithm. Generally, a CRC32 algorithm is used to check the target service migration data.

[0066] According to an embodiment of the present invention, by performing a data consistency check before storing the difference metadata and the target business migration data, the integrity of the data during the data migration process can be ensured, the accuracy of the migrated data can be guaranteed, and the data quality of the migrated data can be ensured.

[0067] According to an embodiment of the present invention, the difference metadata and target business migration data are stored based on an asynchronous storage mechanism. The above method also includes: in the event that the storage of the difference metadata and / or the target business migration data fails, the difference metadata in the target node and the target business migration data of the target business node are rolled back.

[0068] According to an embodiment of the present invention, network communication and data storage are required when writing difference metadata and target business migration data. In order to reduce time consumption, they can be designed as asynchronous storage, that is, the writing of the above two data does not require the storage of target business migration data after the difference metadata is stored. Therefore, when the storage of any one of the data fails, the corresponding difference metadata or target business migration data needs to be rolled back to the data state before this migration.

[0069] According to an embodiment of the present invention, the above-mentioned target business migration data includes multiple sub-target business migration data, and the multiple sub-target business migration data are arranged in order. Storing the target business migration data in the target business node includes: allocating a first cache space based on the first sub-target business migration data in the target business migration data; when the storage space required for the current sub-target business migration data is less than the first cache space, writing the current sub-target business migration data into the first cache space; and writing the current sub-target business migration data into the target business node and clearing the first cache space.

[0070] According to an embodiment of the present invention, the above method also includes: when the storage space required for the current sub-target business migration data is greater than the first cache space, releasing the first cache space; and allocating a second cache space based on the current sub-target business migration data to write the current sub-target business migration data into the second cache space; and writing the current sub-target business migration data into the target business node and clearing the second cache space.

[0071] According to the embodiment of the present invention, when the amount of migration data is large, the amount of corresponding target service migration data is also large, so it is necessary to pre-allocate and reuse the cache space according to the sub-target service migration data.

[0072] Exemplarily, a first cache space is first allocated according to the first sub-target business migration data, and the first sub-target business migration data is written into the corresponding storage space of the target business node through the first cache space, and the first cache space is retained and not released. If the storage space required for subsequent sub-target business migration data is smaller than the first cache space, the data is written through the first cache space. If the storage space required for subsequent sub-target business migration data is larger than the first cache space, the first cache space is released, and the second cache space is reapplied for writing.

[0073] According to an embodiment of the present invention, a first cache space is allocated through first sub-target business migration data, and then it is determined whether the storage space required for the current sub-target business migration data is less than the first cache space to determine whether to write the current sub-target business migration data through the first cache space. By reusing the cache space, frequent cache allocation and release that affects the data migration speed can be avoided, thereby improving the data migration efficiency.

[0074] Figure 3 A flow chart of an object data migration method applied to a management node of a distributed database to be migrated according to an embodiment of the present invention is shown.

[0075] like Figure 3 As shown, the object data migration method applied to the management node of the distributed database to be migrated in this embodiment includes operations S310 to S350.

[0076] According to an embodiment of the present invention, the distributed database to be migrated further includes a plurality of storage nodes for storing metadata, and the distributed database to be migrated further stores a first mapping relationship between object data and the storage nodes.

[0077] In operation S310, in response to the received second data migration request, an identifier of the object data to be migrated and an identifier of the target node in the second data migration request are obtained.

[0078] In operation S320, metadata to be migrated is determined based on the identifier of the object data to be migrated and the first mapping relationship.

[0079] According to the embodiment of the present invention, the storage node storing the object data to be migrated can be found according to the above-mentioned identifier of the object data to be migrated, and the metadata to be migrated can be read therefrom.

[0080] In operation S330, a target node identifier and metadata to be migrated are sent to a management node of a target distributed database.

[0081] In operation S340 , in response to the received difference metadata, target business migration data corresponding to the difference metadata is determined based on the difference metadata.

[0082] In operation S350, target service migration data is sent to the target node.

[0083] According to an embodiment of the present invention, the above-mentioned sending of the target business migration data to the target node of the target distributed database includes: using multiple threads to read the target business migration data to obtain multiple sub-target business migration data, and the multiple sub-target business migration data include a reading sequence identifier; based on the reading sequence identifier, sorting the sub-target business migration data to obtain sorted sub-target business migration data; and sending the sorted sub-target business migration data to the target node of the target distributed database in sequence.

[0084] According to an embodiment of the present invention, after receiving the differential metadata, the distributed database to be migrated needs to find the corresponding target business migration data through the corresponding target business migration data location stored in the differential metadata, and read the corresponding target business migration data in a multi-threaded manner. Each thread has at least one queue, and the aforementioned queue is used to complete the storage and sending of data. When sending the target business migration data to the target distributed database, the data in the thread needs to be read in the reading order for sending.

[0085] According to the embodiment of the present invention, when reading the target service migration data, the reading may be performed according to the number of columns in the corresponding data table.

[0086] According to an embodiment of the present invention, target business migration data is read through multiple threads, sorted and put into a queue, and the data is sent sequentially through the queue in each thread, thereby reducing the additional overhead caused by locking between different threads and further reducing the system burden.

[0087] Based on the above object data migration method, the present invention also provides an object data migration device. Figure 4 The device is described in detail.

[0088] Figure 4 A structural block diagram of an object data migration device according to an embodiment of the present invention is shown.

[0089] like Figure 4 As shown, the object data migration device 400 of this embodiment includes an acquisition module 410 , a determination module 420 , a first storage module 430 and a second storage module 440 .

[0090] The acquisition module 410 is used to respond to the first data migration request received from the management node of the distributed database to be migrated, and obtain the target node identifier for storing the data to be migrated and the metadata to be migrated in the first data migration request. In one embodiment, the acquisition module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0091] The determination module 420 is used to determine the difference metadata based on the metadata to be migrated and the metadata pre-stored in the target node corresponding to the target node identifier. In one embodiment, the determination module 420 can be used to perform the operation S220 described above, which will not be described in detail here.

[0092] The first storage module 430 is used to respond to the target business migration data corresponding to the received difference metadata, store the target business migration data to the target business node based on the difference metadata and the target node, and store the difference metadata to the target node. In one embodiment, the first storage module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0093] The second storage module 440 is used to store the mapping relationship between the target node identifier and the difference metadata in the target distributed database. In one embodiment, the second storage module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0094] According to an embodiment of the present invention, the above-mentioned target business migration data includes multiple sub-target business migration data, and the above-mentioned multiple sub-target business migration data are arranged in order, and the above-mentioned first storage module includes: a first allocation sub-module, used to allocate a first cache space based on the first sub-target business migration data in the above-mentioned target business migration data; a first writing sub-module, used to write the above-mentioned current sub-target business migration data into the above-mentioned first cache space when the storage space required for the current sub-target business migration data is less than the above-mentioned first cache space; and a first clearing sub-module, used to write the above-mentioned current sub-target business migration data into the target business node and clear the above-mentioned first cache space.

[0095] According to an embodiment of the present invention, the above-mentioned first storage module also includes: a release sub-module, which is used to release the above-mentioned first cache space when the storage space required for the current sub-target business migration data is greater than the above-mentioned first cache space; and a second allocation sub-module, which is used to allocate the second cache space based on the above-mentioned current sub-target business migration data; a second writing sub-module, which is used to write the above-mentioned current sub-target business migration data into the above-mentioned second cache space; and a second clearing sub-module, which is used to write the above-mentioned current sub-target business migration data into the target business node and clear the above-mentioned second cache space.

[0096] According to an embodiment of the present invention, the above-mentioned difference metadata and the above-mentioned target business migration data are stored based on an asynchronous storage mechanism, and the above-mentioned device also includes: a rollback module, which is used to roll back the difference metadata in the above-mentioned target node and the target business migration data of the above-mentioned target business node when the storage of the above-mentioned difference metadata and / or the above-mentioned target business migration data fails.

[0097] According to an embodiment of the present invention, the above-mentioned device also includes: a consistency verification module, which is used to verify the above-mentioned target business migration data using a preset data consistency verification algorithm to obtain a verification result; in response to the above-mentioned verification result being passed, performing a storage operation.

[0098] According to an embodiment of the present invention, any multiple modules among the acquisition module 410, the determination module 420, the first storage module 430, and the second storage module 440 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 410, the determination module 420, the first storage module 430, and the second storage module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 410, the determination module 420, the first storage module 430, and the second storage module 440 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0099] Figure 5 A structural block diagram of an object data migration device installed on a distributed database management node to be migrated according to an embodiment of the present invention is shown.

[0100] like Figure 5 As shown, the object data migration device 500 of this embodiment includes a second acquisition module 510 , a to-be-migrated metadata determination module 520 , a first sending module 530 , a target business migration data determination module 540 and a second sending module 550 .

[0101] The second acquisition module 510 is used to obtain the identifier of the object data to be migrated and the target node identifier in the second data migration request in response to the received second data migration request. In one embodiment, the second acquisition module 510 can be used to perform the operation S310 described above, which will not be described in detail here.

[0102] The metadata to be migrated determining module 520 is used to determine the metadata to be migrated based on the object data identifier to be migrated and the first mapping relationship. In one embodiment, the metadata to be migrated determining module 520 can be used to perform the operation S320 described above, which will not be described in detail here.

[0103] The first sending module 530 is used to send the target node identifier and the metadata to be migrated to the management node of the target distributed database. In one embodiment, the first sending module 530 can be used to perform the operation S330 described above, which will not be described in detail here.

[0104] The target business migration data determination module 540 is used to determine the target business migration data corresponding to the difference metadata based on the difference metadata in response to the received difference metadata. In one embodiment, the target business migration data determination module 540 can be used to perform the operation S340 described above, which will not be repeated here.

[0105] The second sending module 550 is used to send the target service migration data to the target node. In one embodiment, the second sending module 550 can be used to perform the operation S350 described above, which will not be described in detail here.

[0106] According to an embodiment of the present invention, the above-mentioned second sending module includes: a reading sub-module, which is used to use multi-threading to read the above-mentioned target business migration data to obtain multiple sub-target business migration data, and the above-mentioned multiple sub-target business migration data include a reading sequence identifier; a sorting sub-module, which is used to sort the above-mentioned sub-target business migration data based on the above-mentioned reading sequence identifier to obtain sorted sub-target business migration data; and a sorting and sending sub-module, which is used to send the above-mentioned sorted sub-target business migration data to the target node of the above-mentioned target distributed database in sequence.

[0107] According to an embodiment of the present invention, any multiple modules among the second acquisition module 510, the metadata determination module 520 to be migrated, the first sending module 530, the target business migration data determination module 540, and the second sending module 550 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the second acquisition module 510, the metadata determination module 520 to be migrated, the first sending module 530, the target business migration data determination module 540, and the second sending module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the second acquisition module 510, the metadata to be migrated determination module 520, the first sending module 530, the target business migration data determination module 540 and the second sending module 550 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0108] Figure 6A block diagram of an electronic device suitable for implementing an object data migration method according to an embodiment of the present invention is shown.

[0109] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0110] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0111] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0112] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0113] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0114] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the object data migration method provided by the embodiment of the present invention.

[0115] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0116] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0117] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0118] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0121] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for migrating object data, characterized in that: The object data includes metadata and business data. The method is applied to a target distributed database and includes: In response to a first data migration request received from a management node of a distributed database to be migrated, obtaining a target node identifier for storing the data to be migrated and metadata to be migrated in the first data migration request; Determine difference metadata based on the metadata to be migrated and metadata pre-stored in the target node corresponding to the target node identifier; In response to the received target service migration data corresponding to the differential metadata, based on the differential metadata and the target node, respectively storing the target service migration data to the target service node, and storing the differential metadata to the target node; Storing the mapping relationship between the target node identifier and the difference metadata in the target distributed database; The target service migration data includes a plurality of sub-target service migration data, the plurality of sub-target service migration data are arranged in order, and storing the target service migration data in the target service node includes: Allocating a first cache space based on first sub-target business migration data in the target business migration data; When the storage space required for the current sub-target service migration data is smaller than the first cache space, writing the current sub-target service migration data into the first cache space; and Writing the current sub-target service migration data into the target service node and clearing the first cache space; In a case where the storage space required by the current sub-target service migration data is larger than the first cache space, releasing the first cache space; and Allocating a second cache space based on the current sub-target service migration data; Writing the current sub-target service migration data into the second cache space; and The current sub-target service migration data is written into the target service node, and the second cache space is cleared.

2. The method according to claim 1, characterized in that The difference metadata and the target service migration data are stored based on an asynchronous storage mechanism, and the method further includes: In the case that the storage of the difference metadata and / or the target service migration data fails, the difference metadata in the target node and the target service migration data of the target service node are rolled back.

3. The method according to claim 1, characterized in that The target service migration data is stored in the target service node based on the difference metadata and the target node respectively. Before the difference metadata is stored in the target node, the method further includes: Using a preset data consistency verification algorithm to verify the target business migration data to obtain a verification result; In response to the verification result being passed, a storage operation is performed.

4. A method for object data migration, characterized in that: The method is applied to a management node of a distributed database to be migrated, the distributed database to be migrated further includes a plurality of storage nodes for storing metadata, the distributed database to be migrated further stores a first mapping relationship between object data and the storage nodes, and the method includes: In response to the received second data migration request, obtaining an identifier of the object data to be migrated and an identifier of the target node in the second data migration request; Determine the metadata to be migrated based on the data identifier of the object to be migrated and the first mapping relationship; Sending the target node identifier and the metadata to be migrated to the management node of the target distributed database; In response to the received difference metadata, determining target business migration data corresponding to the difference metadata based on the difference metadata; The target service migration data is sent to the target node.

5. The method according to claim 4, characterized in that The sending the target service migration data to the target node includes: Using multiple threads to read the target business migration data to obtain a plurality of sub-target business migration data, wherein the plurality of sub-target business migration data includes a reading sequence identifier; Based on the read sequence identifier, sorting the sub-target business migration data to obtain sorted sub-target business migration data; The sorted sub-target business migration data are sent to the target node of the target distributed database in sequence.

6. An object data migration device, characterized in that: The object data includes metadata and business data. The device is installed in a target distributed database. The device includes: an acquisition module, configured to, in response to a first data migration request received from a management node of a distributed database to be migrated, acquire a target node identifier for storing data to be migrated and metadata to be migrated in the first data migration request; A determination module, configured to determine difference metadata based on the metadata to be migrated and metadata pre-stored in the target node corresponding to the target node identifier; A first storage module, configured to respond to the target business migration data corresponding to the received difference metadata, store the target business migration data in the target business node based on the difference metadata and the target node, and store the difference metadata in the target node; A second storage module, used for storing the mapping relationship between the target node identifier and the difference metadata in the target distributed database; The target service migration data includes a plurality of sub-target service migration data, the plurality of sub-target service migration data are arranged in order, and the first storage module includes: A first allocation submodule, configured to allocate a first cache space based on first sub-target business migration data in the target business migration data; A first writing submodule, configured to write the current sub-target service migration data into the first cache space when the storage space required by the current sub-target service migration data is smaller than the first cache space; A first clearing submodule, configured to write the current sub-target service migration data into a target service node and clear the first cache space; The first storage module also includes: A release submodule, configured to release the first cache space when the storage space required for the current sub-target business migration data is greater than the first cache space; A second allocation submodule is used to allocate a second cache space based on the current sub-target business migration data; a second writing submodule is used to write the current sub-target business migration data into the second cache space; and a second clearing submodule is used to write the current sub-target business migration data into the target business node and clear the second cache space.

7. An object data migration device, characterized in that: The device is installed on a management node of a distributed database to be migrated, the distributed database to be migrated further includes a plurality of storage nodes, the distributed database to be migrated further stores a first mapping relationship between object data and the storage nodes, and the device includes: A second acquisition module, configured to acquire, in response to the received second data migration request, an identifier of the object data to be migrated and an identifier of the target node in the second data migration request; A metadata to be migrated determining module, configured to determine the metadata to be migrated based on the object data identifier to be migrated and the first mapping relationship; A first sending module, configured to send the target node identifier and the metadata to be migrated to a management node of a target distributed database; a target business migration data determination module, configured to determine, in response to the received difference metadata, target business migration data corresponding to the difference metadata based on the difference metadata; The second sending module is used to send the target service migration data to the target node.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 5.

Images