Data migration method and device, electronic equipment and storage medium
By establishing a relational mapping dictionary between ES and CK, generating DDL statements to create the CK table, and using Java reflection to generate DML statements, the problem of time-consuming manual operations in the data migration from ES to CK is solved, realizing an automated and efficient data migration process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING CHANGYANG TECH CO LTD
- Filing Date
- 2022-11-09
- Publication Date
- 2026-05-15
AI Technical Summary
In existing technologies, the data migration process from ElasticSearch (ES) to Clickhouse (CK) requires a lot of manual operation, which is labor-intensive and inefficient.
By establishing a relationship mapping dictionary between the index to be migrated in Elasticsearch and the CK table, DDL statements are generated to create the CK table. Then, Java reflection is used to obtain the field information of the data entities and generate DML statements to achieve automated data migration.
It enables automated migration of batch data entities from Elasticsearch to Core Query (CK), improving data migration efficiency, reducing manual operations, and providing visualized migration process monitoring and progress feedback.
Smart Images
Figure CN115587085B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data migration technology, and in particular to a data migration method, apparatus, electronic device and storage medium. Background Technology
[0002] With the advent of the big data era, the diversification and complexity of information are increasing rapidly, requiring different storage strategies for log-type data during product development. The selection of such metadata storage components is vast. The Elasticsearch (ES) storage method initially designed for product updates and iterations, coupled with a surge in production data volume, necessitates a more suitable storage component like Clickhouse (CK) for reasons of data accuracy, product stability, and subsequent component maintenance. Therefore, it is necessary to migrate data from ES to CK.
[0003] Currently, the method for migrating data from Elasticsearch (ES) to Cache (CK) involves extracting data from ES, generating CSV files, and then manually inserting these CSV files into CK in batches. However, manually migrating large amounts of data files manually is extremely time-consuming. Therefore, there is an urgent need for an automated data migration method. Summary of the Invention
[0004] This invention provides a data migration method, apparatus, electronic device, and storage medium that can realize automatic data migration from ES to CK.
[0005] In a first aspect, embodiments of the present invention provide a data migration method, comprising:
[0006] Based on the index to be migrated in ES, establish a relationship mapping dictionary between the index to be migrated and the CK table;
[0007] Generate DDL statements for the corresponding data entities based on the relation mapping dictionary;
[0008] Execute the DDL statement to create a CK table in CK corresponding to the index to be migrated;
[0009] Based on the index to be migrated, a batch of data entities are read from ES into memory. The field information of the data entities is obtained using the Java reflection mechanism, and DML statements for inserting data into CK are generated based on the field information.
[0010] Execute the DML statement to migrate the bulk data entities read from ES to the corresponding CK table in CK.
[0011] In one possible implementation, establishing the relationship mapping dictionary between the index to be migrated and the CK table includes:
[0012] Annotate and mark the data entities to be migrated in ES; the annotation and mark are custom CK table information;
[0013] The migration index corresponding to the data entity to be migrated is matched one by one with the CK table information marked by annotations to form the relation mapping dictionary.
[0014] In one possible implementation, prior to reading the bulk data entities from the Elasticsearch (ES) into memory, the following steps are also included:
[0015] The relation mapping dictionary is loaded into the front end, and the target index for the required migration input by the user is obtained, so as to execute the reading of batch data entities from ES into memory based on the target index.
[0016] In one possible implementation, after creating the CK table in CK corresponding to the index to be migrated, the method further includes:
[0017] The successfully created CK table is marked in the relational mapping dictionary loaded into the front end;
[0018] The step of obtaining the target index for migration input by the user includes: obtaining the target index for migration selected by the user from the indexes to be migrated corresponding to the successfully created CK table.
[0019] In one possible implementation, the annotation tags include: CK table names for marking data entities, CK table column names for marking attribute columns of data entities, and CK table primary key column names for marking primary key attributes of data entities.
[0020] In one possible implementation, the step of using Java reflection to obtain field information of the data entity, and generating a DML statement for inserting data into CK based on the field information, includes:
[0021] Use Java reflection to retrieve the first attribute value of an attribute column of a data entity and the second attribute value of the primary key attribute of the data entity.
[0022] Based on the annotation tags, the mapping relationship between the first attribute value and the column name of the CK table is determined, and the mapping relationship between the second attribute value and the primary key column name of the CK table is determined. DML statements are generated according to the determined mapping relationship.
[0023] In one possible implementation, the method further includes: obtaining relevant information about the migration process and feeding back the relevant information about the migration process to the front end; the relevant information about the migration process includes at least: migration progress and / or migration success rate.
[0024] Secondly, embodiments of the present invention also provide a data migration apparatus, comprising:
[0025] A building unit is used to build a relationship mapping dictionary between the index to be migrated and the CK table based on the index to be migrated in ES;
[0026] The DDL generation unit is used to generate DDL statements for corresponding data entities based on the relation mapping dictionary.
[0027] A creation unit is used to execute the DDL statement to create a CK table in CK corresponding to the index to be migrated;
[0028] The DML generation unit is used to read batch data entities from ES into memory based on the index to be migrated, obtain field information of the data entities using Java reflection mechanism, and generate DML statements for inserting data into CK based on the field information.
[0029] The migration unit is used to execute the DML statements to migrate the batch data entities read from ES to the corresponding CK table in CK.
[0030] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of this specification.
[0031] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of this specification.
[0032] This invention provides a data migration method, apparatus, electronic device, and storage medium. Since the data storage methods in Elasticsearch (ES) and Controller (CK) differ, a relationship mapping dictionary is established between the index to be migrated in ES and the CK table to determine the CK table that needs to be created. DDL statements are generated to create the CK table. After the CK table is created, the batch of data entities to be migrated are read into memory based on the index to be migrated. Java reflection is used to obtain the field information of the data entities to determine the mapping location of the field information in CK, and then DML statements are generated to realize the migration of the batch of data entities to CK. Therefore, this solution can achieve the migration of batch data entities from ES to CK. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0034] Figure 1 This is a flowchart of a data migration method provided in an embodiment of the present invention;
[0035] Figure 2 This is a hardware architecture diagram of an electronic device provided in an embodiment of the present invention;
[0036] Figure 3 This is a structural diagram of a data migration device provided in an embodiment of the present invention;
[0037] Figure 4 This is a structural diagram of another data migration device provided in an embodiment of the present invention. Detailed Implementation
[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0039] Please refer to Figure 1 This invention provides a data migration method, which includes:
[0040] Step 100: Based on the index to be migrated in ES, establish a relationship mapping dictionary between the index to be migrated and the CK table;
[0041] Step 102: Generate DDL statements for the corresponding data entities based on the relation mapping dictionary;
[0042] Step 104: Execute the DDL statement to create a CK table in CK corresponding to the index to be migrated;
[0043] Step 106: Based on the index to be migrated, read a batch of data entities from ES into memory, use Java reflection mechanism to obtain the field information of the data entities, and generate DML statements to insert data into CK according to the field information;
[0044] Step 108: Execute the DML statement to migrate the bulk data entities read from ES to the corresponding CK table in CK.
[0045] In this embodiment of the invention, the data storage methods in ES and CK differ. A relationship mapping dictionary is established between the index to be migrated in ES and the CK table to determine the CK table that needs to be created. DDL statements are generated to create the CK table. After the CK table is created, the batch of data entities to be migrated are read into memory based on the index to be migrated. Java reflection is used to obtain the field information of the data entities to determine the mapping location of the field information in CK, and then DML statements are generated to realize the migration of the batch of data entities to CK. Therefore, this solution can achieve the migration of batch data entities from ES to CK.
[0046] The following description Figure 1 The execution method for each step is shown.
[0047] First, the following steps will be explained: Step 100, “Based on the index to be migrated in ES, establish a relational mapping dictionary between the index to be migrated and the CK table”; Step 102, “Generate DDL statements for corresponding data entities according to the relational mapping dictionary”; and Step 104, “Execute the DDL statements to create a CK table in CK corresponding to the index to be migrated”.
[0048] The Elasticsearch (ES) repository stores a large amount of ES data, from which the required ES data can be located using ES indexes. The index to be migrated can be determined by the ES data to be synchronized, or it can be determined by the user through input commands.
[0049] To migrate ES data to CK, preprocessing based on the data storage characteristics of the CK library is required. Since the CK library stores data using CK tables, this embodiment of the invention requires the creation of a CK table to receive the data to be migrated. The CK table information can be determined by the data entity to be migrated or it can be customized. These two methods are described below.
[0050] Method 1: The CK table information is determined by the data entity to be migrated.
[0051] In one embodiment of the present invention, when the CK table information is determined by the data entity to be migrated, the method of establishing the relationship mapping dictionary between the index to be migrated and the CK table may include:
[0052] Use the class name of the data entity corresponding to the index to be migrated as the table name of the CK table;
[0053] Use the attribute column names of the data entities corresponding to the index to be migrated as the column names of the CK table;
[0054] Use the primary key attribute name of the data entity corresponding to the index to be migrated as the primary key column name of the CK table;
[0055] The CK table, formed based on the table name, column names, and primary key column names of the CK table, is matched one-to-one with the corresponding indexes to be migrated to obtain a relational mapping dictionary.
[0056] For example, the resulting relational mapping dictionary is shown in Table 1 below:
[0057] Table 1
[0058] ES_index_name CK_table_name es_index1 ck_index1 es_index2 ck_index2 es_index3 ck_index3
[0059] Method 2: The CK table information is implemented using a custom method.
[0060] In another embodiment of the present invention, when the CK table information is implemented in a custom manner, the method for establishing the relationship mapping dictionary between the index to be migrated and the CK table may include:
[0061] Annotate and mark the data entities to be migrated in ES; the annotation and mark are custom CK table information;
[0062] The migration index corresponding to the data entity to be migrated is matched one by one with the CK table information marked by annotations to form the relation mapping dictionary.
[0063] Annotation tags may include:
[0064] @ClickHouseTable: Used to mark the CK table name on a data entity;
[0065] @ClickHouseColumn: Used to mark the name of the CK table column on the attribute column of a data entity;
[0066] @ClickHouseID: The name of the primary key column of the CK table used to mark the primary key attribute of a data entity.
[0067] These annotations constitute the CK table information. For example, an Elasticsearch data entity can be annotated as follows:
[0068] @ClickHouseTable(name="isg informacollect log")
[0069] public class IndustrialInformation extends ClickHouseBaseEntity imp
[0070] / / id identifier
[0071] @ClickHouseID
[0072] @ClickHouseColumn(name="id")
[0073] private Long id;
[0074] @ClickHousecolumn(name="isgrd")
[0075] private String isgId;
[0076] @ClickHouseColumn(name="cmdId")
[0077] Under this second method, the resulting relation mapping dictionary is shown in Table 2 below:
[0078] Table 2
[0079] ES_index_name CK_table_name es_index1 ck_table 1 es_index2 ck_table 2 es_index3 ck_table 3
[0080] As can be seen, using annotations to customize CK table information makes it easier for users to migrate ES data to a custom CK table, thus improving the user experience.
[0081] Furthermore, to improve the user experience, the established relational mapping dictionary can be loaded onto the front end to enable visualization of the dictionary, thereby allowing the data migration process to be understood through visualization.
[0082] In this embodiment of the invention, once the relationship mapping dictionary between the index to be migrated and the CK table is established, it indicates that the data of the index to be migrated has been determined to be migrated to the corresponding CK table in CK. At this time, it is necessary to create a CK table in CK that has a relationship mapping with the index to be migrated. Specifically, it is necessary to generate DDL statements to create the CK table by executing the DDL statements.
[0083] Regardless of whether Method 1 or Method 2 is used, once the relationship mapping dictionary between the index to be migrated and the CK table is established, it indicates that the CK table information is determined, and DDL statements can be generated based on the CK table information. Taking Method 2 as an example, DDL statements for splitting into CK can be generated through annotations, the name of the CK table to be created can be obtained through annotations on data entities, the column names of the CK table can be obtained through attribute column annotations on data entities, and the primary key column name of the CK table can be obtained through primary key attribute annotations on data entities.
[0084] For example, you can create the following DDL statement:
[0085]
[0086] COMMENT 'Host Guardian Information Collection Log';
[0087] After the DDL statement is generated, the generated DDL can be executed. During execution, each CK table is created sequentially in the CK. Due to potential compatibility issues or other problems during the creation of CK tables, not all CK tables can be created successfully. To allow users to intuitively and visually determine which CK tables have been created successfully, one embodiment of the present invention may further include: marking the successfully created CK tables in the relational mapping dictionary loaded to the front end.
[0088] This marking method can be such that Table 2, initially loaded into the front end, also includes a marking column, as shown in Table 3:
[0089] Table 3
[0090] ES_index_name CK_table_name is_success es_index1 ck_table 1 false es_index2 ck_table 2 false es_index3 ck_table 3 false
[0091] Each time a CK table is successfully created, the corresponding false value for that CK table is marked as true. Users can then determine whether a CK table has been successfully created based on this marker column in the relational mapping dictionary.
[0092] Next, we will explain step 106, "reading a batch of data entities from ES into memory based on the index to be migrated, using Java reflection to obtain the field information of the data entities, and generating a DML statement to insert data into CK based on the field information" and step 108, "executing the DML statement to migrate the batch of data entities read from ES to the corresponding CK table in CK".
[0093] This step can achieve data migration using the index to be migrated in step 100. However, during the creation of the CK table, some CK tables may fail to be created successfully. Therefore, the index to be migrated corresponding to the CK tables that failed to be created cannot be migrated. Based on this, in one embodiment of the present invention, before step 106, the following steps may be included: loading the relation mapping dictionary into the front end, obtaining the target index to be migrated input by the user, and performing the reading of batch data entities from ES into memory based on the target index (step 106).
[0094] Furthermore, during the creation of the CK table, since the successfully created CK table can be marked in the relational mapping dictionary loaded to the front end, the user can select the target index to be migrated based on the relational mapping dictionary. Specifically, the target index to be migrated is obtained from the index to be migrated corresponding to the successfully created CK table. In other words, the target index is selected by the user in the successfully created CK table.
[0095] In this embodiment of the invention, step 106 may include: obtaining the first attribute value on the attribute column of the data entity using Java reflection, obtaining the second attribute value of the primary key attribute of the data entity; determining the mapping relationship between the first attribute value and the column name of the CK table based on the annotation mark, and determining the mapping relationship between the second attribute value and the primary key column name of the CK table; and generating a DML statement according to the determined mapping relationship. The mapping relationship is used to indicate the insertion position when the attribute value is migrated to CK.
[0096] Java reflection refers to the ability, during program execution, to construct objects of any class, understand the class of any object, learn about the member variables and methods of any class, and invoke the attributes and methods of any object. This ability to dynamically obtain program information and dynamically invoke objects is called Java's reflection mechanism. Using Java reflection, one can obtain field information of the data to be migrated, and then use this field information to determine the DML statements to insert the data.
[0097] Furthermore, during the data migration process, relevant information about the migration progress can be obtained and then fed back to the front end. For example, relevant information can be identified based on a relational mapping dictionary. This migration process information includes at least: migration progress and / or migration success rate. This allows users to easily monitor the entire migration process.
[0098] In this embodiment of the invention, not only can it replace manual operation to automatically migrate Elasticsearch (ES) data to Continuous Tracking (CK), but it can also identify successfully created CK tables. Users can selectively perform migration of data corresponding to which indexes to be migrated through a web interface. Furthermore, the entire migration process is displayed visually, allowing for clear identification of data anomalies. This saves time and effort, is more user-friendly, and allows for control over migration strategies and progress.
[0099] like Figure 2 , Figure 3 As shown, this embodiment of the invention provides a data migration device. The device embodiment can be implemented through software, hardware, or a combination of both. From a hardware perspective, as... Figure 2 The diagram shown is a hardware architecture diagram of an electronic device containing a data migration device according to an embodiment of the present invention. (Except for...) Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 3As shown, a device in a logical sense is formed by the CPU of its electronic device reading the corresponding computer program from non-volatile memory into memory for execution. This embodiment provides a data migration device, including:
[0100] Establishment unit 301 is used to establish a relationship mapping dictionary between the index to be migrated and the CK table based on the index to be migrated in ES;
[0101] DDL generation unit 302 is used to generate DDL statements for corresponding data entities based on the relation mapping dictionary;
[0102] Creation unit 303 is used to execute the DDL statement to create a CK table in CK corresponding to the index to be migrated;
[0103] DML generation unit 304 is used to read batch data entities from ES into memory based on the index to be migrated, obtain field information of the data entities using Java reflection mechanism, and generate DML statements for inserting data into CK based on the field information.
[0104] Migration unit 305 is used to execute the DML statement to migrate the batch data entities read from ES to the corresponding CK table in CK.
[0105] In one embodiment of the present invention, the establishment unit 301 is specifically used to: annotate and mark the data entities to be migrated in ES; the annotation and mark are custom CK table information; and match the index to be migrated corresponding to the data entities to be migrated with the CK table information of the annotation and mark one by one to form the relation mapping dictionary.
[0106] In one embodiment of the present invention, please refer to Figure 4 The data migration apparatus may further include:
[0107] The interaction unit 306 is used to load the relation mapping dictionary into the front end, obtain the target index of the migration required by the user input, and trigger the DML generation unit to perform corresponding operations based on the target index.
[0108] In one embodiment of the present invention, the interaction unit is further configured to mark the relational mapping dictionary loaded to the front end based on the successfully created CK table;
[0109] When the interaction unit executes the process of obtaining the target index for migration from the user's input, it is specifically used to: obtain the target index for migration selected by the user from the indexes to be migrated corresponding to the successfully created CK table.
[0110] In one embodiment of the present invention, the annotation markers include: CK table names for marking data entities, CK table column names for marking attribute columns of data entities, and CK table primary key column names for marking primary key attributes of data entities.
[0111] In one embodiment of the present invention, the DML generation unit is specifically used to: obtain the first attribute value on the attribute column of the data entity using the Java reflection mechanism, obtain the second attribute value of the primary key attribute of the data entity; determine the mapping relationship between the first attribute value and the column name of the CK table based on the annotation mark, and determine the mapping relationship between the second attribute value and the primary key column name of the CK table, and generate DML statements according to the determined mapping relationship.
[0112] In one embodiment of the present invention, the interaction unit is further configured to obtain relevant information about the migration process and feed back the relevant information about the migration process to the front end; the relevant information about the migration process includes at least: migration progress and / or migration success rate.
[0113] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on a data migration apparatus. In other embodiments of the present invention, a data migration apparatus may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0114] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.
[0115] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a data migration method according to any embodiment of this invention.
[0116] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform a data migration method according to any embodiment of this invention.
[0117] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.
[0118] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0119] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.
[0120] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0121] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.
[0122] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0123] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data migration method, characterized in that, include: Based on the index to be migrated in Elasticsearch, the data entities to be migrated in Elasticsearch are annotated and marked; the annotation marks constitute custom CK table information; The migration index corresponding to the data entity to be migrated is matched one by one with the CK table information marked by the annotation to establish a relationship mapping dictionary between the migration index and the CK table; the annotation includes: the CK table name used to mark the data entity, the CK table column name used to mark the attribute column of the data entity, and the CK table primary key column name used to mark the primary key attribute of the data entity. Generate DDL statements for the corresponding data entities based on the annotation tags in the relation mapping dictionary; Execute the DDL statement to create a CK table in CK corresponding to the index to be migrated; Based on the index to be migrated, a batch of data entities are read from ES into memory. The field information of the data entities is obtained using the Java reflection mechanism, and DML statements for inserting data into CK are generated based on the field information. Execute the DML statement to migrate the bulk data entities read from ES to the corresponding CK table in CK; Before reading the batch of data entities from ES into memory, the method further includes: loading the relation mapping dictionary into the front end, obtaining the target index of the desired migration input by the user, and performing the reading of the batch of data entities from ES into memory based on the target index.
2. The data migration method according to claim 1, characterized in that, After creating the CK table in CK corresponding to the index to be migrated, the process further includes: The successfully created CK table is marked in the relational mapping dictionary loaded into the front end; The step of obtaining the target index for migration input by the user includes: obtaining the target index for migration selected by the user from the indexes to be migrated corresponding to the successfully created CK table.
3. The data migration method according to claim 1, characterized in that, The method of using Java reflection to obtain field information of data entities, and generating DML statements for inserting data into CK based on the field information, includes: Use Java reflection to retrieve the first attribute value of an attribute column of a data entity and the second attribute value of the primary key attribute of the data entity. Based on the annotation tags, the mapping relationship between the first attribute value and the column name of the CK table is determined, and the mapping relationship between the second attribute value and the primary key column name of the CK table is determined. DML statements are generated according to the determined mapping relationship.
4. The data migration method according to claim 1, characterized in that, Also includes: Obtain relevant information about the migration process and feed that information back to the front end; The relevant information for the migration process includes at least: migration progress and / or migration success rate.
5. A data migration device, characterized in that, The apparatus for performing the method according to any one of claims 1-4, comprising: A building unit is used to build a relationship mapping dictionary between the index to be migrated and the CK table based on the index to be migrated in ES; The DDL generation unit is used to generate DDL statements for corresponding data entities based on the relation mapping dictionary. A creation unit is used to execute the DDL statement to create a CK table in CK corresponding to the index to be migrated; The DML generation unit is used to read batch data entities from ES into memory based on the index to be migrated, obtain field information of the data entities using Java reflection mechanism, and generate DML statements for inserting data into CK based on the field information. The migration unit is used to execute the DML statements to migrate the batch data entities read from ES to the corresponding CK table in CK.
6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the method as described in any one of claims 1-4.
7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-4.