MongoDB nested array document migration synchronization method and system based on DataX

Abstract

The application discloses a MongoDB nested array document migration synchronization method and system based on DataX, belongs to the technical field of databases and data migration, and will convert a nested array document Document object list read according to configuration into different types of data according to different requirement configuration parameters exportType; if the custom parameter exportType is empty, the original Document document format of the migrated data is reserved; if the custom parameter exportType is json, the migrated data is in a json array format; if the custom parameter exportType is csv, the migrated data is in a csv format; if the custom parameter exportType is expand, the nested array document of the migrated data is separated into independent records and output; different data formats are distinguished through parameters, different apis are called, and different business requirements are met. The application is suitable for the synchronization of the nested array document in the MongoDB to a new database system.

Description

Technical Field

[0001] This invention relates to the field of database and data migration technology, specifically to a method and system for synchronizing and migrating nested array documents in MongoDB based on DataX. Background Technology

[0002] With the advent of the information age, information is exploding in size, and the amount of information stored is growing ever larger, making information migration and synchronization increasingly complex. DataX provides a convenient data migration and synchronization tool that supports data migration and synchronization between mainstream relational databases, non-relational databases, and big data computing systems. A common method for synchronizing nested array documents in MongoDB involves concatenating the data within the nested array documents as individual documents, with the concatenated document content in key-value format. This type of data is very inconvenient to parse and use in the new database system. Summary of the Invention

[0003] The technical objective of this invention is to address the above-mentioned shortcomings by providing a method and system for migrating and synchronizing nested array documents in MongoDB based on DataX. This method is applicable to synchronizing nested array documents in MongoDB to a new database system and can meet different business needs.

[0004] The technical solution adopted by this invention to solve its technical problem is:

[0005] The MongoDB nested array document migration and synchronization method based on DataX reads a list of nested array document Document objects according to the configuration, converts it into different types of data according to different requirements and configuration parameters of exportType, and obtains different functions according to different parameters passed.

[0006] If the custom parameter exportType is empty, the exported data retains the original Document format; if the custom parameter exportType is json, the exported data is in JSON array format; if the custom parameter exportType is csv, the exported data is in CSV format; if the custom parameter exportType is expand, the exported data will be split into individual records and output.

[0007] Multiple different data formats are used to distinguish and call different APIs through parameters to meet different business needs, and the converted data is stored in the corresponding fields of other types of databases.

[0008] DataX uses a Document object string (Document[key=value]) to migrate and synchronize nested array documents from MongoDB to a new database system. This format is not user-friendly in the new database system, requiring complex string manipulation operations such as truncation to obtain the required data. This proposed method, based on DataX, enhances the original migration functionality and is suitable for synchronizing nested array documents from MongoDB to a new database system.

[0009] Furthermore, if the custom parameter `exportType` is set to `json`, then the JSON conversion API is called to convert the list of Document objects into a JSON array format. The implementation process is as follows:

[0010] Call the JSON conversion API;

[0011] Iterate through the list of Document objects;

[0012] Serialize the Document object into a JSON string;

[0013] Assemble and concatenate the data into a string in JSON format.

[0014] Furthermore, by setting the custom parameter `exportType` to `csv`, the CSV conversion API is called to convert the list of Document objects into CSV format data. The implementation process is as follows:

[0015] Call the CSV conversion API;

[0016] Iterate through the list of Document objects;

[0017] Parse the document data and convert it into CSV format based on the delimiter;

[0018] Convert all row data into a single CSV file.

[0019] Furthermore, by setting the custom parameter `exportType` to `expand`, the `expand` conversion API is called to convert the list of Document objects into individual records. The implementation process is as follows:

[0020] Call the expand conversion API

[0021] Traversing Document objects

[0022] Parse Document data and merge data at the same level as embedded documents.

[0023] They are assembled into individual data points.

[0024] Preferably, the specific process of this method is as follows:

[0025] S1. Retrieve nested array document fields of type array configured in DataX;

[0026] S2. Retrieve the list of Document objects corresponding to this field;

[0027] S3. Determine the specific type of the output configuration parameter exportType;

[0028] If S4 and exportType are empty, proceed to step S18; if they are json, proceed to step S5; if they are csv, proceed to step S9; if they are expand, proceed to step S13.

[0029] S5. Call the JSON conversion API.

[0030] S6. Iterate through the Document objects.

[0031] S7. Serialize the Document object into JSON.

[0032] S8. Assemble all the JSON strings into a JSON array and execute step S17;

[0033] S9. Call the CSV conversion API.

[0034] S10. Traverse the Document objects.

[0035] S11. Parse the Document data and convert it into CSV format based on the delimiter.

[0036] S12. Convert all row data into a complete CSV format and proceed to step S17.

[0037] S13. Call the expand conversion API.

[0038] S14. Traverse the Document objects.

[0039] S15. Parse the Document data and merge data at the same level as embedded documents.

[0040] S16. Assemble into individual data entries;

[0041] S17. Place the converted data into the corresponding positions in the output fields;

[0042] S18. Store in the new database system that has been migrated and synchronized.

[0043] This invention also claims protection for a DataX-based MongoDB nested array document migration and synchronization system, including a data transformation module and a data storage module.

[0044] The data conversion module will read the nested array of Document objects based on the configuration, and then convert them into different types of data according to the configured exportType parameter.

[0045] If the custom parameter exportType is empty, the exported data retains the original Document format; if the custom parameter exportType is json, the exported data is in JSON array format; if the custom parameter exportType is csv, the exported data is in CSV format; if the custom parameter exportType is expand, the exported data will be split into individual records and output.

[0046] Multiple different data formats are used to distinguish and call different APIs through parameters to meet different business needs. The data storage module stores the converted data into the corresponding fields of other types of databases.

[0047] Furthermore, if the custom parameter `exportType` is set to `json`, then the JSON conversion API is called to convert the list of Document objects into a JSON array format. The implementation process is as follows:

[0048] Call the JSON conversion API;

[0049] Iterate through the list of Document objects;

[0050] Serialize the Document object into a JSON string;

[0051] Assemble and concatenate the data into a string in JSON format.

[0052] Furthermore, by setting the custom parameter `exportType` to `csv`, the CSV conversion API is called to convert the list of Document objects into CSV format data. The implementation process is as follows:

[0053] Call the CSV conversion API;

[0054] Iterate through the list of Document objects;

[0055] Parse the document data and convert it into CSV format based on the delimiter;

[0056] Convert all row data into a single CSV file.

[0057] Furthermore, by setting the custom parameter `exportType` to `expand`, the `expand` conversion API is called to convert the list of Document objects into individual records. The implementation process is as follows:

[0058] Call the expand conversion API

[0059] Traversing Document objects

[0060] Parse Document data and merge data at the same level as embedded documents.

[0061] They are assembled into individual data points.

[0062] Preferably, the specific process for document migration and synchronization in this system is as follows:

[0063] S1. Retrieve nested array document fields of type array configured in DataX;

[0064] S2. Retrieve the list of Document objects corresponding to this field;

[0065] S3. Determine the specific type of the output configuration parameter exportType;

[0066] If S4 and exportType are empty, proceed to step S18; if they are json, proceed to step S5; if they are csv, proceed to step S9; if they are expand, proceed to step S13.

[0067] S5. Call the JSON conversion API.

[0068] S6. Iterate through the Document objects.

[0069] S7. Serialize the Document object into JSON.

[0070] S8. Assemble all the JSON strings into a JSON array and execute step S17;

[0071] S9. Call the CSV conversion API.

[0072] S10. Traverse the Document objects.

[0073] S11. Parse the Document data and convert it into CSV format based on the delimiter.

[0074] S12. Convert all row data into a complete CSV format and proceed to step S17.

[0075] S13. Call the expand conversion API.

[0076] S14. Traverse the Document objects.

[0077] S15. Parse the Document data and merge data at the same level as embedded documents.

[0078] S16. Assemble into individual data entries;

[0079] S17. Place the converted data into the corresponding positions in the output fields;

[0080] S18. Store in the new database system that has been migrated and synchronized.

[0081] Compared with existing technologies, the MongoDB nested array document migration and synchronization method and system based on DataX of the present invention have the following advantages:

[0082] This method provides a versatile and easy-to-use approach to transforming nested MongoDB array documents, including methods for converting nested MongoDB array documents to JSON arrays, converting them to CSV format, and splitting them into individual records. The DataX-based MongoDB nested array document migration and synchronization method proposed in this paper enhances the original migration functionality. It retrieves different functions based on different parameters passed; it supports various data formats and distinguishes between API calls based on parameters to meet diverse business needs. It is suitable for synchronizing nested array documents in MongoDB to a new database system. Attached Figure Description

[0083] Figure 1 This is a flowchart illustrating the implementation of the MongoDB nested array document migration and synchronization method based on DataX, provided in an embodiment of the present invention. Detailed Implementation

[0084] The present invention will be further described below with reference to specific embodiments.

[0085] This invention provides a method for synchronizing and migrating nested array documents in MongoDB based on DataX. The method reads a list of nested array document objects according to the configuration, converts them into different types of data according to the exportType parameter, and obtains different functions based on the different parameters passed.

[0086] If the custom parameter exportType is empty, the exported data retains the original Document format; if the custom parameter exportType is json, the exported data is in JSON array format; if the custom parameter exportType is csv, the exported data is in CSV format; if the custom parameter exportType is expand, the exported data will be split into individual records and output.

[0087] The `[exportType:json]` option calls a JSON conversion API to convert a list of Document objects into a JSON array. The process is as follows:

[0088] Call the JSON conversion API;

[0089] Iterate through the list of Document objects;

[0090] Serialize the Document object into a JSON string;

[0091] Assemble and concatenate the data into a string in JSON format.

[0092] [exportType:csv] calls the CSV conversion API to convert the list of Document objects into CSV format data. The process is as follows:

[0093] Call the CSV conversion API;

[0094] Iterate through the list of Document objects;

[0095] Parse the document data and convert it into CSV format based on the delimiter;

[0096] Convert all row data into a single CSV file.

[0097] The `[exportType:expand]` option calls the expand API to convert a list of Document objects into individual records. The process is as follows:

[0098] Call the expand conversion API

[0099] Traversing Document objects

[0100] Parse Document data and merge data at the same level as embedded documents.

[0101] They are assembled into individual data points.

[0102] Multiple different data formats are used to distinguish and call different APIs through parameters to meet different business needs, and the converted data is stored in the corresponding fields of other types of databases.

[0103] DataX uses a Document object string (Document[key=value]) to migrate and synchronize nested array documents from MongoDB to a new database system. This format is not user-friendly in the new database system, requiring complex string manipulation operations such as truncation to obtain the required data. This proposed method, based on DataX, enhances the original migration functionality and is suitable for synchronizing nested array documents from MongoDB to a new database system.

[0104] like Figure 1 As shown, the specific process of this method is as follows:

[0105] S1. Retrieve nested array document fields of type array configured in DataX;

[0106] S2. Retrieve the list of Document objects corresponding to this field;

[0107] S3. Determine the specific type of the output configuration parameter exportType;

[0108] If S4 and exportType are empty, proceed to step S18; if they are json, proceed to step S5; if they are csv, proceed to step S9; if they are expand, proceed to step S13.

[0109] S5. Call the JSON conversion API.

[0110] S6. Iterate through the Document objects.

[0111] S7. Serialize the Document object into JSON.

[0112] S8. Assemble all the JSON strings into a JSON array and execute step S17;

[0113] S9. Call the CSV conversion API.

[0114] S10. Traverse the Document objects.

[0115] S11. Parse the Document data and convert it into CSV format based on the delimiter.

[0116] S12. Convert all row data into a complete CSV format and proceed to step S17.

[0117] S13. Call the expand conversion API.

[0118] S14. Traverse the Document objects.

[0119] S15. Parse the Document data and merge data at the same level as embedded documents.

[0120] S16. Assemble into individual data entries;

[0121] S17. Place the converted data into the corresponding positions in the output fields;

[0122] S18. Store in the new database system that has been migrated and synchronized.

[0123] This invention also provides a MongoDB nested array document migration and synchronization system based on DataX, including a data conversion module and a data storage module.

[0124] The data conversion module will read the nested array of Document objects based on the configuration, and then convert them into different types of data according to the configured exportType parameter.

[0125] If the custom parameter exportType is empty, the exported data retains the original Document format; if the custom parameter exportType is json, the exported data is in JSON array format; if the custom parameter exportType is csv, the exported data is in CSV format; if the custom parameter exportType is expand, the exported data will be split into individual records and output.

[0126] If the custom parameter `exportType` is set to `json`, then the JSON conversion API is called to convert the list of Document objects into a JSON array format. The implementation process is as follows:

[0127] Call the JSON conversion API;

[0128] Iterate through the list of Document objects;

[0129] Serialize the Document object into a JSON string;

[0130] Assemble and concatenate the data into a string in JSON format.

[0131] The custom parameter `exportType` is set to `csv`, and the CSV conversion API is called to convert the list of Document objects into CSV format data. The implementation process is as follows:

[0132] Call the CSV conversion API;

[0133] Iterate through the list of Document objects;

[0134] Parse the document data and convert it into CSV format based on the delimiter;

[0135] Convert all row data into a single CSV file.

[0136] With the custom parameter `exportType` set to `expand`, the `expand` conversion API is called to convert the list of Document objects into individual records. The implementation process is as follows:

[0137] Call the expand conversion API

[0138] Traversing Document objects

[0139] Parse Document data and merge data at the same level as embedded documents.

[0140] They are assembled into individual data points.

[0141] Multiple different data formats are used to distinguish and call different APIs through parameters to meet different business needs. The data storage module stores the converted data into the corresponding fields of other types of databases.

[0142] The specific process of document migration and synchronization in this system is as follows:

[0143] S1. Retrieve nested array document fields of type array configured in DataX;

[0144] S2. Retrieve the list of Document objects corresponding to this field;

[0145] S3. Determine the specific type of the output configuration parameter exportType;

[0146] If S4 and exportType are empty, proceed to step S18; if they are json, proceed to step S5; if they are csv, proceed to step S9; if they are expand, proceed to step S13.

[0147] S5. Call the JSON conversion API.

[0148] S6. Iterate through the Document objects.

[0149] S7. Serialize the Document object into JSON.

[0150] S8. Assemble all the JSON strings into a JSON array and execute step S17;

[0151] S9. Call the CSV conversion API.

[0152] S10. Traverse the Document objects.

[0153] S11. Parse the Document data and convert it into CSV format based on the delimiter.

[0154] S12. Convert all row data into a complete CSV format and proceed to step S17.

[0155] S13. Call the expand conversion API.

[0156] S14. Traverse the Document objects.

[0157] S15. Parse the Document data and merge data at the same level as embedded documents.

[0158] S16. Assemble into individual data entries;

[0159] S17. Place the converted data into the corresponding positions in the output fields;

[0160] S18. Store in the new database system that has been migrated and synchronized.

[0161] Through the specific embodiments described above, those skilled in the art can easily implement the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. Based on the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features to achieve different technical solutions.

[0162] Except for the technical features described in the specification, all other technologies are known to those skilled in the art.

Claims

1. A method for migrating and synchronizing MongoDB nested array document based on DataX, characterized in that, The list of nested array document Document objects read according to the configuration is converted into different types of data according to the different requirement configuration parameters exportType; If the custom parameter exportType is empty, the migrated data retains the original Document document format; if the custom parameter exportType is json, the migrated data is in json array format; if the custom parameter exportType is csv, the migrated data is in csv format; if the custom parameter exportType is expand, the migrated data will be separated from the nested array document into independent records and output; Different data formats are distinguished by parameters to call different apis to meet different business requirements, and the converted data is stored in other types of database corresponding fields; If the custom parameter exportType is json, the json conversion api is called to convert the list of Document objects into json array format, and the implementation process is as follows: Call the json conversion api; Iterate through the list of Document objects; Serialize the Document object into a json string; Assemble and splice the string into a json data format; If the custom parameter exportType is csv, the csv conversion api is called to convert the list of Document objects into csv format data, and the implementation process is as follows: Call the csv conversion api; Iterate through the list of Document objects; Parse the Document data and convert it into csv format data according to the separator; Convert all row data into overall csv format data; If the custom parameter exportType is expand, the expand conversion api is called to convert the list of Document objects into independent records, and the implementation process is as follows: Call the expand conversion api; Iterate through the Document objects; Parse the Document data and merge the embedded document sibling data; Assemble into independent data.

2. The DataX-based MongoDB nested array document migration synchronization method according to claim 1, characterized in that, The specific process of the method is as follows: S1, obtain the array type nested array document field configured by DataX; S2, obtain the list of Document objects corresponding to the field; S3, determine the specific type of the output configuration parameter exportType; S4, if exportType is empty, execute step S18; if exportType is json, execute step S5; if exportType is csv, execute step S9; if exportType is expand, execute step S13; S5, call the json conversion api, S6, iterate through the Document objects, S7, serialize the Document object into a json, S8, assemble and splice all json strings into a json array, and execute step S17; S9, call the csv conversion api, S10, iterate through the Document objects, S11, parse the Document data and convert it into csv format data according to the separator, S12, convert all row data into whole csv format data, execute step S17; S13, call expand conversion api, S14, traverse Document object, S15, parse Document data, merge embedded document sibling data, S16, assemble into a piece of independent data; S17, put the converted data into the corresponding position of the output field; S18, store in the new database system of migration synchronization.

3. The MongoDB nested array document migration synchronization system based on DataX, characterized in that, Including data conversion module and data storage module, The data conversion module will read the nested array document Document object list according to the configuration, and convert it into different types of data according to different demand configuration parameters exportType, Custom parameter exportType is empty, then the migrated data retains the original Document document format; Custom parameter exportType is json, then the migrated data is json array format; Custom parameter exportType is csv, then the migrated data is csv format; Custom parameter exportType is expand, then the migrated data will be separated into a piece of independent record output; A variety of different data formats, through parameter distinction calling different api, to meet different business needs, through the data storage module will convert the data into other types of database corresponding field; Custom parameter exportType is json, then call json conversion api, convert Document object list into json array format, the implementation process is as follows: Call json conversion api; Traverse Document object list; Document object serialization into json string; Assemble into a piece of independent data. Custom parameter exportType is csv, call csv conversion api, convert Document object list into csv format data, the implementation process is as follows: Call csv conversion api; Traverse Document object list; Parse Document data, convert to csv format data according to the separator; Convert all row data into whole csv format data; Custom parameter exportType is expand, call expand conversion api, convert Document object list into a piece of independent record, the implementation process is as follows: Call expand conversion api Traverse Document object Parse Document data, merge embedded document sibling data Assemble into a piece of independent data.

4. The DataX-based MongoDB nested array document migration synchronization system according to claim 3, wherein, The specific process of the system for document migration synchronization is as follows: S1, get the array type nested array document field of DataX configuration; S2, get the Document object list corresponding to the field; S3, judge the specific type of output configuration parameter exportType; S4, if exportType is empty, execute step S18; if exportType is json, execute step S5; if exportType is csv, execute step S9; if exportType is expand, execute step S13; S5, call json conversion api, S6, traverse Document object, S7, serialize Document object into json, S8, assemble all json strings into json array, execute step S17; S9, call csv conversion api, S10, traverse Document object, S11, parse Document data, convert into csv format data according to separator, S12, convert all row data into whole csv format data, execute step S17; S13, call expand conversion api, S14, traverse Document object, S15, parse Document data, merge embedded document sibling data, S16, assemble into independent data, S17, put converted data into corresponding position of output field; S18, store in new database system of migration synchronization.

Images