Data synchronization method and device based on MongoDB database

Abstract

The invention discloses a data synchronization method and device based on a MongoDB database, and the method comprises the steps: firstly responding to a data synchronization instruction, and determining to-be-converted original data from the MongoDB database, the original data having an original format adaptive to the MongoDB database; then, according to a preset mapping rule, converting the original data into target row-level data; and finally, sending the target row-level data to a target data system, the target row-level data having a target format adapted to the target data system. According to the design, the original data is automatically converted into the target row-level data through the preset mapping rule, so that the complexity and the error rate of manual code writing are reduced, the data synchronization efficiency is greatly improved, and the data synchronization between the MongoDB database and other systems is effectively realized.

Description

Technical Field

[0001] The present invention relates to the technical field of database management, and in particular, to a data synchronization method and device based on a MongoDB database. Background Art

[0002] MongoDB is an open-source document-oriented database that uses a non-relational data model and stores data in the form of JSON-like documents. Although MongoDB provides rich functions and excellent performance, when synchronizing data between multiple systems, due to the different data formats and processing methods of each system, it is often necessary to convert the original data of MongoDB into a format that other systems can understand and process. This involves the problems of data mapping and conversion. Traditional data synchronization methods may require manual coding for data mapping and conversion, which is a cumbersome and inefficient process. Therefore, there is an urgent need for an automated and efficient data synchronization method that can simplify the data mapping and conversion process and achieve data synchronization between the MongoDB database and other systems. Summary of the Invention

[0003] The purpose of the present invention is to provide a data synchronization method and device based on a MongoDB database.

[0004] In a first aspect, an embodiment of the present invention provides a data synchronization method based on a MongoDB database, including:

[0005] In response to a data synchronization instruction, determine the original data to be converted from the MongoDB database, where the original data has an original format adapted to the MongoDB database;

[0006] Convert the original data into target row-level data according to a pre-set mapping rule;

[0007] Send the target row-level data to a target data system, where the target row-level data has a target format adapted to the target data system.

[0008] In a possible implementation manner, the converting the original data into target row-level data according to a pre-set mapping rule includes:

[0009] Obtain the original characters included in the original data;

[0010] Based on the original characters, convert the original data into the target row-level data according to the pre-set mapping rule.

[0011] In a possible implementation, the original data is an original JSON field, and converting the original data into the target row-level data based on the original characters according to the preset mapping rules includes:

[0012] If there is a first JOSN character in the tag of the original JSON field, split the data included in the original JSON field and store the split data in an additional table to obtain the target row-level data.

[0013] In a possible implementation, the original data is an original JSON field, and converting the original data into the target row-level data based on the original characters according to the preset mapping rules includes:

[0014] If there is a second JOSN character in the tag of the original JSON field, determine that the original JSON field is an array field;

[0015] Convert the array field into a comma-separated string to map the array field to a target table field and obtain the target row-level data.

[0016] In a possible implementation, the original data is an original JSON field, and converting the original data into the target row-level data based on the original characters according to the preset mapping rules includes:

[0017] If there is a third JOSN character in the tag of the original JSON field, determine that the original JSON field is an array field;

[0018] Obtain the array number of the array field, and combine the outermost tag and the array number to obtain the target row-level data.

[0019] In a possible implementation, the original data is an original JSON field, and converting the original data into the target row-level data based on the original characters according to the preset mapping rules includes:

[0020] If there is a fourth JOSN character in the tag of the original JSON field, determine that the original JSON field is not the first-layer data;

[0021] Split the field tag of the original JSON field by a dot character to obtain the target row-level data.

[0022] In a possible implementation, the target data system includes KAFAK, Greenplum, and Doris.

[0023] In a second aspect, an embodiment of the present invention provides a data synchronization device based on a MongoDB database, including:

[0024] An acquisition module, configured to determine original data to be converted from the MongoDB database in response to a data synchronization instruction, where the original data has an original format adapted to the MongoDB database;

[0025] A synchronization module, configured to convert the original data into target row-level data according to a preset mapping rule; and send the target row-level data to a target data system, where the target row-level data has a target format adapted to the target data system.

[0026] In a third aspect, an embodiment of the present invention provides a computer device, including a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the data synchronization method based on the MongoDB database described in at least one possible implementation manner of the first aspect.

[0027] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, including a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the data synchronization method based on the MongoDB database described in at least one possible implementation manner of the first aspect.

[0028] Compared with the prior art, the beneficial effects provided by the present invention include: By adopting a data synchronization method and device based on a MongoDB database disclosed in the present invention, in response to a data synchronization instruction, original data to be converted is determined from the MongoDB database, and these original data have an original format adapted to the MongoDB database. Then, according to a preset mapping rule, these original data are converted into target row-level data. Finally, these target row-level data are sent to a target data system, and these target row-level data have a target format adapted to the target data system. Designed in this way, by automatically converting the original data into target row-level data through a preset mapping rule, not only the complexity and error rate of manually writing code are reduced, but also the efficiency of data synchronization is greatly improved, effectively realizing data synchronization between the MongoDB database and other systems. Description of the Drawings

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0030] Figure 1 It is a schematic flowchart of the steps of the data synchronization method based on the MongoDB database provided by the embodiment of the present invention;

[0031] Figure 2 It is a schematic block diagram of the structure of the data synchronization device based on the MongoDB database provided by the embodiment of the present invention;

[0032] Figure 3 It is a schematic block diagram of the structure of the computer device provided by the embodiment of the present invention. Detailed implementation manners

[0033] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0034] The following will, with reference to the accompanying drawings, elaborate on the detailed implementation manners of the present invention.

[0035] To solve the technical problems in the foregoing background art, Figure 1 It is a schematic flowchart of the data synchronization method based on the MongoDB database provided by the embodiment of the present disclosure. The data synchronization method based on the MongoDB database will be introduced in detail below.

[0036] Step S201, in response to a data synchronization instruction, determine the original data to be converted from the MongoDB database, and the original data has an original format adapted to the MongoDB database;

[0037] Step S202, convert the original data into target row-level data according to a preset mapping rule;

[0038] Step S203, send the target row-level data to a target data system, and the target row-level data has a target format adapted to the target data system.

[0039] In an embodiment of the present invention, by way of example, assume there is an e-commerce platform that uses MongoDB as the main data storage and processing system. The database of this platform contains various information related to products, orders, and users. Now, the platform needs to synchronize a specific type of order data to another data system (target data system) for in-depth analysis and report generation. The platform administrator first specifies the rules and conditions for data synchronization through a configuration parser in response to a data synchronization instruction. For example, the administrator specifies to only synchronize the order data with completed payments and selects specific fields for synchronization. The database linker is responsible for establishing a connection with the MongoDB database and querying the raw data to be converted according to the conditions defined in the configuration parser. In this example, the database linker will execute a query to obtain the order data with completed payments. The data format conversion parser converts the raw data into target row-level data according to the pre-set mapping rules. In this example, some processing and conversions may be required for the raw order data, such as adding additional calculated fields or changing the date and time format. The target data system can be various different systems, such as Kafka, Greenplum, or Doris. In this example, we assume the target data system is Kafka, which is used to receive and store the converted target row-level data. KAFKA DATA OUTPUTTER: The data outputter is responsible for sending the target row-level data to the target data system (Kafka). For each converted target row-level data, the data outputter adapts its format to the data format required by Kafka and sends it to the corresponding topic or queue. In summary, the data synchronization method based on the MongoDB database can achieve the extraction, transformation, and loading (ETL) of data through components such as a parser, linker, converter, and outputter, extract the raw data from the MongoDB database, convert it into target row-level data, and send it to the target data system (such as Kafka) for further processing and analysis.

[0040] In an embodiment of the present invention, the foregoing step S202 may be implemented by the following method.

[0041] (1) Obtain the original characters included in the original data;

[0042] (2) Based on the original characters, convert the original data into the target row-level data according to the pre-set mapping rules.

[0043] In an embodiment of the present invention, by way of example, the platform administrator pre-sets a mapping rule through a configuration parser to specify how to convert the original characters into target row-level data. For example, the administrator can define the product name, price, and inventory quantity as the fields of the target row-level data. The database linker is still responsible for establishing a connection with the MongoDB database and obtaining the original characters included in the original data. In this example, the administrator may specify to query the product collection and obtain the original character data such as the product name, price, and inventory quantity. Obtain the original character data: Obtain the original character data of the product from the MongoDB database, including the product name, price, and inventory quantity.

[0044] Convert according to the mapping rule: According to the pre-set mapping rule, convert the original character data into target row-level data. For example, use the product name as the name field of the target row-level data, the price as the price field of the target row-level data, and the inventory quantity as the inventory field of the target row-level data. Finally, the converted target row-level data will be sent to the target data system (Kafka) for further processing and analysis. In this example, the target row-level data may include information on multiple products, and each product has fields such as its name, price, and inventory quantity. Designed in this way, by extracting the original character data and converting it according to the pre-set mapping rule, the original data is converted into target row-level data and sent to the target data system for further processing and analysis.

[0045] In an embodiment of the present invention, the original data is an original JSON field, and the step of converting the original data into the target row-level data based on the original characters according to the pre-set mapping rule can be implemented in the following manner.

[0046] If there is a first JOSN character in the tag of the original JSON field, split the data included in the original JSON field and store the split data in an additional table to obtain the target row-level data.

[0047] In an embodiment of the present invention, by way of example, the platform administrator pre-sets mapping rules and splitting rules through a configuration parser, specifying how to split the data in the original JSON field according to the pre-set markers, and storing the split data in an additional table. For example, the administrator can define the marker as "|", and determine in which fields the split data should be stored. The database linker is responsible for establishing a connection with the MongoDB database and obtaining the original data containing the original JSON field. In this example, the administrator may specify to query the product collection and obtain the original JSON field data of the product. In this scenario, the data format conversion parser needs to perform a splitting operation based on the obtained original JSON field data. According to the pre-set mapping rules and splitting rules, the following operations can be performed:

[0048] Obtain the original JSON field data: Obtain the original JSON field data of the product from the MongoDB database.

[0049] Determine whether there is a first JSON character: For each original JSON field, determine whether there is a first "|" character as a marker. If it exists, perform the following operations:

[0050] Split the data and store it in the additional table: According to the pre-set marker "|", split the data in the original JSON field, and store the split data in the additional table. For example, if the original JSON field contains multiple attributes of a product, each attribute separated by "|", it can be split according to this marker, and the split data can be stored in the corresponding fields of the additional table. Finally, the split target row-level data will be sent to the target data system (Kafka) for further processing and analysis. In this example, the target row-level data may include the basic information of the product and the split attribute information, and each attribute has its corresponding field. Designed in this way, by parsing the original JSON field and splitting and storing the data according to the pre-set mapping rules and splitting rules, the target row-level data is obtained and sent to the target data system for further processing and analysis. In this scenario, we assume that the marker "|" is used to split the original JSON field data.

[0051] In an embodiment of the present invention, the original data is the original JSON field, and the step of converting the original data into the target row-level data according to the pre-set mapping rules based on the original character can be implemented by the following method.

[0052] (1) If there is a second JOSN character in the marker of the original JSON field, it is determined that the original JSON field is an array field;

[0053] (2) Convert the array field into a comma-separated string to map the array field to a target table field, and obtain the target row-level data.

[0054] In an embodiment of the present invention, exemplarily, the platform administrator presets mapping rules and determination rules through a configuration parser, specifying how to convert and map the array fields in the original JSON fields. For example, the administrator can define the second JSON character as "[,]" and determine to convert the array field into a comma-separated string. The database linker is responsible for establishing a connection with the MongoDB database and obtaining the original data containing the original JSON fields. In this example, the administrator may specify to query the product collection and obtain the original JSON field data of the products. In this scenario, the data format conversion parser needs to perform conversion operations based on the obtained original JSON field data. According to the preset mapping rules and determination rules, the following operations can be performed:

[0055] Obtain the original JSON field data: Obtain the original JSON field data of the products from the MongoDB database.

[0056] Determine whether there is a second JSON character: For each original JSON field, determine whether there is a second "[,]" character as a marker. If it exists, perform the following operations:

[0057] Array field conversion: Convert the array field in the original JSON field, and splice the elements in the array into a string with commas. For example, if the original JSON field contains the label information of the product, and each label is separated by "[,]", these labels can be converted into a comma-separated string.

[0058] Map to the target table field: According to the mapping rules, map the converted string to the corresponding field of the target table. For example, map the converted label string to the label field of the target row-level data. Finally, the converted target row-level data will be sent to the target data system (Kafka) for further processing and analysis. In this example, the target row-level data may include the basic information of the product and the converted label field. Designed in this way, by parsing the original JSON field and performing the conversion and mapping of the array field according to the preset mapping rules and determination rules, the target row-level data is obtained and sent to the target data system for further processing and analysis. In this scenario, we assume that the second JSON character is "[,]" to determine the array field and convert it into a comma-separated string.

[0059] In an embodiment of the present invention, the original data is an original JSON field. The step of converting the original data into the target row-level data according to the preset mapping rule based on the original characters may be executed in the following manner.

[0060] (1) If a third JSON character exists in the tag of the original JSON field, it is determined that the original JSON field is an array field;

[0061] (2) Obtain the array number of the array field, and combine the outermost tag and the array number to obtain the target row-level data.

[0062] In an embodiment of the present invention, by way of example, the platform administrator presets the mapping rule and the determination rule through a configuration parser, specifying how to obtain the target row-level data according to the array field and number in the original JSON field. For example, the administrator can define the third JSON character as "[]" and determine how to combine the outermost tag and the array number. The database linker is responsible for establishing a connection with the MongoDB database and obtaining the original data containing the original JSON field. In this example, the administrator may specify to query the order collection and obtain the original JSON field data of the order. In this scenario, the data format conversion parser needs to perform processing operations based on the obtained original JSON field data. According to the preset mapping rule and determination rule, the following operations can be performed:

[0063] Obtain the original JSON field data: Obtain the original JSON field data of the order from the MongoDB database.

[0064] Determine whether a third JSON character exists: For each original JSON field, determine whether there is a third "[]" character as a tag. If it exists, perform the following operations:

[0065] Array field determination: Mark the original JSON field as an array field.

[0066] Obtain the array number: According to the position of the outermost tag and the array number, obtain the array number of the array field. For example, if the original JSON field contains multiple product information in an order, each product information is surrounded by "[]" and the numbers start from 1 and increase incrementally, the target row-level data of each product information can be obtained based on these tags and numbers.

[0067] Combine the tag and number to obtain the target row-level data: According to the mapping rules, combine the outermost tag and the array number to generate the target row-level data. For example, combine other information of the order with specific product information to form the target row-level data. Finally, the obtained target row-level data will be sent to the target data system (Kafka) for further processing and analysis. In this example, the target row-level data may include the basic information of the order and the detailed content of the corresponding product information. Designed in this way, by parsing the original JSON fields and obtaining the numbers of the array fields according to the preset mapping rules and determination rules, and combining the outermost tag and the array number to construct the target row-level data. Finally, send the target row-level data to the target data system for further processing and analysis. In this scenario, we assume that the third JSON character is "[]" to determine the array field, and use the outermost tag and the array number to obtain the target row-level data.

[0068] In an embodiment of the present invention, the original data is the original JSON field. The step of converting the original data into the target row-level data based on the original characters according to the preset mapping rules can be executed in the following manner.

[0069] (1) If the fourth JSON character exists in the tag of the original JSON field, it is determined that the original JSON field is not the first-level data;

[0070] (2) Split the field tag of the original JSON field by the dot character to obtain the target row-level data.

[0071] In an embodiment of the present invention, for example, the platform administrator presets the mapping rules and determination rules through the configuration parser, specifying how to split and process the field tags according to the dot characters in the original JSON field to obtain the target row-level data. For example, the administrator can define the fourth JSON character as "." and determine how to use the dot character to split the field tags. The database linker is responsible for establishing a connection with the MongoDB database and obtaining the original data containing the original JSON field. In this example, the administrator may specify to query the user collection and obtain the original JSON field data of the user. In this scenario, the data format conversion parser needs to perform processing operations based on the obtained original JSON field data. According to the preset mapping rules and determination rules, the following operations can be performed:

[0072] Obtain the original JSON field data: Obtain the original JSON field data of the user from the MongoDB database.

[0073] Determine whether there is a fourth JSON character: For each original JSON field, determine whether there is a fourth "." character as a marker. If it exists, perform the following operations:

[0074] Field marker splitting: Use the dot character to split the field markers of the original JSON field. For example, if the original JSON field contains the user's detailed address information, the dot character can be used to split the field marker into different hierarchical field markers such as country, province, city, etc.

[0075] Obtain target row-level data: According to the mapping rules, use the split field markers to obtain the target row-level data. For example, combine the user's other information with each hierarchical field of the detailed address information to form the target row-level data. Finally, the obtained target row-level data will be sent to the target data system (Kafka) for further processing and analysis. In this example, the target row-level data may include the user's basic information and each hierarchical field of the detailed address information. Designed in this way, by parsing the original JSON field and following the pre-set mapping rules and determination rules, use the dot character to split the field marker to obtain the target row-level data. Finally, send the target row-level data to the target data system for further processing and analysis. In this scenario, we assume that the fourth JSON character is "." to determine the field marker and use the dot character to split to obtain the target row-level data.

[0076] In the embodiment of the present invention, the target data system includes KAFAK, Greenplum, and Doris.

[0077] In the embodiment of the present invention, exemplarily, the database linker is responsible for establishing a connection with the MongoDB database and obtaining the original data containing the original JSON field. In this example, the administrator may specify to query the order collection and obtain the original JSON field data of the order. The data format conversion parser is responsible for converting the original data into the target row-level data and sending it to the target data system for further processing and analysis.

[0078] Obtain the original JSON field data: Obtain the original JSON field data of the order from the MongoDB database.

[0079] Convert the original data into the target row-level data according to the pre-set mapping rules.

[0080] Send the target row-level data to the target data system (Kafka). As the target data system, Kafka receives and stores the target row-level data. In this example, Kafka can be used for real-time streaming processing, such as real-time monitoring and streaming computing. As part of the target data system, Greenplum can be used for long-term storage and analysis of the target row-level data. For example, administrators can use Greenplum for offline analysis of order data, OLAP queries, data mining, etc. As another part of the target data system, Doris is a distributed database that supports online analytical processing (OLAP) scenarios. Administrators can use Doris to perform real-time queries and analysis on the target row-level data to meet business requirements. In the data synchronization scenario based on the MongoDB database, the original data can be converted into target row-level data and sent to the target data system including Kafka, Greenplum, and Doris. This can meet the requirements of both real-time streaming processing and offline analysis, enabling the data to be processed in real time and stored long-term, and provided to users for various types of query and analysis operations. In this scenario, we assume that the target data system includes Kafka, Greenplum, and Doris.

[0081] The following provides an overall implementation process of an embodiment of the present invention.

[0082] After the system starts, it connects to the database through the configuration file information and extracts the corresponding database table data. Then, according to the mapping configuration file in the configuration file, the data is converted into row-level data through the data format conversion parser and then written into the corresponding target.

[0083] Implementation concept:

[0084] In the mapping file of the table fields, the one-to-one correspondence between each field of the mongo data, that is, the json type data, and the fields stored in the target database has been defined. Therefore, after reading the data from mongo, through the field mapping configuration file, the specific data is taken out one by one and then directly spliced into the insert statement of the target database. Since the json data may have multiple layers and the sub-layer may also be an array, the system has built-in processing methods for these situations.

[0085] Basic rules

[0086] 1. If the mark exists in the original field of the configuration file, the data corresponding to this field in the original json should be separately split and stored in an additional table.

[0087] 2. If the mark [,] exists in the original field, the field in the original json that is an array field is mapped to the target table field, and the array is converted into a string concatenated by commas.

[0088] 3. If there are [] in the tags of the original field, the field in the original JSON is an array field, and the content of the target field consists of the outermost _id and the array number of the original field layer.

[0089] 4. If there is a. in the tags of the original field, it means that it is not the first layer in the JSON, and each layer of field tags is separated by..

[0090] Mapping configuration example:

[0091] agentMo.yml configuration file

[0092] _id: id / / The _id in the original JSON is mapped to the table field id

[0093] name: -- / / The name in the original JSON is mapped to the table field name

[0094] orderId: order_id / / The orderId in the original JSON is mapped to the table field order_id insurer.insurerId: insurer_id / / The sub-layer insurerId of the insurer field in the original JSON is mapped to the table field insurer_id

[0095] phone[,]: phone / / The array field of pthone in the original JSON is mapped to the table field phone and the array is converted into a string concatenated by,

[0096] agentTypes|agent_types: / / The agentTypes field in the original JSON is mapped to the affiliated table agent_types

[0097] typeName: type_name / / The sub-layer typeName of the agentTypes field in the original JSON is mapped to the type_name of the affiliated table agent_types

[0098] _id[]: id / / The id field of the affiliated table agent_types consists of the outermost _id and the array number of the agentTypes layer

[0099] _id: fid / / The fid field of the affiliated table agent_types consists of the outermost _id

[0100] agentType: agent_type

[0101] Original data:

[0102]

[0103] After conversion, Table 1 and Table 2 can be referred to. Among them, Table 1 is the agentMo table, and Table 2 is the agent_types table.

[0104] Table 1

[0105]

[0106] Table 2

[0107]

[0108]

[0109] Please refer to Figure 2 , Figure 2 A data synchronization device 110 based on a MongoDB database provided by an embodiment of the present invention, including:

[0110] An acquisition module 1101, configured to determine original data to be converted from the MongoDB database in response to a data synchronization instruction, where the original data has an original format adapted to the MongoDB database;

[0111] A synchronization module 1102, configured to convert the original data into target row-level data according to a preset mapping rule; and send the target row-level data to a target data system, where the target row-level data has a target format adapted to the target data system.

[0112] It should be noted that the implementation principle of the aforementioned data synchronization device 110 based on the MongoDB database can refer to the implementation principle of the aforementioned data synchronization method based on the MongoDB database, which will not be elaborated here. It should be understood that the division of each module of the above device is only a division of logical functions. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the data synchronization device 110 based on the MongoDB database can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and the function of the above data synchronization device 110 based on the MongoDB database can be called and executed by a certain processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together or can be independently implemented. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the processor element or the instruction in the form of software.

[0113] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a processing element scheduling program code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. Again, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0114] An embodiment of the present invention provides a computer device 100. The computer device 100 includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned data synchronization device 110 based on the MongoDB database. As Figure 3 shown, Figure 3The computer device 100 provided in the embodiment of the present invention is a structural block diagram. The computer device 100 includes a data synchronization device 110 based on a MongoDB database, a memory 111, a processor 112 and a communication unit 113.

[0115] In order to realize data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be realized through one or more communication buses or signal lines. The data synchronization device 110 based on the MongoDB database includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the data synchronization device 110 based on the MongoDB database stored in the memory 111, such as the software function modules and computer programs included in the data synchronization device 110 based on the MongoDB database.

[0116] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, the computer device where the readable storage medium is located is controlled to execute the aforementioned data synchronization method based on the MongoDB database.

[0117] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A data synchronization method based on the MongoDB database, characterized in that including: In response to a data synchronization instruction, determine the raw data to be converted from the MongoDB database, where the raw data has a raw format adapted to the MongoDB database; Convert the raw data into target row-level data according to a preset mapping rule; Send the target row-level data to a target data system, where the target row-level data has a target format adapted to the target data system.

2. The method according to claim 1, wherein The step of converting the raw data into target row-level data according to a preset mapping rule includes: Obtain the raw characters included in the raw data; Based on the raw characters, convert the raw data into the target row-level data according to the preset mapping rule.

3. The method according to claim 2, wherein When the raw data is a raw JSON field, the step of converting the raw data into the target row-level data based on the raw characters according to the preset mapping rule includes: If a first JOSN character exists in the tag of the raw JSON field, split the data included in the raw JSON field and store the split data in an additional table to obtain the target row-level data.

4. The method according to claim 2, characterized in that When the raw data is a raw JSON field, the step of converting the raw data into the target row-level data based on the raw characters according to the preset mapping rule includes: If a second JOSN character exists in the tag of the raw JSON field, determine that the raw JSON field is an array field; Convert the array field into a string concatenated with commas to map the array field to a target table field and obtain the target row-level data.

5. The method according to claim 2, wherein When the raw data is a raw JSON field, the step of converting the raw data into the target row-level data based on the raw characters according to the preset mapping rule includes: If a third JOSN character exists in the tag of the raw JSON field, determine that the raw JSON field is an array field; Obtain the array number of the array field, and combine the outermost tag and the array number to obtain the target row-level data.

6. The method according to claim 2, wherein When the raw data is a raw JSON field, the step of converting the raw data into the target row-level data based on the raw characters according to the preset mapping rule includes: If a fourth JOSN character exists in the tag of the raw JSON field, determine that the raw JSON field is not the first-layer data; Split the field tag of the raw JSON field through a dot character to obtain the target row-level data.

7. The method according to claim 1, wherein The target data system includes KAFAK, Greenplum, and Doris.

8. A data synchronization device based on a MongoDB database, characterized in that including: An acquisition module, configured to determine, in response to a data synchronization instruction, the raw data to be converted from the MongoDB database, where the raw data has a raw format adapted to the MongoDB database; A synchronization module, configured to convert the raw data into target row-level data according to a preset mapping rule; send the target row-level data to a target data system, where the target row-level data has a target format adapted to the target data system.

9. A computer device, characterized in that, The computer device includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the data synchronization method based on the MongoDB database described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the data synchronization method based on the MongoDB database described in any one of claims 1-7.

Images