JSON (JavaScript Object Notation) data synchronization method and device, storage medium and data synchronization equipment

Abstract

The invention discloses a JSON (JavaScript Object Notation) data synchronization method and device, a storage medium and data synchronization equipment. JSON objects are processed by utilizing a pre-configured real-time data integration pipeline component, first sub JSON data streams and second sub JSON data streams are constructed, and the second sub JSON data streams with the same quantity as the first sub JSON data streams are copied according to the quantity of the first sub JSON data streams; and determining and outputting the target data stream according to the first sub-JSON data stream and the second sub-JSON data stream which are in one-to-one correspondence, thereby realizing flattening processing of the JSON data with the JSON array, and solving the technical problem that the workload of extracting useful JSON data and performing synchronization by means of an SQL statement is relatively large when the JSON data is synchronized.

Description

Technical Field

[0001] The present application relates to the field of data synchronization technology, and in particular to a JSON data synchronization method, apparatus, storage medium and data synchronization device. Background Art

[0002] When facing JSON data synchronized from an interface or embedding point to a database, we usually have to store the entire JSON string in a single field, and then use SQL statements to parse the JSON and extract the necessary fields. Due to the complexity of JSON, this process often requires us to write a lot of SQL code, which is extremely laborious. On the other hand, some JSON data itself is huge in size. If we only need to access one or two key parameters, storing all the JSON data will obviously lead to a lot of unnecessary storage consumption.

[0003] SeaTunnel is a data synchronization tool that includes data source, target, data processing engine and Table API. The data source is used to read data from various data sources, including all common databases, lakes and warehouses, as well as domestic and foreign SaaS services. The target is mainly responsible for writing data, including common databases, lakes, warehouses, and domestic and foreign SaaS services. The data processing engine is used to synchronize data using its own engine SeaTunnelEngine. Table API Based on the Table API, upper-level users can create applications as Web or in the form of Table API.

[0004] Currently, using SeaTunnel for JSON data synchronization still has the problem of heavy workload. Summary of the invention

[0005] The main purpose of the present application is to provide a JSON data synchronization method, apparatus, storage medium and data synchronization device, aiming to solve the technical problem of heavy workload in extracting useful JSON data and synchronizing it with SQL statements when synchronizing JSON data.

[0006] To achieve the above-mentioned purpose, the present application provides a JSON data synchronization method, comprising: obtaining JSON data containing JSON objects stored in a source database; processing the JSON objects using a preconfigured real-time data integration pipeline component, obtaining a first key-value pair group of each layer of the JSON object and a path name corresponding to each value of the first key-value pair group, as well as a second key-value pair group and a path name corresponding to each value of the second key-value pair group, wherein the first key-value pair group includes a JSON array having multiple JSON objects, and the second key-value pair group includes all other key-value pairs except the JSON array; obtaining the value and path name of each JSON object in the JSON array, and constructing a corresponding first sub-JSON data stream using the value and path name of each JSON object; obtaining the value and path name of the second key-value pair group, constructing a second sub-JSON data stream according to the value and path name of the second key-value pair group, and copying the same number of second sub-JSON data streams according to the number of the first sub-JSON data streams; determining and outputting a target data stream according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream.

[0007] Optionally, before obtaining the JSON data containing JSON objects stored in the source database, the JSON data synchronization method further includes: in response to a data synchronization request, adding an extended JSONpath component to the real-time data integration pipeline component to obtain the preconfigured real-time data integration pipeline component.

[0008] Optionally, the method of processing the JSON object using a preconfigured real-time data integration pipeline component to obtain the first key-value pair group of each layer of the JSON object and the path name corresponding to each value of the first key-value pair group, as well as the second key-value pair group and the path name corresponding to each value of the second key-value pair group includes: processing the JSON object using a prebuilt first JSONpath expression to obtain all key-value pairs of the first layer of the JSON object, and obtaining the first key-value pair group and the second key-value pair group based on all the key-value pairs of the first layer; processing the first key-value pair group and the second key-value pair group using a prebuilt second JSONpath expression to obtain all key elements contained in the first key-value pair group and the second key-value pair group and the value corresponding to each key element; and obtaining the path name of the value corresponding to each key element using a prebuilt third JSONpath expression.

[0009] Optionally, determining and outputting a target data stream based on the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream includes: splicing the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream to obtain a JSON data stream; processing the JSON data stream according to pre-built filtering rules to obtain a target data stream; and outputting the target data stream to a target database.

[0010] Optionally, processing the JSON data stream according to pre-constructed filtering rules to obtain a target data stream includes: parsing the JSON data stream to obtain multiple key-value pairs in the JSON data stream; filtering the multiple key-value pairs using pre-constructed filtering rules to obtain multiple remaining key-value pairs; and combining the multiple remaining key-value pairs according to a preset first combination rule to obtain the target data stream.

[0011] Optionally, obtaining the value and path name of each JSON object in the JSON array, and constructing a corresponding first sub-JSON data stream using the value and path name of each JSON object, includes: obtaining the value of each JSON object in the JSON array; obtaining a corresponding path name based on the value of each JSON object; creating a corresponding third key-value pair based on each value of each JSON object and the path name corresponding to the value; and combining multiple third key-value pairs according to a preset second combination rule to obtain the first sub-JSON data stream corresponding to the JSON object.

[0012] Optionally, the obtaining of JSON data containing JSON objects stored in the source database includes: using the input module group of the real-time data integration pipeline component to obtain a JSON file stored on the file system of the source database; and parsing the JSON file to obtain the JSON data.

[0013] 8. A JSON data synchronization device, comprising:

[0014] A data acquisition module is used to acquire JSON data containing JSON objects stored in a source database;

[0015] A processing module, configured to process the JSON object by using a preconfigured real-time data integration pipeline component, and obtain a first key-value pair group, a second key-value pair group, and a path name corresponding to each value of each layer of the JSON object, wherein the first key-value pair group includes a JSON array having a plurality of JSON objects, and the second key-value pair group includes all other key-value pairs except the JSON array;

[0016] A first sub-data stream generating module, configured to obtain a value and a path name of each JSON object of the first key-value pair group, and generate a first sub-JSON data stream of each JSON object using the value and the path name;

[0017] A second sub-data stream generating module is used to obtain the value and path name of the second key-value pair group, construct a second sub-JSON data stream according to the value and path name of the second key-value pair group, and copy the same number of second sub-JSON data streams as the number of the first sub-JSON data streams;

[0018] An output module is used to determine and output a target data stream according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream.

[0019] In addition, to achieve the above objectives, the present application also provides a computer-readable storage medium, which includes instructions, and when the instructions are executed on a computer, the computer executes any of the JSON data synchronization methods described above.

[0020] In addition, to achieve the above-mentioned purpose, the present application also provides a data synchronization device, which includes: at least one processor, a memory and an input-output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the JSON data synchronization method according to any one of the above items. The embodiment of the present application proposes a JSON data synchronization method, device, storage medium and data synchronization device, which obtains JSON data containing JSON objects stored in a source database; uses a preconfigured real-time data integration pipeline component to process the JSON object, obtains the first key-value pair group of each layer of the JSON object and the path name corresponding to each value of the first key-value pair group, and the second key-value pair group and the path name corresponding to each value of the second key-value pair group, wherein the first key-value pair group includes a JSON array with multiple JSON objects, and the second key value includes all other key-value pairs except the JSON array; obtains the value and path name of each JSON object in the JSON array, and uses the value and path name of each JSON object to construct a corresponding first sub-JSON data stream; obtains the value and path name of the second key-value pair group, root A second sub-JSON data stream is constructed according to the value and path name of the second key-value pair group, and the same number of second sub-JSON data streams are copied according to the number of the first sub-JSON data stream; a target data stream is determined and output according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream, thereby realizing flattening processing of JSON data with JSON arrays, and utilizing a preconfigured real-time data integration pipeline component to process JSON objects, distinguishing between a first key-value pair group with a JSON array and a second key-value pair group without a JSON array, and then flattening the first key-value pair group with a JSON array and the second key-value pair group without a JSON array and merging them to obtain a target data stream, thereby solving the technical problem of heavy workload in extracting useful JSON data and synchronizing with SQL statements when synchronizing JSON data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flowchart of a JSON data synchronization method according to an embodiment of the present application;

[0022] Figure 2 This is a structural block diagram of a JSON data synchronization device according to an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of a storage medium provided in one embodiment of the present application;

[0024] Figure 4 A schematic diagram of the structure of a data synchronization device provided in one embodiment of the present application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] Figure 1 This is a flowchart of a JSON data synchronization method according to an embodiment of the present application. Figure 1 , JSON data synchronization methods can include:

[0028] S10, obtaining JSON data containing JSON objects stored in the source database;

[0029] It should be noted that JSON (JavaScript Object Notation) is a lightweight data exchange format. JSON data is easy for people to read and write, and is also easy for machines to parse and generate. JSON data containing JSON objects means that one or more JSON objects are nested in JSON format data. JSON objects are composite structures consisting of key-value pairs, where the key is a string and the value can be a string, a number, a Boolean value, an array, a null value (null) or another JSON object.

[0030] In an exemplary embodiment, step S10 may include the following implementation steps:

[0031] S11. Using the input module group of the real-time data integration pipeline component, obtain a JSON file stored in the file system of the source database;

[0032] Specifically, the input module group of the real-time data integration pipeline component is used to connect to the source database and query the data containing the JSON object. The query process is usually completed by executing an SQL query or using a database API.

[0033] Exemplarily, the input module group of the real-time data integration pipeline component (SeaTunnel) is used to obtain the JSON file saved on the source database file system, and the source database file system includes a local file system, HDFS, Minio or SFTP file system, wherein HDFS refers to the Hadoop distributed file system, which is designed to be a distributed file system (Distributed File System) suitable for running on commodity hardware; Minio is an object storage service based on the Apache License v2.0 open source protocol, Minio is a very lightweight service that can be easily combined with other applications, such as Node JS, Redis or MySQL; SFTP refers to the SSH file transfer protocol, which can ensure the use of private and secure data streams to securely transmit data.

[0034] S11, parse the JSON file to obtain JSON data;

[0035] Specifically, this application uses the SeaTunnel tool to obtain a JSON file saved in a local file system, HDFS, Minio, or SFTP file system. SeaTunnel can define the received data source, data processing process, and data sending target according to the configuration file. Exemplarily, SeaTunnel can create a configuration file that contains input, conversion, and output parts, and then parse multi-source JSON data from the JSON file. SeaTunnel can implement data extraction, transformation, and loading (ETL) operations through pre-written configuration files.

[0036] In an exemplary embodiment, before step S10, the JSON data synchronization method may further include the following steps:

[0037] S60. In response to the data synchronization request, an extended JSONpath component is added to the real-time data integration pipeline component to obtain a preconfigured real-time data integration pipeline component.

[0038] In this embodiment, the specific execution steps of step S60 may include:

[0039] S61. Use the visualization component to add a JSONPath extension plug-in to the processing module group of the real-time data integration pipeline component.

[0040] S62. A preconfigured real-time data integration pipeline component is obtained based on the processing module group, input module group and output module group to which the JSONPath extension plug-in is added.

[0041] Specifically, the processing module group can use the JSONPath parser to match the JSONPath extension plug-in, and the JSONPath parser is responsible for parsing the JSONPath expression. The JSONPath parser can extract the required data fields or objects according to the predefined path. In the embodiment of the present application, the key-value pairs of each level of the JSON object are obtained by cooperating with the JSONPath parser through the JSONPath plug-in, so as to lay the foundation for flattening the JSON data.

[0042] It should be noted that JSONPath is a query language that can be used to locate and extract specific data elements from JSON data. JSONPath can specify one or more path expressions to match specific parts of JSON data. These path expressions are composed of a series of operators, wildcards, and attribute keys, where operators, wildcards, and attribute keys are used to describe the location and structure of the required data. JSONPath can also support filters and filter out data that meets the requirements based on conditions. By using JSONPath, the embodiment of the present application can more efficiently process and parse JSON data while reducing redundant code and manual traversal work. In another embodiment of the present application, JSONPath can also increase the details of data processing, thereby better meeting the customization needs of customers. Since the JSONPath plug-in of the pre-configured real-time data integration pipeline component SeaTunnel does not directly support the function of extracting key-value pairs in a specific order, it is necessary to extend the JSONPath plug-in so that the JSONPath extension plug-in can parse JSON data and identify array key-value pairs and non-array key-value pairs.

[0043] S20, using a preconfigured real-time data integration pipeline component to process the JSON object, obtain a first key-value pair group of each layer of the JSON object and a path name corresponding to each value of the first key-value pair group, as well as a second key-value pair group and a path name corresponding to each value of the second key-value pair group, wherein the first key-value pair group includes a JSON array having a plurality of JSON objects, and the second key-value pair group includes all other key-value pairs except the JSON array;

[0044] Among them, the JSONPath extension plug-in can first obtain the key-value pairs of the first layer of JSON data, and then determine whether the key-value pair belongs to the first key-value pair or the second key-value pair, classify multiple first key-value pairs into a first key-value pair group, and classify multiple second key-value pairs into a second key-value pair group, where the first key-value pair refers to a JSON array with multiple JSON objects, and the second key-value pair refers to other key-value pairs that do not include a JSON array. The JSONPath extension plug-in can also output the path name of each value of the first key-value pair group and the path name of each value of the second key-value pair group.

[0045] In an embodiment of the present application, the implementation steps of step S20 may specifically include:

[0046] S21, obtaining a first key-value pair and a second key-value pair of the first layer of the JSON object;

[0047] S22. Process the second key-value pair group using the pre-built first JSON path expression to obtain all key-value pairs of the second key-value pair group;

[0048] The first pre-built JSONpath expression may be, for example, $[*].key1, which selects all key-value pairs (*) at the first level ($), and then obtains the first key of each key-value pair through .key1.

[0049] S23. Process the first key-value pair using a pre-built second JSON path expression to obtain all key elements contained in the first key-value pair group and the value corresponding to each key element;

[0050] Assuming that the keys of the first layer are key1 and key2, the pre-built second JSONpath expression can be:

[0051] $[? (@.key1)].key1[*],$[? (@.key2)].key2[*].

[0052] For the second-level key-value pairs, the second JSONpath expression can first get the first-level keys of the JSON array, and then drill down to the corresponding values. $[? (@.key1)] is a filter that selects all key-value pairs containing key1. .key1[*] will get all the values ​​of key1 under these key-value pairs. The same logic applies to key2, which will not be repeated here.

[0053] S24. Use a pre-built third JSON path expression to obtain the path name of the value corresponding to each of the key elements.

[0054] The third JSONpath expression may be:

[0055] $.store.book[*].title,$.store.book[*].author.

[0056] This expression means:

[0057] $.store.book[*] - selects all elements in the book array under the store object.

[0058] $.store.book[*].title - selects the title attribute from these elements.

[0059] $.store.book[*].author - selects the author attribute from these elements.

[0060] In this way, JSONpath can get the path name of the title and author values ​​corresponding to each book element.

[0061] Exemplarily, when the processing module group of the real-time data integration pipeline component adds the JSONPath extension plug-in, other levels of data can be extracted from the JSON data through the JSONPath field setting. For example, when the processing module group processes the following JSON data, in order to obtain the three keys of the child level, the processing module group can choose to configure it as: ["address.state", "$address.zip"]. For the key "zip", the processing module group can map its field type to Decimal(3,1). If the processing module group needs to parse the key type in the array phone, the processing module group can use the following configuration mode: ["$phone[*].type"], which will return two values, "mobile" and "home".

[0062] S30, obtaining the value and path name of each JSON object in the JSON array, and constructing each first sub-JSON data stream of the JSON array using the value and path name of each JSON object;

[0063] In an embodiment of the present application, the specific implementation steps of step S30 may include:

[0064] S31. Get the value of each JSON object in the JSON array;

[0065] S32. Obtain a corresponding path name based on the value of each JSON object;

[0066] S33. Based on each value of each JSON object and the path name corresponding to the value, create a corresponding third key-value pair;

[0067] S34. Combine the plurality of third key-value pairs according to a preset second combination rule to obtain the first sub-JSON data stream corresponding to the JSON object.

[0068] By executing steps S31-S34, the processing module group can traverse each JSON object in the first key-value pair group according to the expanded JSONPath, the processing module group records the key and value of each object and extracts the path name corresponding to each value of each JSON object, and then, for any JSON object, the processing module group can use the path name corresponding to each value and the value to construct a third key-value pair, and use the pre-configured second combination rule to combine multiple third key-value pairs to obtain the first sub-JSON data stream of the JSON object. For example, for a JSON array [{"name":"John","age":30},{"name":"Alice","age":25}], the processing module group can create a sub-JSON data stream for each object, and for the first object, two first sub-JSON data streams {"name":"John"},{"age":30}, and {"name":"Alice"},{"age":25} can be created. It should be noted that the above example only involves one JSON array, and the path involved in the JSON array is relatively simple. In other embodiments, the JSON array can also be multiple including complex paths. The second combination rule is to combine the values ​​of the same attributes together.

[0069] S40, obtaining the value and path name of the second key-value pair, constructing a second sub-JSON data stream according to the value and path name of the second key-value pair, and copying the same number of second sub-JSON data streams according to the number of first sub-JSON data streams;

[0070] Specifically, for the second key-value pair group that does not include a JSON array, the processing module group can use JSONPath to traverse all key-value pairs in the second key-value pair group, the processing module group records the key and value and extracts the path name corresponding to each value, and then the processing module group can construct multiple fourth key-value pairs according to the path name and value, and combine the multiple fourth key-value pairs using the preset third combination rule to obtain a second sub-JSON data stream. The processing module group copies the same number of second sub-JSON data streams according to the number of the first sub-JSON data stream. For example, continuing to illustrate the above example, assuming that the second key-value pair group is {"location":"New York"}, the processing module group can create a second sub-data stream {"location":"NewYork"}, and then the processing module group can copy this second sub-data stream twice. The third combination rule can determine the position of each fourth key-value pair in the second sub-JSON data stream in turn according to the arrangement position of each fourth key-value pair in the original JSON data. The third combination rule can be the same as the first combination rule.

[0071] S50, determining and outputting a target data stream according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream.

[0072] The processing module group may combine the first sub-JSON data stream and the copied second sub-JSON data stream to form a target data stream. For example, the target data stream may be: [

[0074] {"name":"John","age":30,"location":"New York"},

[0075] {"name":"Alice","age":25,"location":"New York"}

[0076] ].

[0077] In an embodiment of the present application, the specific execution steps of step S50 may include:

[0078] S51, splicing the first sub-JSON data stream and the second sub-JSON data stream corresponding to each other to obtain a JSON data stream;

[0079] Specifically, the processing module group can splice the second sub-JSON data stream before and after each first sub-JSON data stream to obtain a JSON data stream, or insert the second sub-JSON data stream in the middle of each first sub-JSON data stream to obtain a JSON data stream, which is not limited here.

[0080] S52, processing the JSON data stream according to the pre-built filtering rules to obtain the target data stream;

[0081] The pre-built filtering rules may be defined based on business requirements or data quality requirements, and the filtering rules may include checking the value of a specific field, excluding records that do not meet the conditions, converting the field format, etc., which are not limited here. The processing module group processes the JSON data stream according to the pre-built filtering rules to obtain the target data stream.

[0082] In an embodiment of the present application, step S52 may include the following specific implementation steps:

[0083] Step S521, parsing the JSON data stream to obtain a plurality of key-value pairs in the JSON data stream;

[0084] Step S522: Filter the plurality of key-value pairs using pre-built filtering rules to obtain a plurality of remaining key-value pairs;

[0085] Step S523: Combine the plurality of remaining key-value pairs according to a preset first combination rule to obtain a target data stream.

[0086] The first combination rule includes arranging the remaining key-value pairs in alphabetical order to obtain the target data stream.

[0087] S53: Output the target data stream to the target database.

[0088] Specifically, the processing module group can establish a connection according to the type and API of the target database, and then insert the target data stream as a record into the target database. This process can adopt batch insertion operations to improve efficiency. During the insertion process, the processing module group can monitor and handle any errors that may occur, such as network problems, database connection problems, or data format problems. Finally, after the target data stream is successfully inserted into the database, the processing module group can confirm that the output operation has been completed and check whether the integrity and accuracy of the data are as expected.

[0089] A JSON data synchronization method proposed in an embodiment of the present application obtains JSON data containing JSON objects stored in a source database; uses a preconfigured real-time data integration pipeline component to process the JSON objects, obtains a first key-value pair group of each layer of the JSON objects and a path name corresponding to each value of the first key-value pair group, and a second key-value pair group and a path name corresponding to each value of the second key-value pair group, wherein the first key-value pair group includes a JSON array having multiple JSON objects, and the second key value includes all other key-value pairs except the JSON array; obtains the value and path name of each JSON object in the JSON array, and constructs a corresponding first sub-JSON data stream using the value and path name of each JSON object; obtains the value and path name of the second key-value pair group, and constructs a corresponding first sub-JSON data stream according to the value and path name of the second key-value pair group. The second sub-JSON data stream is constructed based on the value and path name, and the same number of second sub-JSON data streams are copied according to the number of the first sub-JSON data stream; the target data stream is determined and output according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream, thereby realizing the flattening processing of JSON data with JSON arrays, and processing JSON objects with the preconfigured real-time data integration pipeline component to distinguish the first key-value pair group with the JSON array and the second key-value pair group without the JSON array, and then the first key-value pair group with the JSON array and the second key-value pair group without the JSON array are flattened and merged to obtain the target data stream, thereby solving the technical problem of the large workload of extracting useful JSON data and synchronizing with SQL statements when synchronizing JSON data.

[0090] Based on the above embodiments, the present application also provides a JSON data synchronization device. Figure 2 This is a structural diagram of a JSON data synchronization device according to an embodiment of the present application. Figure 2 , the JSON data synchronization device can be used to solve the technical problem of the large workload of extracting useful JSON data and synchronizing it with the help of SQL statements when synchronizing JSON data. The JSON data synchronization device 1000 may include a data acquisition module 100, a processing module 200, a first sub-data stream generation module 300, a second sub-data stream generation module 400 and an output module 500, wherein the data acquisition module 100 can be used to obtain JSON data containing JSON objects stored in a source database; the processing module 200 can be used to process JSON objects using a preconfigured real-time data integration pipeline component to obtain the first key-value pair group, the second key-value pair group and the path name corresponding to each value of each layer of the JSON object, wherein the first key-value pair group includes a JSON array with multiple JSON objects, and the second ... All other key-value pairs outside the array; the first sub-data stream generation module 300 can be used to obtain the value and path name of each JSON object of the first key-value pair group, and use the value and path name to generate the first sub-JSON data stream of each JSON object; the second sub-data stream generation module 400 can be used to obtain the value and path name of the second key-value pair group, construct the second sub-JSON data stream according to the value and path name of the second key-value pair group, and copy the same number of second sub-JSON data streams as the number of first sub-JSON data streams; the output module 500 can be used to determine the target data stream and output it according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream.

[0091] In an exemplary embodiment, the JSON data synchronization device may further include a preconfiguration module 600, wherein the preconfiguration module 600 may be used to add an extended JSONpath component to a real-time data integration pipeline component in response to a data synchronization request to obtain a preconfigured real-time data integration pipeline component.

[0092] In an exemplary embodiment, the processing module 200 can be specifically used to process a JSON object using a prebuilt first JSONpath expression to obtain all key-value pairs of the first layer of the JSON object, and obtain a first key-value pair group and a second key-value pair group based on all the key-value pairs of the first layer; process the first key-value pair group and the second key-value pair group using a prebuilt second JSONpath expression to obtain all key elements contained in the first key-value pair group and the second key-value pair group and the value corresponding to each key element; and obtain the path name of the value corresponding to each key element using a prebuilt third JSONpath expression.

[0093] In an exemplary embodiment, the output module 500 can be specifically used to splice the one-to-one corresponding first sub-JSON data stream and the second sub-JSON data stream to obtain a JSON data stream; process the JSON data stream according to pre-built filtering rules to obtain a target data stream; and output the target data stream to a target database.

[0094] In an exemplary embodiment, the output module 500 can also be used to parse the JSON data stream to obtain multiple key-value pairs of the JSON data stream; filter the multiple key-value pairs using pre-built filtering rules to obtain multiple remaining key-value pairs; and combine the multiple remaining key-value pairs according to a preset first combination rule to obtain the target data stream.

[0095] In an exemplary embodiment, the first sub-data stream generation module 300 can be specifically used to obtain the value of each JSON object in the JSON array; obtain the corresponding path name based on each value; create a corresponding third key-value pair based on each value and the path name corresponding to the value; combine multiple third key-value pairs according to a preset second combination rule to obtain the first sub-JSON data stream corresponding to the JSON object.

[0096] In an exemplary embodiment, the data acquisition module 100 can be specifically used to acquire a JSON file stored on a source database file system by using an input module group of a real-time data integration pipeline component; and parse the JSON file to obtain JSON data.

[0097] Based on the above embodiments, the present application also provides a computer-readable storage medium, referring to Figure 3, the computer-readable storage medium shown is a CD 50, on which a computer algorithm (i.e., an algorithm product) is stored. When the computer algorithm is executed by the processor, the steps recorded in the above method implementation are implemented, for example, obtaining JSON data containing JSON objects stored in a source database; using a preconfigured real-time data integration pipeline component to process the JSON object, obtaining a first key-value pair group of each layer of the JSON object and a path name corresponding to each value of the first key-value pair group, and a second key-value pair group and a path name corresponding to each value of the second key-value pair group, wherein the first key-value pair group includes a JSON array having multiple JSON objects, and the second key value includes all other key-value pairs except the JSON array; obtaining the value and path name of each JSON object in the JSON array, and constructing a corresponding first sub-JSON data stream using the value and path name of each JSON object; obtaining the value and path name of the second key-value pair group, constructing a second sub-JSON data stream according to the value and path name of the second key-value pair group, and copying the same number of second sub-JSON data streams according to the number of the first sub-JSON data streams; determining and outputting a target data stream according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream, and the specific implementation method of each step is not repeated here.

[0098] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0099] In addition, based on the above embodiment, the embodiment of the present application further provides a data synchronization device, Figure 4 A block diagram of an exemplary data synchronization device 60 suitable for implementing the embodiments of the present application is shown. The data synchronization device 60 may be a computer system or a server. Figure 4 The data synchronization device 60 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0100] refer to Figure 4 , the components of the data synchronization device 60 may include but are not limited to: one or more processors or processing units 601, a system memory 602, and a bus 603 connecting different system components (including the system memory 602 and the processing unit 601).

[0101] The data synchronization device 60 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the data synchronization device 60, including volatile and non-volatile media, removable and non-removable media.

[0102] The system memory 602 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 6021 and / or cache memory 6022. The data synchronization device 60 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the ROM 6023 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 is not shown in the Figure 4 As shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to a bus 603 connecting different system components through one or more data medium interfaces. The system memory 602 may include at least one algorithm product, which has a set (e.g., at least one) algorithm module, which is configured to perform the functions of each embodiment of the present application.

[0103] An algorithm / utility 6025 having a set (at least one) of algorithm modules 6024 may be stored, for example, in system memory 602, and such algorithm modules 6024 include, but are not limited to, an operating system, one or more application algorithms, other algorithm modules, and algorithm data, each of which or some combination may include an implementation of a network environment. Algorithm modules 6024 generally perform the functions and / or methods of the embodiments described herein.

[0104] The data synchronization device 60 may also communicate with one or more external devices 604 (e.g., a keyboard, a pointing device, a display, etc.). Such communication may be performed via an input / output (I / O) interface 605. Furthermore, the data synchronization device 60 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 606. Figure 4 As shown, the network adapter 606 communicates with other modules (such as the processing unit 601, etc.) of the data synchronization device 60 via the bus 603 that connects different system components. Figure 4 Not shown, other hardware and / or software program modules may be used in conjunction with the data synchronization device 60 .

[0105] The processing unit 601 executes various functional applications and data processing by running the algorithms stored in the system memory 602, for example, obtaining JSON data containing JSON objects stored in the source database; using the preconfigured real-time data integration pipeline component to process the JSON objects, obtaining the first key-value pair group of each layer of the JSON objects and the path name corresponding to each value of the first key-value pair group, as well as the second key-value pair group and the path name corresponding to each value of the second key-value pair group, wherein the first key-value pair group includes a JSON array having multiple JSON objects, and the second key value includes all other key-value pairs except the JSON array; obtaining the value and path name of each JSON object in the JSON array, and constructing a corresponding first sub-JSON data stream using the value and path name of each JSON object; obtaining the value and path name of the second key-value pair group, constructing a second sub-JSON data stream according to the value and path name of the second key-value pair group, and copying the same number of second sub-JSON data streams according to the number of the first sub-JSON data streams; determining and outputting the target data stream according to the one-to-one corresponding first sub-JSON data stream and the second sub-JSON data stream. The specific implementation method of each step will not be repeated here. The specific implementation method of each step will not be repeated here. It should be noted that, although several units / modules or sub-units / sub-modules of the motion trajectory processing device of the robot are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0106] In the description of the present application, it should be noted that the terms "first", "second" and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0111] If the function is implemented in the form of a software program functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software program product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store algorithm codes.

[0112] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0113] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A JSON data synchronization method, characterized in that: include: Get the JSON data containing JSON objects saved in the source database; Processing the JSON object using a preconfigured real-time data integration pipeline component to obtain a first key-value pair group of each layer of the JSON object and a path name corresponding to each value of the first key-value pair group, as well as a second key-value pair group and a path name corresponding to each value of the second key-value pair group, wherein the first key-value pair group includes a JSON array having a plurality of JSON objects, and the second key-value pair group includes all other key-value pairs except the JSON array; Obtaining a value and a path name of each JSON object in the JSON array, and constructing a corresponding first sub-JSON data stream using the value and the path name of each JSON object; Obtain the value and path name of the second key-value pair, construct a second sub-JSON data stream according to the value and path name of the second key-value pair, and copy the same number of second sub-JSON data streams according to the number of the first sub-JSON data streams; The target data stream is determined and output according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream.

2. A JSON data synchronization method as claimed in claim 1, characterized in that: Before acquiring the JSON data containing the JSON object stored in the source database, the JSON data synchronization method further includes: In response to the data synchronization request, the extended JSONpath component is added to the real-time data integration pipeline component to obtain the preconfigured real-time data integration pipeline component.

3. A JSON data synchronization method as claimed in claim 2, characterized in that: The method of using a preconfigured real-time data integration pipeline component to process the JSON object to obtain a first key-value pair group of each layer of the JSON object and a path name corresponding to each value of the first key-value pair group, and a second key-value pair group and a path name corresponding to each value of the second key-value pair group includes: Processing the JSON object using a pre-built first JSONpath expression to obtain all key-value pairs of the first layer of the JSON object, and obtaining the first key-value pair group and the second key-value pair group according to all the key-value pairs of the first layer; Processing the first key-value pair and the second key-value pair by using a pre-built second JSON path expression to obtain all key elements contained in the first key-value pair and the second key-value pair and the value corresponding to each key element; The pre-built third JSON path expression is used to obtain the path name of the value corresponding to each key element.

4. A JSON data synchronization method as claimed in claim 1, characterized in that: The determining and outputting a target data stream according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream comprises: splicing the first sub-JSON data stream and the second sub-JSON data stream corresponding to each other one by one to obtain a JSON data stream; Process the JSON data stream according to pre-built filtering rules to obtain a target data stream; The target data stream is output to a target database.

5. A JSON data synchronization method as claimed in claim 4, characterized in that: The step of processing the JSON data stream according to the pre-built filtering rules to obtain the target data stream includes: Parsing the JSON data stream to obtain a plurality of key-value pairs of data in the JSON data stream; Filtering the plurality of key-value pair data using a pre-built filtering rule to obtain a plurality of remaining key-value pair data; The target data stream is obtained by combining the plurality of remaining key-value pairs according to a preset first combination rule.

6. A JSON data synchronization method as claimed in claim 1, characterized in that: The obtaining of the value and path name of each JSON object in the JSON array, and constructing a corresponding first sub-JSON data stream using the value and path name of each JSON object, includes: Get the value of each JSON object in the JSON array; Get the corresponding path name based on the value of each JSON object; Based on the value of each JSON object and the corresponding path name, create a corresponding third key-value pair; The plurality of third key-value pairs are combined according to a preset second combination rule to obtain the first sub-JSON data stream corresponding to the JSON object.

7. A JSON data synchronization method as claimed in claim 1, characterized in that: The step of obtaining JSON data containing JSON objects stored in the source database includes: Using the input module group of the real-time data integration pipeline component to obtain the JSON file stored on the source database file system; The JSON file is parsed to obtain the JSON data.

8. A JSON data synchronization device, characterized in that: include: A data acquisition module is used to acquire JSON data containing JSON objects stored in a source database; A processing module, configured to process the JSON object by using a preconfigured real-time data integration pipeline component, and obtain a first key-value pair group, a second key-value pair group, and a path name corresponding to each value of each layer of the JSON object, wherein the first key-value pair group includes a JSON array having a plurality of JSON objects, and the second key-value pair group includes all other key-value pairs except the JSON array; A first sub-data stream generating module, configured to obtain a value and a path name of each JSON object of the first key-value pair group, and generate a first sub-JSON data stream of each JSON object using the value and the path name; A second sub-data stream generating module is used to obtain the value and path name of the second key-value pair group, construct a second sub-JSON data stream according to the value and path name of the second key-value pair group, and copy the same number of second sub-JSON data streams as the number of the first sub-JSON data streams; An output module is used to determine and output a target data stream according to the one-to-one correspondence between the first sub-JSON data stream and the second sub-JSON data stream.

9. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the JSON data synchronization method according to any one of claims 1 to 7.

10. A data synchronization device, characterized in that: The data synchronization device comprises: at least one processor, memory, and input-output unit; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the JSON data synchronization method according to any one of claims 1 to 7.

Images