A report data synchronization method and device based on DataX

Abstract

The present invention provides a report data synchronization method and device based on DataX. The method initializes paging control parameters based on the data structure of the data to be obtained in the Reader plug-in, thereby cyclically sending data acquisition requests through HTTP requests, cyclically obtaining paging data responses of the paging size from the source end until the termination condition is reached, and parsing the data fields of the nested structure in the paging data response to generate intermediate structured data objects corresponding to each data field. Then, data preprocessing is performed on the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure to form preprocessed data objects. The preprocessed data objects are then passed to the Writer plug-in based on the Reader plug-in, and the preprocessed data objects are cached according to a batch strategy based on the Writer plug-in and submitted to the target data system, thereby improving the synchronization accuracy and scalability of multi-source heterogeneous data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for synchronizing report data based on DataX. Background Art

[0002] With the continuous development of big data technology, enterprises are increasingly demanding the integration of heterogeneous data sources. In traditional manufacturing industries such as steel and metallurgy, the collection, synchronization, and processing of massive amounts of data have become a crucial foundation for supporting digital operations. However, current mainstream data synchronization technologies still face numerous challenges in practical application, particularly in handling heterogeneous data and demonstrating significant deficiencies in system scalability.

[0003] First, data heterogeneity in data synchronization scenarios is becoming increasingly serious. For example, Anshan Iron and Steel Group has built a big data lake platform that provides data services to external systems through APIs. However, the data returned by these APIs is complex, often organized in a multi-layered nested JSON format with deep field paths and irregular structures. The returned JSON objects contain a mixture of multiple nested arrays and map structures. Traditional ETL tools often experience field misalignment and data omissions when expanding these data into two-dimensional structured tables, resulting in synchronization accuracy that fails to meet production requirements.

[0004] Secondly, the existing system lacks scalability. In practice, business systems typically rely on customized Java programs to connect and synchronize various data sources. This point-to-point development model, facing the rapidly growing variety of data sources, lacks a unified synchronization framework and intermediate data standards, requiring repeated development and testing for each new data source. Especially during cross-system integration, field semantics, data accuracy, and time formats differ significantly between different data sources. Manual integration is not only time-consuming and labor-intensive, but also extremely costly to maintain, placing a heavy burden on the operations and maintenance team. Summary of the Invention

[0005] The present invention provides a report data synchronization method and device based on DataX, which are used to solve the defects of insufficient synchronization accuracy and poor system scalability in the prior art.

[0006] The present invention provides a report data synchronization method based on DataX, comprising:

[0007] In the Datax Reader plug-in, paging control parameters are initialized based on the data structure of the data to be acquired; the paging control parameters include an initial page number and a paging size, and a URL address for a data acquisition request for the data to be acquired is constructed; the data to be acquired includes unstructured report-type data;

[0008] Based on the Reader plug-in, the data acquisition request is cyclically sent via HTTP request mode, and paging data responses of the paging size are cyclically obtained from the source end until a preset termination condition is reached. The nested structured data fields in the paging data response are parsed to generate intermediate structured data objects corresponding to each data field; the source end includes multiple different types of data sources;

[0009] Based on the Reader plug-in, data preprocessing is performed on the intermediate structured data objects corresponding to the nested structured data fields and the non-nested structured data fields to form preprocessed data objects;

[0010] Based on the Reader plug-in, the pre-processed data object is passed to the Writer plug-in of datax, and based on the Writer plug-in, the pre-processed data object is cached according to the batch strategy and submitted to the target data system.

[0011] According to a datax-based report data synchronization method provided by the present invention, parsing the nested data fields in the paging data response to generate intermediate structured data objects corresponding to each data field includes:

[0012] Constructing a complete field path tree of the data fields of the nested structure;

[0013] For any data field of the nested structure, based on the field name and field type of each node in the complete field path tree of the data field and the order between the nodes, determine the intermediate data structure corresponding to the data field;

[0014] Based on the intermediate data structure corresponding to any one of the data fields, an intermediate structured data object corresponding to any one of the data fields is generated.

[0015] According to a report data synchronization method based on DataX provided by the present invention, determining the intermediate data structure corresponding to any data field based on the field name and field type of each node in the complete field path tree of any data field and the order between the nodes includes:

[0016] Extracting a field semantic vector representation of a corresponding node based on the field name and field type of each node in the complete field path tree of any data field;

[0017] Based on the field semantic vector representation of each node in the complete field path tree of any data field and the order between each node, as well as the field semantic vector representation of each field contained in each standard nested data structure and the order between each field, matching is performed to determine that the standard nested data structure with the highest matching degree is the intermediate data structure corresponding to any data field.

[0018] According to a report data synchronization method based on DataX provided by the present invention, the reader plug-in performs data preprocessing on intermediate structured data objects corresponding to nested structured data fields and non-nested structured data fields to form preprocessed data objects, including:

[0019] Determining, based on the field type of the data field of the non-nested structure, an intermediate data type corresponding to the data field of the non-nested structure;

[0020] Convert the corresponding data field based on the intermediate data type corresponding to the data field of the non-nested structure to obtain the intermediate type data object corresponding to the data field of the non-nested structure;

[0021] Based on the type of the target data system, type conversion is performed on the intermediate structured data objects corresponding to the data fields of the nested structure and the intermediate type data objects corresponding to the data fields of the non-nested structure to obtain preprocessed data objects corresponding to each data field.

[0022] According to a report data synchronization method based on DataX provided by the present invention, the intermediate structured data objects corresponding to the data fields of the nested structure and the intermediate type data objects corresponding to the data fields of the non-nested structure are converted into types to obtain pre-processed data objects corresponding to the respective data fields, and then the method further includes:

[0023] Performing anomaly detection on the pre-processed data object corresponding to each data field to obtain anomaly detection results for each data field; wherein the anomaly detection result for any data field includes whether any data field has an anomaly, and the type of anomaly when any data field has an anomaly;

[0024] Based on the exception handling strategy for each exception type in the preset strategy library, the pre-processed data object corresponding to the data field with the exception is repaired, so as to pass the pre-processed data object corresponding to the data field with the exception after the repair to the Writer plug-in of datax.

[0025] According to a datax-based report data synchronization method provided by the present invention, the step of cyclically obtaining a paging data response of the paging size from a source end until a preset termination condition is reached includes:

[0026] updating the paging size based on the field type of the data field in the paging data response returned by the current request, the data volume of the paging data response, and the request response time, so as to send the next request based on the updated paging size;

[0027] Based on the identification field in the paging data response returned by the current request, it is dynamically determined whether a preset termination condition is reached.

[0028] According to a datax-based report data synchronization method provided by the present invention, the pre-processed data objects are cached in batches and submitted to a target data system, including:

[0029] Cache the pre-processed data objects corresponding to each data field in a memory buffer queue;

[0030] Based on a set batch quantity or time window, the pre-processed data objects in the memory buffer queue are written in batches to the target data system.

[0031] The present invention also provides a report data synchronization device based on DataX, comprising:

[0032] An initialization unit is configured to initialize paging control parameters in the datax Reader plug-in based on the data structure of the data to be acquired; the paging control parameters include an initial page number and a paging size, and to construct a URL address for a data acquisition request for the data to be acquired; the data to be acquired includes unstructured report-type data;

[0033] A data acquisition and parsing unit is configured to cyclically send the data acquisition request via HTTP request based on a Reader plug-in, cyclically obtain paging data responses of the paging size from the source end until a preset termination condition is reached, and parse the data fields of the nested structure in the paging data response to generate intermediate structured data objects corresponding to each data field; the source end includes multiple different types of data sources;

[0034] A data preprocessing unit is used to perform data preprocessing on the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure based on the Reader plug-in to form preprocessed data objects;

[0035] The data batch writing unit is used to transfer the pre-processed data objects to the Writer plug-in of datax based on the Reader plug-in, and cache the pre-processed data objects according to the batch strategy based on the Writer plug-in and submit them to the target data system.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-described datax-based reporting data synchronization methods.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described datax-based reporting data synchronization methods.

[0038] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described datax-based reporting data synchronization methods.

[0039] The present invention provides a report data synchronization method and device based on DataX. The method and device initialize paging control parameters based on the data structure of the data to be obtained in the Reader plug-in of DataX, thereby cyclically sending data acquisition requests through HTTP requests based on the Reader plug-in, cyclically obtaining paging data responses of the paging size from the source end until a preset termination condition is reached, and parsing the nested structured data fields in the paging data response to generate intermediate structured data objects corresponding to each data field. Then, based on the Reader plug-in, data preprocessing is performed on the intermediate structured data objects corresponding to the nested structured data fields and the non-nested structured data fields to form preprocessed data objects. The preprocessed data objects are then passed to the Writer plug-in of DataX based on the Reader plug-in, and the preprocessed data objects are cached according to a batch strategy based on the Writer plug-in and submitted to the target data system, thereby improving the synchronization accuracy and scalability of multi-source heterogeneous data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a flowchart of the datax-based report data synchronization method provided by the present invention;

[0042] Figure 2 It is a flowchart of the method for parsing nested structure data fields provided by the present invention;

[0043] Figure 3 It is a structural diagram of a report data synchronization device based on DataX provided by the present invention;

[0044] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0046] Figure 1 This is a flow chart of the data synchronization method for reporting based on DataX provided by the present invention. Figure 1 As shown, the method includes:

[0047] Step 110: In the datax Reader plug-in, paging control parameters are initialized based on the data structure of the data to be acquired; the paging control parameters include an initial page number and a paging size, and a URL address for a data acquisition request for the data to be acquired is constructed; the data to be acquired includes unstructured report data;

[0048] Step 120: Based on the Reader plug-in, the data acquisition request is cyclically sent via HTTP request mode, and paging data responses of the paging size are cyclically obtained from the source end until a preset termination condition is reached. The nested structured data fields in the paging data response are parsed to generate intermediate structured data objects corresponding to the respective data fields. The source end includes multiple different types of data sources.

[0049] Step 130: Preprocess the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure based on the Reader plug-in to form preprocessed data objects.

[0050] Step 140: The pre-processed data object is transferred to the Writer plug-in of datax based on the Reader plug-in, and the pre-processed data object is cached according to a batch strategy based on the Writer plug-in and submitted to the target data system.

[0051] Here, we refactored the Reader plugin in the DataX framework and rewrote the core logic of data acquisition and parsing, the startRead method. We added customized features such as dynamic paging control, nested structure parsing, and type conversion. This allowed us to store the parsed data in the RecordSender object defined by the DataX framework and write the data using the DataX Writer plugin. This refactored Reader plugin now includes the aforementioned features, such as dynamic paging control, nested structure parsing, and type conversion.

[0052] Specifically, when the reconstructed Reader plug-in is initialized, the system will configure appropriate paging control parameters based on the data interface structure that needs to be synchronized, that is, the data structure of the data to be obtained. These parameters may include the initial page number, the paging size, and the paging termination condition identifier. The initial page number is used to define which page to start obtaining data from, and the paging size controls the amount of data requested each time, thus laying the foundation for subsequent batch write operations. The paging termination condition identifier can be flexibly set according to different interface characteristics. For example, the paging termination condition identifier can be set to "has_next_page" or "next_page", so as to determine whether to continue requesting the next page by judging whether the response contains "has_next_page" as false or whether the "next_page" field is empty. Based on this, the Reader plug-in constructs a complete URL address for the request for the data to be obtained, encapsulates the HTTP request information, and prepares for the subsequent cyclic paging acquisition process.

[0053] Next, the Reader plug-in starts to initiate HTTP requests to the source interface in a loop, obtaining data page by page until the identification field in the paging data response returned by the current request indicates that the preset termination condition has been reached. Among them, the source end includes many different types of data sources. Each request will return a paging data response in JSON format, and the paging data response may contain a large number of nested data fields. In view of this feature, the embodiment of the present invention embeds a configurable path parser and structure mapping module in the Reader plug-in, so that the system can accurately extract data areas from complex nested JSON data. For example, in some typical interfaces, the main data may be located in a "list" array under the name "data", and each record therein is nested with multiple substructures such as price information, timestamp, and region fields. Therefore, by configuring the path parser and structure mapping module of the Reader plug-in, the system can accurately locate the data fields of these nested structures, parse them, convert unstructured data into structured data, and thus generate corresponding intermediate structured data objects.

[0054] In some embodiments, in order to improve data acquisition efficiency, after each HTTP request is initiated, the paging control parameter of the paging size can be updated based on the field type of the data field in the paging data response returned by the current request, the data volume of the paging data response, and the request response time, so as to send the next request based on the updated paging size.

[0055] Here, the paging size parameter is not only set at the initial stage, but can also be dynamically adjusted during the execution of the data acquisition loop based on the actual network environment and the characteristics of the returned data, thereby improving the stability and throughput efficiency of data synchronization. After the request is sent, the key characteristics of the returned paging data response can be analyzed. This analysis includes but is not limited to: the amount of data in the paging data response returned this time (such as the actual number of data items, the number of fields in the JSON data structure), the field type of the data field (including the nesting level of data fields in nested structures, the complexity of the field type, such as whether it contains arrays, objects, long text, file encoding fields, etc.), and the time it takes from sending the request to receiving the response (i.e., the request response time).

[0056] Specifically, if the system detects that the field structure in the current paginated data response is complex, including fields with highly structured types such as nested objects and arrays, it will determine that the parsing and normalization of this batch of data will require significant computational resources. The system can then reduce the page size for the next request to avoid system processing bottlenecks. Conversely, if the current response data structure is relatively simple, such as if most fields are strings or numbers, and the number of data entries has not yet reached the paging limit, the system can appropriately increase the page size for the next page, thereby reducing the number of requests and improving overall synchronization efficiency. Request response time is also an important input factor for dynamically adjusting the page size. For example, if the system detects that the response time for a particular page of data is significantly higher than average (e.g., longer than 3 seconds), it may indicate insufficient network bandwidth or high processing pressure on the interface server. In this case, the page size will be reduced to reduce the data volume of the next request, thereby increasing the request success rate and reducing system load. If the system consistently responds to several requests quickly (e.g., each time within 1 second), the page size can be appropriately increased to further improve data fetching efficiency. The above-mentioned adaptive control method of paging size can reduce the performance bottleneck and the number of failed retries when facing data heterogeneity and network uncertainty, and improve the stability, robustness and throughput of the entire data synchronization process.

[0057] In some embodiments, as Figure 2 As shown, the following methods can be used to implement the parsing of nested structure data fields:

[0058] Step 210: construct a complete field path tree of the data fields of the nested structure;

[0059] Step 220: for any data field in the nested structure, based on the field name and field type of each node in the complete field path tree of the data field and the order of each node, determine the intermediate data structure corresponding to the data field;

[0060] Step 230: Generate an intermediate structured data object corresponding to any data field based on the intermediate data structure corresponding to any data field.

[0061] To accurately convert complex JSON structures into intermediate structured data objects, a field path tree can be constructed as a foundation to comprehensively identify the semantic and structural relationships between nested fields. This intermediate data structure is then constructed to facilitate subsequent data standardization and write operations. Since paginated data responses returned by the source often have multi-layered nested structures, such as nested arrays, nested objects, and structures within structures, to effectively parse the data fields of these nested structures, a recursive scan can be performed on each JSON structure in the paginated data response to identify the data fields within the nested structure and construct a complete field path tree for each nested structure. In this path tree, each node represents a field in the data fields of the nested structure, and the node path is determined by the hierarchical order of the fields. The path representation uses the typical "parent field.child field.grandchild field" model. For example, a field "data.details.price" represents the "price" field in the "details" object under the "data" object. The field path tree not only records the name of each field, but also its data type (such as string, array, object, number, etc.) and its position in the hierarchical structure.

[0062] When the complete field path tree is built, according to the field name and field type of each node in the complete field path tree, as well as the order of each node in the complete field path tree, a standard nested data structure (such as Array<String, String> ) that matches the data field of the nested structure is selected from the pre-built standard nested data structure library. <struct>or Map<String, Object> etc.) as its corresponding intermediate data structure. In other embodiments, the field semantic vector representation of the corresponding node can be extracted using a pre-trained language model based on the field name and field type of each node in the complete field path tree of the data field of the nested structure. Subsequently, semantic matching is performed based on the field semantic vector representation of each node in the complete field path tree of the data field and the order between each node, as well as the field semantic vector representation of each field contained in each standard nested data structure (which can be obtained based on the pre-trained language model) and the order between each field, and the standard nested data structure with the highest matching degree is determined to be the intermediate data structure corresponding to the data field.

[0063] After determining the intermediate data structure corresponding to any nested data field, an intermediate structured data object corresponding to that data field can be generated based on the definition of the intermediate data structure. This data object not only retains the hierarchy and type information of the original field, but can also be efficiently converted into a data structure supported by the target data system during subsequent processing, helping to adapt to the differentiated structures of multi-source data interfaces.

[0064] After the nested structure is parsed, the Reader plug-in further preprocesses the extracted data fields to generate clearly structured preprocessed data objects that meet the target data system's format requirements. During this process, the data types of different data fields can be forcibly converted based on the preset field type mapping rules. It should be noted that, prior to this, based on the field type of the non-nested structure data field, the intermediate data type corresponding to the non-nested structure data field can be determined. This intermediate data type is a data type that can be easily converted to the formats of various target data systems. Then, based on the intermediate data type corresponding to the non-nested structure data field, the corresponding data field is converted to obtain the intermediate type data object corresponding to the non-nested structure data field. Next, based on the target data system type (e.g., MySQL, Elasticsearch, HDFS, etc.), the intermediate structured data objects corresponding to each nested structure data field and the intermediate type data objects corresponding to each non-nested structure data field are converted to obtain the preprocessed data object corresponding to each data field that meets the target data system's format requirements.

[0065] In addition, anomaly detection can be performed on the pre-processed data objects corresponding to each data field to obtain anomaly detection results for each data field. The anomaly detection results for any data field include whether there is an anomaly in the data field, and the type of anomaly when an anomaly exists in the data field. Subsequently, based on the exception handling strategies for each anomaly type in the preset strategy library, the pre-processed data objects corresponding to the data fields with anomalies are repaired, further improving the availability and standardization of the overall data. For example, for missing or formatted field values, the default value strategy can be automatically applied to fill them in, and for illegal field values, the discard strategy can be applied.

[0066] After completing the above preprocessing, the Reader plug-in passes the organized preprocessed data objects (or preprocessed data objects after anomalies are corrected) to the datax data pipeline. The datax Writer plug-in receives them and is responsible for writing the preprocessed data objects corresponding to each data field to the target data system. In an embodiment of the present invention, the Writer plug-in can adopt a batch submission optimization strategy, which greatly improves writing efficiency compared to the traditional line-by-line writing method. Specifically, the Writer plug-in maintains a memory buffer queue and temporarily stores the preprocessed data objects corresponding to each data field in the memory buffer queue after receiving them. Only when the cached data volume reaches a preset threshold (for example, every 100 data items) or reaches a preset time window, a unified submission operation is triggered, and the preprocessed data objects in the memory buffer queue are batch-written to the target data system. This mechanism effectively reduces the impact of frequent I / O operations on system performance and significantly improves throughput. In addition, during the submission process, the Writer plug-in can select the optimal data writing method based on the type of target data system, such as partition append for Hive tables, batch insert for MySQL, or the high-speed write interface of DWS. Because the Reader plug-in has standardized field types and formats, the Writer plug-in can efficiently complete data writing, improving the stability and compatibility of data synchronization.

[0067] In addition, in order to enhance the robustness and fault tolerance of the system, the embodiment of the present invention also designs an adaptive exception handling and retry mechanism. In the process of the Reader plug-in initiating a paging request or performing field parsing, if there are abnormal situations such as network interruption, data format inconsistency, field missing, etc., the system will automatically start the retry strategy. This strategy not only includes a limit on the maximum number of retries, but also can dynamically adjust the retry interval to improve the system's recovery capabilities. When certain data fields cannot be processed after multiple attempts, they can be marked as abnormal records and transferred to a dedicated log or intermediate storage area. They can be manually reviewed or re-recorded later to avoid affecting the overall task execution.

[0068] In addition, the synchronization method of the present invention has good scalability. In the case of a new data source or interface change, the user only needs to modify the parameters such as the URL template, field path mapping and data type definition in the Reader plug-in through configuration, without modifying the underlying logic or rewriting the code, thereby significantly reducing maintenance costs. The method also supports a plug-in integrated monitoring mechanism that can record indicators such as the success rate of each request response, the number of fields extracted, and the submission time, providing operation and maintenance personnel with rich observability information, which helps to quickly locate potential problems in the data synchronization process.

[0069] To sum up, the method provided by the embodiment of the present invention initializes the paging control parameters based on the data structure of the data to be obtained in the Reader plug-in of datax, and then based on the Reader plug-in, cyclically sends data acquisition requests through HTTP requests, cyclically obtains paging data responses of the paging size from the source end until a preset termination condition is reached, and parses the data fields of the nested structure in the paging data response to generate intermediate structured data objects corresponding to each data field, and then preprocesses the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure based on the Reader plug-in to form preprocessed data objects, and then passes the preprocessed data objects to the Writer plug-in of datax based on the Reader plug-in, and caches the preprocessed data objects according to the batch strategy based on the Writer plug-in and submits them to the target data system, thereby improving the synchronization accuracy and scalability of multi-source heterogeneous data.

[0070] The following describes the reporting data synchronization device based on DataX provided by the present invention. The reporting data synchronization device based on DataX described below and the reporting data synchronization method based on DataX described above can be referenced to each other.

[0071] Based on any of the above embodiments, Figure 3 This is a structural diagram of a datax-based reporting data synchronization device provided by the present invention. Figure 3 As shown, the device includes:

[0072] Initialization unit 310 is used to initialize paging control parameters based on the data structure of the data to be acquired in the datax Reader plug-in; the paging control parameters include an initial page number and a paging size, and construct a URL address for a data acquisition request for the data to be acquired; the data to be acquired includes unstructured report data;

[0073] The data acquisition and parsing unit 320 is configured to cyclically send the data acquisition request via HTTP request based on the Reader plug-in, cyclically obtain paging data responses of the paging size from the source end until a preset termination condition is reached, and parse the data fields of the nested structure in the paging data response to generate intermediate structured data objects corresponding to each data field; the source end includes multiple different types of data sources;

[0074] A data preprocessing unit 330 is configured to perform data preprocessing on the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure based on the Reader plug-in to form preprocessed data objects;

[0075] The data batch writing unit 340 is used to transfer the pre-processed data object to the Writer plug-in of datax based on the Reader plug-in, and cache the pre-processed data object according to the batch strategy based on the Writer plug-in and submit it to the target data system.

[0076] The device provided by the embodiment of the present invention initializes paging control parameters based on the data structure of the data to be obtained in the Reader plug-in of datax, thereby cyclically sending data acquisition requests through HTTP requests based on the Reader plug-in, cyclically obtaining paging data responses of the paging size from the source end until a preset termination condition is reached, and parsing the data fields of the nested structure in the paging data response to generate intermediate structured data objects corresponding to each data field, and then preprocessing the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure based on the Reader plug-in to form preprocessed data objects, and then passing the preprocessed data objects to the Writer plug-in of datax based on the Reader plug-in, and caching the preprocessed data objects according to the batch strategy based on the Writer plug-in and submitting them to the target data system, thereby improving the synchronization accuracy and scalability of multi-source heterogeneous data.

[0077] Based on any of the above embodiments, parsing the data fields of the nested structure in the paging data response to generate intermediate structured data objects corresponding to each data field includes:

[0078] Constructing a complete field path tree of the data fields of the nested structure;

[0079] For any data field of the nested structure, based on the field name and field type of each node in the complete field path tree of the data field and the order between the nodes, determine the intermediate data structure corresponding to the data field;

[0080] Based on the intermediate data structure corresponding to any one of the data fields, an intermediate structured data object corresponding to any one of the data fields is generated.

[0081] Based on any of the above embodiments, determining the intermediate data structure corresponding to any data field based on the field name and field type of each node in the complete field path tree of the any data field and the order of the nodes includes:

[0082] Extracting a field semantic vector representation of a corresponding node based on the field name and field type of each node in the complete field path tree of any data field;

[0083] Based on the field semantic vector representation of each node in the complete field path tree of any data field and the order between each node, as well as the field semantic vector representation of each field contained in each standard nested data structure and the order between each field, matching is performed to determine that the standard nested data structure with the highest matching degree is the intermediate data structure corresponding to any data field.

[0084] Based on any of the above embodiments, the reader plug-in performs data preprocessing on the intermediate structured data objects corresponding to the nested structured data fields and the non-nested structured data fields to form preprocessed data objects, including:

[0085] Determining, based on the field type of the data field of the non-nested structure, an intermediate data type corresponding to the data field of the non-nested structure;

[0086] Convert the corresponding data field based on the intermediate data type corresponding to the data field of the non-nested structure to obtain the intermediate type data object corresponding to the data field of the non-nested structure;

[0087] Based on the type of the target data system, type conversion is performed on the intermediate structured data objects corresponding to the data fields of the nested structure and the intermediate type data objects corresponding to the data fields of the non-nested structure to obtain preprocessed data objects corresponding to each data field.

[0088] Based on any of the above embodiments, the intermediate structured data objects corresponding to the data fields of the nested structure and the intermediate type data objects corresponding to the data fields of the non-nested structure are converted into types to obtain preprocessed data objects corresponding to the respective data fields, and then further comprising:

[0089] Performing anomaly detection on the pre-processed data object corresponding to each data field to obtain anomaly detection results for each data field; wherein the anomaly detection result for any data field includes whether any data field has an anomaly, and the type of anomaly when any data field has an anomaly;

[0090] Based on the exception handling strategy for each exception type in the preset strategy library, the pre-processed data object corresponding to the data field with the exception is repaired, so as to pass the pre-processed data object corresponding to the data field with the exception after the repair to the Writer plug-in of datax.

[0091] Based on any of the above embodiments, the step of cyclically obtaining a paging data response of the paging size from the source end until a preset termination condition is reached includes:

[0092] updating the paging size based on the field type of the data field in the paging data response returned by the current request, the data volume of the paging data response, and the request response time, so as to send the next request based on the updated paging size;

[0093] Based on the identification field in the paging data response returned by the current request, it is dynamically determined whether a preset termination condition is reached.

[0094] Based on any of the above embodiments, the step of caching the pre-processed data objects in batches and submitting the pre-processed data objects to the target data system includes:

[0095] Cache the pre-processed data objects corresponding to each data field in a memory buffer queue;

[0096] Based on a set batch quantity or time window, the pre-processed data objects in the memory buffer queue are written in batches to the target data system.

[0097] Figure 4 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor (processor) 410, a memory (memory) 420, a communication interface (Communications Interface) 430 and a communication bus 440, wherein the processor 410, the memory 420, and the communication interface 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 420 to execute a report-type data synchronization method based on DataX, which includes: in a Reader plug-in of DataX, initializing paging control parameters based on the data structure of the data to be obtained; the paging control parameters include an initial page number and a paging size, and constructing a URL address of a data acquisition request for the data to be obtained; the data to be obtained includes unstructured report-type data; based on the Reader plug-in, cyclically sending the data acquisition request via HTTP request, cyclically obtaining paging data responses of the paging size from the source end until a preset termination condition is reached, and parsing the nested structured data fields in the paging data response to generate intermediate structured data objects corresponding to each data field; the source end includes multiple different types of data sources; based on the Reader plug-in, data preprocessing is performed on the intermediate structured data objects corresponding to the nested structured data fields and the non-nested structured data fields to form preprocessed data objects; based on the Reader plug-in, the preprocessed data objects are passed to the Writer plug-in of DataX, and based on the Writer plug-in, the preprocessed data objects are cached according to a batch strategy and submitted to the target data system.

[0098] Furthermore, the logic instructions in the aforementioned memory 420 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0099] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the report-type data synchronization method based on datax provided by the above methods, the method including: in the Reader plug-in of datax, initializing the paging control parameters based on the data structure of the data to be acquired; the paging control parameters include the initial page number and the paging size, and constructing a URL address for the data acquisition request for the data to be acquired; the data to be acquired includes unstructured report-type data; based on the Reader plug-in, the data is requested via HTTP The data acquisition request is sent cyclically in a request manner, paging data responses of the paging size are cyclically acquired from the source end until a preset termination condition is reached, and the data fields of the nested structure in the paging data response are parsed to generate intermediate structured data objects corresponding to each data field; the source end includes multiple different types of data sources; based on the Reader plug-in, data preprocessing is performed on the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure to form preprocessed data objects; based on the Reader plug-in, the preprocessed data objects are passed to the Writer plug-in of datax, and based on the Writer plug-in, the preprocessed data objects are cached according to a batch strategy and submitted to the target data system.

[0100] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the above-mentioned report-type data synchronization method based on datax, the method comprising: in the Reader plug-in of datax, initializing the paging control parameters based on the data structure of the data to be acquired; the paging control parameters include the initial page number and the paging size, and constructing the URL address of the data acquisition request for the data to be acquired; the data to be acquired includes unstructured report-type data; based on the Reader plug-in, cyclically sending the data acquisition request through HTTP request mode, The paging data response of the paging size is obtained cyclically until a preset termination condition is reached, and the nested structured data fields in the paging data response are parsed to generate intermediate structured data objects corresponding to each data field; the source end includes multiple different types of data sources; based on the Reader plug-in, the intermediate structured data objects corresponding to the nested structured data fields and the non-nested structured data fields are preprocessed to form preprocessed data objects; based on the Reader plug-in, the preprocessed data objects are passed to the Writer plug-in of datax, and based on the Writer plug-in, the preprocessed data objects are cached according to the batch strategy and submitted to the target data system.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0102] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.< / struct>

Claims

1. A report data synchronization method based on DataX, characterized in that: include: In the Reader plug-in of datax, the paging control parameters are initialized based on the data structure of the data to be obtained; The paging control parameters include an initial page number and a paging size, and construct a URL address of a data acquisition request for the data to be acquired; the data to be acquired includes unstructured report data; Based on the Reader plug-in, the data acquisition request is cyclically sent through the HTTP request method, and the paging data response of the paging size is cyclically obtained from the source end until a preset termination condition is reached, and the data fields of the nested structure in the paging data response are parsed to generate intermediate structured data objects corresponding to each data field; the source end includes multiple different types of data sources; wherein, a complete field path tree of the data field of the nested structure is constructed; for any data field of the nested structure, based on the field name and field type of each node in the complete field path tree of the any data field and the order between each node, the intermediate data structure corresponding to the any data field is determined; based on the intermediate data structure corresponding to the any data field, an intermediate structured data object corresponding to the any data field is generated; the intermediate data structure is a standard nested data structure that matches the any data field, and the intermediate structured data object retains the hierarchy and type information of the any data field; Based on the Reader plug-in, data preprocessing is performed on the intermediate structured data objects corresponding to the nested structured data fields and the non-nested structured data fields to form preprocessed data objects; Based on the Reader plug-in, the pre-processed data object is passed to the Writer plug-in of datax, and based on the Writer plug-in, the pre-processed data object is cached according to the batch strategy and submitted to the target data system.

2. The datax-based reporting data synchronization method according to claim 1, characterized in that: The determining of the intermediate data structure corresponding to any data field based on the field name and field type of each node in the complete field path tree of any data field and the order of the nodes includes: Extracting a field semantic vector representation of a corresponding node based on the field name and field type of each node in the complete field path tree of any data field; Based on the field semantic vector representation of each node in the complete field path tree of any data field and the order between each node, as well as the field semantic vector representation of each field contained in each standard nested data structure and the order between each field, matching is performed to determine that the standard nested data structure with the highest matching degree is the intermediate data structure corresponding to any data field.

3. The report data synchronization method based on DataX according to claim 1, characterized in that: The reader plug-in is used to perform data preprocessing on the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure to form preprocessed data objects, including: Determining, based on the field type of the data field of the non-nested structure, an intermediate data type corresponding to the data field of the non-nested structure; Convert the corresponding data field based on the intermediate data type corresponding to the data field of the non-nested structure to obtain the intermediate type data object corresponding to the data field of the non-nested structure; Based on the type of the target data system, type conversion is performed on the intermediate structured data objects corresponding to the data fields of the nested structure and the intermediate type data objects corresponding to the data fields of the non-nested structure to obtain preprocessed data objects corresponding to each data field.

4. The datax-based reporting data synchronization method according to claim 3, characterized in that: The intermediate structured data objects corresponding to the data fields of the nested structure and the intermediate type data objects corresponding to the data fields of the non-nested structure are converted into types to obtain pre-processed data objects corresponding to the respective data fields, and then the method further includes: Performing anomaly detection on the pre-processed data object corresponding to each data field to obtain anomaly detection results for each data field; wherein the anomaly detection result for any data field includes whether any data field has an anomaly, and the type of anomaly when any data field has an anomaly; Based on the exception handling strategy for each exception type in the preset strategy library, the pre-processed data object corresponding to the data field with the exception is repaired, so as to pass the pre-processed data object corresponding to the data field with the exception after the repair to the Writer plug-in of datax.

5. The report data synchronization method based on DataX according to claim 1, characterized in that: The step of cyclically acquiring a paging data response of the paging size from the source end until a preset termination condition is reached includes: updating the paging size based on the field type of the data field in the paging data response returned by the current request, the data volume of the paging data response, and the request response time, so as to send the next request based on the updated paging size; Based on the identification field in the paging data response returned by the current request, it is dynamically determined whether a preset termination condition is reached.

6. The datax-based reporting data synchronization method according to claim 1, characterized in that: The step of caching the pre-processed data objects in batches and submitting the pre-processed data objects to the target data system includes: Cache the pre-processed data objects corresponding to each data field in a memory buffer queue; Based on a set batch quantity or time window, the pre-processed data objects in the memory buffer queue are written in batches to the target data system.

7. A datax-based reporting data synchronization device, characterized in that: include: An initialization unit, used to initialize paging control parameters in the Reader plug-in of datax based on the data structure of the data to be obtained; The paging control parameters include an initial page number and a paging size, and construct a URL address of a data acquisition request for the data to be acquired; the data to be acquired includes unstructured report data; A data acquisition and parsing unit is used to cyclically send the data acquisition request through HTTP request based on a Reader plug-in, cyclically obtain the paging data response of the paging size from the source end until a preset termination condition is reached, and parse the data fields of the nested structure in the paging data response to generate intermediate structured data objects corresponding to each data field; the source end includes multiple different types of data sources; wherein, a complete field path tree of the data fields of the nested structure is constructed; for any data field of the nested structure, based on the field name and field type of each node in the complete field path tree of the any data field and the order between each node, the intermediate data structure corresponding to the any data field is determined; based on the intermediate data structure corresponding to the any data field, an intermediate structured data object corresponding to the any data field is generated; the intermediate data structure is a standard nested data structure that matches the any data field, and the intermediate structured data object retains the hierarchy and type information of the any data field; A data preprocessing unit is used to perform data preprocessing on the intermediate structured data objects corresponding to the data fields of the nested structure and the data fields of the non-nested structure based on the Reader plug-in to form preprocessed data objects; The data batch writing unit is used to transfer the pre-processed data objects to the Writer plug-in of datax based on the Reader plug-in, and cache the pre-processed data objects according to the batch strategy based on the Writer plug-in and submit them to the target data system.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the datax-based reporting data synchronization method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the datax-based reporting data synchronization method according to any one of claims 1 to 6 is implemented.

Images