End-to-end scenarized data exchange system

Abstract

The invention discloses an end-to-end scenarized data exchange system, and relates to the technical field of data exchange, in the system, when a data preprocessing subsystem detects that a source client sends a data exchange request, initial format data is converted into scene encapsulation data according to a target exchange scene corresponding to the initial format data; the development configuration subsystem generates an exchange configuration file according to a preset metadatabase and the scene encapsulation data; and the data synchronization subsystem generates target format data based on the target file format corresponding to the target client and the exchange configuration file, and synchronizes the target format data to the target client. According to the application, scene analysis is performed on the initial format data sent by the source client, and the metadata configured in the preset metadatabase is taken as a footstone, so that the corresponding target format data is generated based on the exchange configuration file and the target file format required by the target client and is synchronized to the target client; and end-to-end data exchange can be realized quickly and conveniently.

Description

Technical Field

[0001] The present application relates to the field of data exchange technology, and in particular to an end-to-end scenario-based data exchange system. Background Art

[0002] Currently, the common method of exchanging data between different systems is to map the data table content into a standard file through data mapping, and then map the standard file into a target data table to generate an ETL (Extract-Transform-Load) job.

[0003] However, the operational development for ETL process has the following disadvantages:

[0004] 1) High technical threshold. Users need to fully understand the implementation details of the underlying platform, use specific development templates for specific usage requirements, and assemble them by splicing building blocks.

[0005] 2) Poor service stability. The traditional SpringMVC architecture has problems such as bloated projects, low deployment efficiency, inability to isolate resources, high degree of coupling between functional modules, and inability to flexibly expand, resulting in poor overall service stability.

[0006] Therefore, how to effectively reduce the complexity of data exchange has become an urgent problem to be solved. Summary of the invention

[0007] The main purpose of this application is to provide an end-to-end scenario-based data exchange system, aiming to solve the technical problem of how to effectively reduce the complexity of data exchange.

[0008] To achieve the above objectives, the present application proposes an end-to-end scenario-based data exchange system, which includes: a data preprocessing subsystem, a development configuration subsystem, and a data synchronization subsystem;

[0009] The data preprocessing subsystem is used to convert the initial format data into scene encapsulation data according to the target exchange scene corresponding to the initial format data when detecting that the source client sends a data exchange request;

[0010] The development configuration subsystem is used to generate an exchange configuration file according to the preset metadata database and the scenario encapsulation data;

[0011] The data synchronization subsystem is used to generate target format data based on the target file format corresponding to the target client and the exchange configuration file, and send the target format data to the target client.

[0012] In one embodiment, the data preprocessing subsystem is further used to perform preset parameter verification on the initial format data;

[0013] The data preprocessing subsystem is further used to send the scene encapsulation data to the development configuration subsystem when the preset parameter verification passes.

[0014] In one embodiment, the data preprocessing subsystem includes a parameter verification module and an access detection module;

[0015] The parameter checking module is used to convert the initial format data into scene encapsulation data according to the target exchange scene when detecting that the source client sends a data exchange request;

[0016] The parameter verification module is further used to perform preset rule verification according to the target exchange scenario to obtain a rule verification result;

[0017] The access detection module is used to perform a preset authority check according to the data exchange request when the rule check result is passed;

[0018] The access detection module is further configured to send the scenario encapsulation data to the development configuration subsystem when the preset permission check passes.

[0019] In one embodiment, the parameter verification module includes: a scene analysis unit, a verification screening unit and a rule verification unit;

[0020] The scenario parsing unit is configured to determine the target processing scenario corresponding to the initial format data when detecting that the source client sends a data exchange request, and encapsulate the initial format data according to the target processing scenario to obtain scenario encapsulation data;

[0021] The verification screening unit is used to screen the preset verification objects according to the target processing scenario to obtain the target verification rules;

[0022] The rule verification unit is used to perform preset rule verification on the scene encapsulation data based on the target verification rule to obtain the rule verification result.

[0023] In one embodiment, the development configuration subsystem includes: a scenario configuration module, a scenario splitting module, and a configuration generation module;

[0024] The scene configuration module is used to perform preset object conversion according to the scene encapsulation data to generate a target user data stream;

[0025] The scene splitting module is used to perform a preset splitting operation on the target user data stream to generate a target operation data stream;

[0026] The configuration generation module is used to generate the exchange configuration file based on the target user data flow, the target job data flow and the preset metadata library.

[0027] In one embodiment, the scene configuration module is used to generate a target processing object according to the data type of the scene encapsulation data;

[0028] The scene configuration module is used to determine the target conversion task corresponding to the scene encapsulation data, and generate the target user data flow according to the target conversion task and the target processing object.

[0029] In one embodiment, the scene splitting module is used to filter the global splitting processor according to the target user data stream to obtain the target splitting processor;

[0030] The scene splitting module is used to generate the target operation data stream based on the target splitting processor, the target user data stream and the preset metadata library.

[0031] In one embodiment, the configuration generation module is used to determine a target switching operation according to the target user data flow;

[0032] The configuration generation module is used to determine a target configuration processor based on the target job data flow;

[0033] The configuration generation module is used to perform preset configuration association between the target exchange job and the target configuration processor according to the preset metadata database to generate the exchange configuration file.

[0034] In one embodiment, the system further comprises: a metadata processing subsystem;

[0035] The metadata processing subsystem is used to obtain target table information from a target data source through a preset data probe when receiving a configuration extension request;

[0036] The metadata processing subsystem is further used to store the target table information into the preset metadata database according to preset comparison rules.

[0037] In one embodiment, the data synchronization subsystem is further used to perform a preset grayscale conversion on the exchange configuration file to obtain a grayscale test operation;

[0038] The data synchronization subsystem is further used to perform instantiation testing based on a preset production job and the grayscale test job when a grayscale test instruction is detected, and obtain a production job result and a grayscale job result;

[0039] The data synchronization subsystem is further used to perform a preset caliber data comparison on the production operation result and the grayscale operation result to obtain a target exchange test result;

[0040] The data synchronization subsystem is further configured to generate the target format data based on the target file format corresponding to the target client and the exchange configuration file when the target exchange test result is passed, and send the target format data to the target client

[0041] The present application provides an end-to-end scenario-based data exchange system, which includes: a data preprocessing subsystem, a development configuration subsystem and a data synchronization subsystem; the data preprocessing subsystem is used to convert the initial format data into scenario encapsulation data according to the target exchange scenario corresponding to the initial format data when detecting that the source client sends a data exchange request; the development configuration subsystem is used to generate an exchange configuration file according to a preset metadata database and scenario encapsulation data; the data synchronization subsystem is used to generate target format data based on the target file format and exchange configuration file corresponding to the target client, and synchronize the target format data to the target client. The present application performs scenario analysis on the initial format data sent by the source client, encapsulates the initial format data into scenario encapsulation data based on the determined target exchange scenario, and then converts the scenario encapsulation data into an exchange configuration file based on the metadata configured in the preset metadata database, thereby generating the corresponding target format data based on the exchange configuration file and the target file format required by the target client, and sending the target format data to the target client. Therefore, the present application proposes an end-to-end scenario-based wizard-style data exchange development method, which takes metadata as the cornerstone and scenario-based as the center, so as to quickly and conveniently realize end-to-end data exchange. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 It is a first structural block diagram of the first embodiment of the end-to-end scenario-based data exchange system of the present invention;

[0045] Figure 2 It is a second structural block diagram of the first embodiment of the end-to-end scenario-based data exchange system of the present invention;

[0046] Figure 3 It is a third structural block diagram of the first embodiment of the end-to-end scenario-based data exchange system of the present invention;

[0047] Figure 4 This is a business architecture diagram of the first embodiment of the end-to-end scenario-based data exchange system of the present invention;

[0048] Figure 5 It is a structural block diagram of the second embodiment of the end-to-end scenario-based data exchange system of the present invention;

[0049] Figure 6 This is a schematic diagram of scenario interaction of the second embodiment of the end-to-end scenario-based data exchange system of the present invention.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of this application is: to propose an integrated end-to-end scenario-based data exchange system, which includes a data preprocessing subsystem, a development configuration subsystem and a data synchronization subsystem; the data preprocessing subsystem is used to convert the initial format data into scenario encapsulation data according to the target exchange scenario corresponding to the initial format data when detecting that the source client sends a data exchange request; the development configuration subsystem is used to generate an exchange configuration file according to a preset metadata library and scenario encapsulation data; the data synchronization subsystem is used to generate target format data based on the target file format and exchange configuration file corresponding to the target client, and synchronize the target format data to the target client.

[0054] The existing ETL job development tools and one-stop development platforms have problems such as ① high technical barriers and difficulty in getting started; ② low degree of automation and cumbersome development; ③ poor service stability and high coupling; ④ confusing functions and poor experience.

[0055] Accordingly, the present application performs scenario analysis on the initial format data sent by the source client, encapsulates the initial format data into scenario encapsulation data based on the determined target exchange scenario, and then converts the scenario encapsulation data into an exchange configuration file based on the metadata configured in the preset metadata database, thereby generating corresponding target format data based on the exchange configuration file and the target file format required by the target client, and sending the target format data to the target client. Therefore, the present application proposes an end-to-end scenario-based wizard-style data exchange development method, which takes metadata as the cornerstone and scenario-based as the center, so as to quickly and conveniently realize end-to-end data exchange.

[0056] The embodiment of the present invention provides an end-to-end scenario-based data exchange system, referring to Figure 1 , Figure 1 FIG. 1 is a first structural block diagram of the first embodiment of the end-to-end scenario-based data exchange system of the present invention. Figure 1 As shown, in this embodiment, the system includes: a data preprocessing subsystem, a development configuration subsystem and a data synchronization subsystem;

[0057] The data preprocessing subsystem is used to convert the initial format data into scene encapsulation data according to the target exchange scene corresponding to the initial format data when detecting that the source client sends a data exchange request;

[0058] The development configuration subsystem is used to generate an exchange configuration file according to the preset metadata database and the scenario encapsulation data;

[0059] The data synchronization subsystem is used to generate target format data based on the target file format corresponding to the target client and the exchange configuration file, and send the target format data to the target client.

[0060] It should be understood that in this embodiment, the data exchange system can adopt a business architecture with front-end and back-end separation, and provide two channels of WEB application access and RESTful API access to connect with the source client. When it is detected that the source client sends a data exchange request, the initial format data transmitted by the source client can be cleaned and converted by the data preprocessing subsystem located at the front end and then flow into the back end, which is mainly composed of a development configuration subsystem and a data synchronization subsystem.

[0061] In a feasible implementation manner, in this embodiment, the data preprocessing subsystem is further used to perform preset parameter verification on the initial format data;

[0062] The data preprocessing subsystem is further used to send the scene encapsulation data to the development configuration subsystem when the preset parameter verification passes.

[0063] It is understandable that the data preprocessing subsystem located at the front end is mainly responsible for the cleaning and conversion of access data. In order to adapt to a variety of data exchange scenarios, this embodiment designs a set of universal registration interfaces to receive initial format data, which is compatible with various end-to-end data synchronization scenarios such as table to table, table to file, and file to table. Therefore, in order to be compatible with various end-to-end data synchronization scenarios such as table to table, table to file, and file to table, this embodiment pre-designs a set of general processes and general verification rules to perform the above-mentioned preset parameter verification, and realizes the inspection and verification of API interface and WEB application access data. Specifically, the data preprocessing subsystem can be used to perform scenario-based parameter verification and data encapsulation on the data exchange request and initial format data corresponding to the source client input by the universal registration interface according to the target exchange scenario corresponding to the initial format data. The above-mentioned target exchange scenarios can be divided into data entering the lake, data leaving the lake, synchronization within the lake, and synchronization outside the lake.

[0064] As an implementation method, refer to Figure 2 , Figure 2 It is a second structural block diagram of the second embodiment of the end-to-end scenario-based data exchange system of the present invention. In this embodiment, the data preprocessing subsystem includes a parameter verification module and an access detection module;

[0065] The parameter checking module is used to convert the initial format data into scene encapsulation data according to the target exchange scene when detecting that the source client sends a data exchange request;

[0066] The parameter verification module is further used to perform preset rule verification according to the target exchange scenario to obtain a rule verification result;

[0067] The access detection module is used to perform a preset authority check according to the data exchange request when the rule check result is passed;

[0068] The access detection module is further configured to send the scenario encapsulation data to the development configuration subsystem when the preset permission check passes.

[0069] It is understandable that in this embodiment, the data preprocessing subsystem can first perform scenario-based parameter rule verification on the data input by the source client through the universal registration interface, and then perform scenario-based access detection on the data that passes the parameter rule verification. Only when both pass, can the scenario encapsulation data generated after the initial format data is encapsulated be sent to the development configuration subsystem for subsequent specific data exchange.

[0070] As an implementable method, in this embodiment, the parameter verification module includes: a scene analysis unit, a verification screening unit and a rule verification unit;

[0071] The scenario parsing unit is configured to determine the target processing scenario corresponding to the initial format data when detecting that the source client sends a data exchange request, and encapsulate the initial format data according to the target processing scenario to obtain scenario encapsulation data;

[0072] The verification screening unit is used to screen the preset verification objects according to the target processing scenario to obtain the target verification rules;

[0073] The rule verification unit is used to perform preset rule verification on the scene encapsulation data based on the target verification rule to obtain the rule verification result.

[0074] It should be understood that when the scene parsing unit obtains the initial format data passed in by the API or WEB, it can obtain the scene data from it, parse the passed in scene data to determine the target processing scene, and then encapsulate the object according to the determined target processing scene result level to obtain the scene encapsulation data. The KEY in the scene encapsulation data is encapsulated according to the stage, range, type, and attribute, and the value of the corresponding attribute is used as VALUE.

[0075] It is understandable that this embodiment can predefine at least 23 general verification rule templates and at least 20 custom rules, namely the above-mentioned preset verification object, which can implement the verification of more than 1,600 attributes, and some general verification rule descriptions can be shown in Table 1. All verification rules will be spliced ​​according to the inspection stage, inspection scope, inspection type, inspection attribute, and inspection rule, and all have corresponding verification rule identifiers.

[0076] Table 1 General verification rules

[0077]

[0078]

[0079] Therefore, when actually performing parameter verification, the verification rules to be checked can be screened from the preset verification objects according to the target exchange scenario. That is, in this embodiment, different data exchange scenarios require different verification rules to be checked, and the rules to be checked for this request are filtered out from all verification rule objects.

[0080] It is easy to understand that after determining the target verification rules, this embodiment can execute the hit rule checks one by one according to the scene data to perform the above-mentioned preset rule verification. For example, the input attribute "Extraction Configuration-ORACLE-Database Information-Table Chinese Name" may involve multiple verification rules such as whether it can be empty (IS_EMPTY) and length verification (LENGTH_MAX_VALUE). After the preset rule verification is completed, the verification result can be returned, that is, "Verification Successful" or "Verification Failed". If the verification fails, the attributes of the failed verification and the reason for the failure will be returned together, and the subsequent process will no longer be carried out.

[0081] When the preset rule check is passed, the present embodiment may perform preset permission check based on the access detection module. Accordingly, in the present embodiment, the access detection module may include: a request judgment module, a permission check module and a configuration trigger module;

[0082] The request judgment module is used to detect whether the data exchange request is a repeated request when the rule verification result is passed;

[0083] The authority checking module is used to perform a user authority check according to the data exchange request;

[0084] The configuration trigger module is used to send the scenario encapsulation data to the development configuration subsystem when the data exchange request is not a duplicate request and the authority check passes.

[0085] Specifically, in this embodiment, the request judgment module can judge whether the data exchange request is a repeated request. Only when it is judged to be a non-duplicate request will the corresponding data exchange request be saved to the access table and the registration reception record table, and the request status is registered as ACCEPT. ACCEPT means successful reception, and REFUSE means rejection, and AGREE means acceptance.

[0086] The authority check module can perform a user authority check after the data exchange request is successfully received. If the user does not have authority, the request status of the registration reception record table is registered as REFUSE, and this request ends; if the authority check passes, the request status of the registration reception record table is registered as AGREE, and this request is saved in the registration acceptance table, and the status is registered as O. O means processing, Y means successful processing, and N means failed processing.

[0087] Therefore, the configuration trigger module will trigger the subsequent process of development configuration only when it determines that the data exchange request is not a duplicate request and the permission check passes. By asynchronously calling the DCS processing process, the packaged scene package data is passed as an input parameter to the development configuration subsystem. At this point, the data cleaning and conversion processing of the general registration interface corresponding to the data preprocessing subsystem at the front end is completed.

[0088] It is important to understand that the existing data exchange methods have a low degree of automation. Lacking metadata support, users need to complete a complex data research process on their own, including data table structure, field mapping research, etc., and lack the ability to develop automation. In this implementation, after receiving the scene encapsulation data, the development configuration subsystem can obtain the target configuration metadata required to correspond to the target exchange scene from the preset metadata database, and then generate an exchange configuration file based on the scene encapsulation data and the target configuration metadata.

[0089] It should be noted that, in the present embodiment, the data synchronization subsystem can generate a low-code data conversion module based on the target file format and exchange configuration file corresponding to the target client. The data conversion module can be deployed to the server of the target client to automatically convert the initial format data into a target format based on the data conversion module in the target client to generate target format data, thereby synchronizing the target format data to the target client according to the current target processing scenario, and then quickly realizing end-to-end scenario-based data exchange between the source client and the target client.

[0090] Therefore, as an implementable method, refer to Figure 3 , Figure 3 It is a third structural block diagram of the first embodiment of the end-to-end scenario-based data exchange system of the present invention. In this embodiment, the system further includes: a metadata processing subsystem;

[0091] The metadata processing subsystem is used to obtain target table information from a target data source through a preset data probe when receiving a configuration extension request;

[0092] The metadata processing subsystem is further used to store the target table information into the preset metadata database according to preset comparison rules.

[0093] It can be understood that in this embodiment, the metadata processing subsystem is mainly used to collect metadata information such as open platform data source information, core MPP data warehouse table information, Hadoop offline analysis platform table information corresponding to the source client and the target client, that is, configuration metadata, and store it in the preset metadata database for use by the development configuration subsystem.

[0094] Specifically, the metadata processing subsystem can provide metadata management and metadata collection functions. That is, the metadata processing subsystem includes a metadata collection module and a metadata management module;

[0095] The metadata collection module is used to, upon receiving a configuration extension request, establish a preset data connection with the target data source through a preset data probe, and obtain configuration metadata from the target data source when the connection is successful;

[0096] The metadata collection module is further used to perform preset data cleaning on the configuration metadata to obtain the target table information;

[0097] The metadata management module is used to perform stream grouping on the target table information to obtain target table elements;

[0098] A metadata management module, used to traverse the information of the preset table library to obtain preset table elements;

[0099] The metadata management module is used to match the target table elements with the preset table elements, and store the target table information in the preset metadata database according to the information matching result.

[0100] Among them, the probe program used in the metadata collection process, that is, the above-mentioned preset data probe, is compatible with the table structure collection of databases such as Oracle, MySQL, Gauss200, GaussDB, TDSQL, TiDB, DB2, SQLServer, GBase, Clickhouse, StarRocks, Elasticsearch, MongoDB, Redis, Kafka, FusionInsight HIVE, CDH HIVE, HBase, etc. The preset data probe attempts to directly connect to the server and database corresponding to the source client and the target client according to the database configuration, that is, the above-mentioned target data source. If the connection is successful, the table information in the database table of the source client and the target client, that is, the above-mentioned target table information, is collected, and a unified cleaning conversion is performed and recorded in the background database. If the connection fails, try again, record the error log and exit after multiple failures.

[0101] After the target table information is generated, metadata management is used to implement version management, that is, the table structure information in the preset metadata database, that is, the preset table information and the latest table structure information corresponding to the pulled target table information are compared. If they are different, a new version is added. Only the latest two version data are saved. If the pulled target table information contains a table structure that has not been saved before, the target table information is added to the metadata table and its table structure table stored in the preset metadata database. If the table version is found to be updated in the pulled target table information, the metadata table and its table structure table in the preset metadata database also need to be updated. The table in the database uses the server code, mode name, database name, and data table name to determine its uniqueness, and the file uses the server code, file name, file format, absolute file path, and mount point path to determine its uniqueness.

[0102] When comparing the target table information with the table information in the preset metadata database based on the preset comparison rules, the input parameters of the data comparison interface are the collected table structure and the table structure set queried from the database. In this embodiment, the metadata information queried from the table can be first grouped into streams, and a map is obtained based on the server code, database name, mode name, and data table name as the key value of the group. Then, the collected table data information is circulated, and the key in the map is matched based on the server code, mode name, database name, and data table field in the table information. After the match is successful, the table structure is compared.

[0103] In addition, this embodiment can also be graded according to the difference in comparison. Different levels of difference may cause different impacts. The difference grading effect is shown in Table 2.

[0104] Table 2 Difference classification

[0105] level Change Notes Level meaning Notification Mechanism Level 1 Modify field sequence number Will not cause an error Inform the user, no action required Level 1 Modify field description Will not cause an error Inform the user, no action required Level 2 Modify field format May cause errors Inform users, evaluate and handle Level 2 Modify field precision May cause errors Inform users, evaluate and handle Level 2 Modify field length May cause errors Inform users, evaluate and handle Level 2 PI May cause errors Inform users, evaluate and handle Level 2 PPI May cause errors Inform users, evaluate and handle Level 3 Modify field type Will directly cause data conversion errors Inform the user and handle it immediately Level 3 Reduce Fields Will directly cause data conversion errors Inform the user and handle it immediately

[0106] As an implementable method, in this embodiment, the data synchronization subsystem is further used to perform a preset grayscale conversion on the exchange configuration file to obtain a grayscale test operation;

[0107] The data synchronization subsystem is further used to perform instantiation testing based on a preset production job and the grayscale test job when a grayscale test instruction is detected, and obtain a production job result and a grayscale job result;

[0108] The data synchronization subsystem is further used to perform a preset caliber data comparison on the production operation result and the grayscale operation result to obtain a target exchange test result;

[0109] The data synchronization subsystem is further configured to generate the target format data based on the target file format corresponding to the target client and the exchange configuration file when the target exchange test result is passed, and send the target format data to the target client.

[0110] It should be noted that, from the above analysis, the data synchronization subsystem can be responsible for deploying the code and configuration generated by the development configuration subsystem to the target client's server for data synchronization, and in order to improve the reliability of data exchange, the data synchronization subsystem in this embodiment can also deploy the exchange configuration file to the development, test, production or other environments, providing a one-click automated testing capability based on real business data in production to ensure the reliability of the exchange configuration file. The grayscale test process can be a user-personalized test function, and the user can decide whether to input a grayscale test instruction to determine whether to start the automatic grayscale test of the exchange configuration file.

[0111] Specifically, in this embodiment, the entire grayscale testing process can be divided into three stages: deployment, operation, and comparison.

[0112] In the deployment phase, the deployment package generated by the development configuration subsystem, that is, the exchange configuration file, can be converted into a deployment package adapted to the grayscale environment by preset grayscale conversion, and the grayscale deployment package can be deployed to the grayscale environment of the exchange platform user and the grayscale domain of the quasi-production environment of the scheduling platform to obtain the grayscale test job. After the deployment is completed, the target exchange scenario is identified to determine whether this operation is a change. If it is a change scenario, the production environment jobs are synchronized to the grayscale environment in batches and used as production control jobs. The production control job is the preset production job mentioned above.

[0113] During the operation phase, you can automatically instantiate the grayscale test job, instantiate the production control job on demand, and automatically set the end-of-day pass to trigger the automatic execution of the job. Monitor the overall operation of the grayscale test job and the production control job. When the jobs are completed, obtain the generated job results and grayscale job results, and trigger the comparison logic.

[0114] In the comparison phase, the grayscale test operation and production control operation operation result tables, that is, the above production operation results and grayscale operation results, are compared. The scope includes:

[0115] (1) Records: Compare the consistency of the records in the grayscale test table and the production comparison table;

[0116] (2) Fields: the number of fields in the grayscale test table and the production comparison table, the attributes of the same field names (type, length, precision, etc.), and the distribution key comparison;

[0117] (3) Content, the process of comparing the preset caliber can be:

[0118] ① Based on the comparison of field contents with completely consistent field names and field types, the grayscale test table and the production comparison table are compared for minus, providing the ratio of data overlap;

[0119] ②Based on the comparison of field contents with the same field name, the grayscale test table and the production comparison table are compared for minus, providing the ratio of data overlap;

[0120] ③For newly added fields, count the distribution of empty strings and values ​​whose field contents are not empty.

[0121] Finally, the comparison conclusion, i.e., the target exchange test result, is returned. It is easy to understand that when the user chooses to perform a grayscale test, it means that only when the target exchange test result passes, the target format data can be generated based on the target file format and exchange configuration file corresponding to the target client. If it fails, the cause of the failure needs to be pushed to the user.

[0122] Therefore, this embodiment can realize efficient and convenient automated grayscale testing, and one-click automated testing based on real production business data, so as to automatically adapt and deploy the grayscale jobs to be put into production, synchronize and deploy control jobs from the production environment, and automatically execute them, and analyze and compare the running results of the grayscale jobs with those of the control jobs.

[0123] In this implementation, the specific architecture diagram of the data exchange system can be as follows: Figure 4 As shown, Figure 4 FIG. 1 is a business architecture diagram of the first embodiment of the end-to-end scenario-based data exchange system of the present invention. Figure 4 As shown, when the source client sends a data exchange request (i.e. Figure 4 When the source client sends the initial format data to the data exchange system through the API interface or WEB application access, the data pre-processing subsystem at the front end of the data exchange system can clean and convert the initial format data before flowing into the back end. The back end of the data exchange system is mainly composed of the metadata processing subsystem (i.e. Figure 4 metadata collection subsystem in the development configuration subsystem, data synchronization subsystem (i.e. Figure 4 The data exchange system proposed in this implementation can handle data access detection in a variety of data exchange scenarios, support more than 100 data synchronization scenarios such as table to table, table to file, table to queue, file to table, file to file, file to queue, etc., and can realize a variety of data exchange demands through API interface or WEB application, and can be flexibly expanded and customized according to actual business development.

[0124] Compared with the prior art, this embodiment can provide a rich variety of data exchange scenarios, can realize simple and rich data exchange demands through API interface or WEB application interface, and can provide one-click automated grayscale testing based on real business data in production to improve data exchange reliability.

[0125] The present embodiment discloses an end-to-end scenario-based data exchange system, which includes: a data preprocessing subsystem, a development configuration module and a data synchronization subsystem; the data preprocessing subsystem is used to convert the initial format data into scenario encapsulation data according to the target exchange scenario corresponding to the initial format data when detecting that the source client sends a data exchange request; the development configuration module is used to generate an exchange configuration file according to the preset metadata library and the scenario encapsulation data; the data synchronization subsystem is used to generate the target format data based on the target file format and exchange configuration file corresponding to the target client, and synchronize the target format data to the target client. This embodiment performs scenario analysis on the initial format data sent by the source client, encapsulates the initial format data into scenario encapsulation data based on the determined target exchange scenario, and then converts the scenario encapsulation data into an exchange configuration file based on the metadata configured in the preset metadata library, thereby generating the corresponding target format data based on the exchange configuration file and the target file format required by the target client, and sending the target format data to the target client. Therefore, this embodiment proposes an end-to-end scenario-based wizard-style data exchange development method, which takes metadata as the cornerstone and scenario-based as the center to quickly and conveniently realize end-to-end data exchange.

[0126] Reference Figure 5 , Figure 5 This is a structural block diagram of the second embodiment of the end-to-end scenario-based data exchange system of the present invention. Figure 1 The illustrated embodiment proposes a second embodiment of the end-to-end scenario-based data exchange system of the present invention.

[0127] like Figure 5 As shown, in this embodiment, the development configuration subsystem includes: a scene configuration module, a scene splitting module and a configuration generation module;

[0128] The scene configuration module is used to perform preset object conversion according to the scene encapsulation data to generate a target user data stream;

[0129] The scene splitting module is used to perform a preset splitting operation on the target user data stream to generate a target operation data stream;

[0130] The configuration generation module is used to generate the exchange configuration file based on the target user data flow, the target job data flow and the preset metadata library.

[0131] It should be noted that in this embodiment, the development configuration subsystem is the core system of this application, which can realize functions such as scene configuration, scene splitting, and configuration generation, and convert the user's visual configuration information on the page into the underlying code and configuration of the exchange platform to achieve low-code development.

[0132] Among them, the main function of the scene configuration unit is to convert the data objects input by the data preprocessing module into data objects that can be processed by the development configuration subsystem, and set up conversion tasks to finally generate user data streams.

[0133] Therefore, in one possible implementation mode, in this embodiment, the scene configuration module is used to generate a target processing object according to the data type of the scene encapsulation data;

[0134] The scene configuration module is used to determine the target conversion task corresponding to the scene encapsulation data, and generate the target user data flow according to the target conversion task and the target processing object.

[0135] It is understandable that in this embodiment, the scene configuration module can generate corresponding source objects and target objects, i.e., the above-mentioned target processing objects, according to the data source and target type corresponding to the scene encapsulation data. Among them, the object data is stored in data tables such as table information, standard file information, and non-standard file information, and the source and target field information is stored in the field information table.

[0136] Then, the scene configuration module generates the corresponding conversion tasks. Specifically, the conversion tasks that can be supported by this embodiment include file landing loading, file non-landing loading (direct extraction), cross-regional file transmission (cross-Shenzhen / Shanghai cluster transmission), file merging, file sorting, file splitting, file transcoding and other conversion functions. Therefore, the scene configuration module can determine the current corresponding target conversion task based on the source and target objects, and the scene configuration selected by the user input parameters. So that the subsequent target conversion tasks can flexibly generate direct extraction operations (high efficiency) and extraction loading (high data sharing) operations according to the downstream data sharing situation and the timeliness level of the data.

[0137] Finally, the scene configuration module can save the corresponding user data stream, that is, the scene configuration module can save the source object information, target object information, and conversion tasks as the target user data stream. For example, according to the exchange platform's specification that "a source object can only be extracted once", when saving the user data stream, for non-direct extraction scenarios, the five elements (source server code, source database serial number, source database name, source schema name, source table name) can be used to check whether there are duplicate source objects. If the object is duplicated, a reuse mark is added, indicating that no new source object data generation jobs will be added during subsequent scene splitting; according to the exchange platform's specification that "a target object can only be loaded once", when saving the user data stream, the five elements (target server code, target database serial number, target database name, target schema name, target table name) can be used to check whether there are duplicate target objects. If the object is duplicated, an error is reported and exited, and the request status of the registration acceptance table is registered as N through the data preprocessing module.

[0138] It should be understood that the scene splitting module can split and generate corresponding job data streams and job link relationship information, etc. according to the target user data stream information generated by the scene configuration module. Therefore, in one embodiment, the scene splitting module is used to screen the global splitting processor according to the target user data stream to obtain the target splitting processor;

[0139] The scene splitting module is used to generate the target operation data stream based on the target splitting processor, the target user data stream and the preset metadata library.

[0140] It is easy to understand that the scene splitting module can pre-acquire all the split processor configurations of the current configuration, that is, the above-mentioned global split processor. In this embodiment, 9 types of split processors can be supported, and the weight order of each split processor is different. The specific order can be shown in Table 3. The smaller the sequence number in Table 3, the higher the priority, indicating that the job priority generated by the splitting of this type of processor is higher.

[0141] Table 3 Global Split Processor Details

[0142]

[0143] Therefore, in this embodiment, the scene splitting module can filter the pre-acquired global splitting processor according to the target user data stream, and specifically select the corresponding target splitting processor according to the source object information, target object information, and conversion task configuration in the user data stream.

[0144] Then, the split processors are executed in sequence according to the weight order of each processor in the target split processor. Each split processor will obtain the naming specifications such as the exchange platform job name and standard file name from the preset metadata database to generate job information such as the job name and standard file name; then, according to the server type and database type associated with the source object and the target object, the processing script information corresponding to the job is generated; finally, according to the execution order of the split processors, the job link information is generated.

[0145] Finally, the scene splitting module can save the job information, job link information, and processing script information involved in the operation of the above-mentioned target splitting processor as the target job data stream, and can persist it to the background database for data backup.

[0146] It should be noted that, in this embodiment, the configuration generation module can generate various configurations related to various operations in the scene according to the operation data flow information generated by the scene splitting. Therefore, in one embodiment, the configuration generation module is used to determine the target switching operation according to the target user data flow;

[0147] The configuration generation module is used to determine a target configuration processor based on the target job data flow;

[0148] The configuration generation module is used to perform preset configuration association between the target exchange job and the target configuration processor according to the preset metadata database to generate the exchange configuration file.

[0149] It should be noted that the data exchange scenario implemented in this embodiment refers to the tasks of extracting, loading, converting, etc. data between various databases and platforms such as data sources, exchange platforms, and data targets. Specifically, it is divided into four major scenarios according to categories, namely ① data entering the lake, ② data leaving the lake, ③ synchronization within the lake, and ④ synchronization outside the lake. The specific scenario interaction diagram is shown in the figure below. Figure 6 As shown, Figure 6 This is a schematic diagram of scenario interaction of the second embodiment of the end-to-end scenario-based data exchange system of the present invention. Figure 6 The data source can be the source client, and the target database can be the database of the target client. The configuration steps corresponding to the four major data interaction scenarios are shown in Table 4. In actual operation, the user can perform corresponding configuration according to the steps shown in Table 4 based on the target exchange scenario, and the configuration information can be correspondingly saved in the user data flow information. Therefore, the configuration generation module can first query the job list corresponding to the configuration information determined by the user based on the user data flow information generated after the scene splitting, and learn which jobs need to be generated.

[0150] Table 4 Configuration steps for data exchange scenarios

[0151] Scenario step Data into the lake ①Extract configuration, ②Load configuration, ③Library configuration, ④More configuration Data out of the lake ①Extract configuration, ②Load configuration, ③Convert configuration (optional), ④More configurations Synchronization within the lake ①Extract configuration, ②Load configuration, ③Library configuration, ④More configuration Synchronization outside the lake ①Extract configuration, ②Load configuration, ③Convert configuration (optional), ④More configurations

[0152] Specifically, the configuration generation module may first obtain the configuration generation processor required by each job. That is, the configuration generation module may query the configuration generation processor required by the script corresponding to each job one by one according to the job list in the target job data stream to obtain the target configuration processor. It is easy to understand that in this embodiment, each job may be associated with one or more configuration generation processors.

[0153] It is understandable that the configuration generation module can execute the target configuration processors one by one, generate corresponding job configurations, and obtain the final exchange configuration file. The exchange configuration file can be temporarily stored locally or packaged and uploaded to the background database for data backup.

[0154] At this point, in this embodiment, the development configuration subsystem can convert the user's visual configuration information on the page into the underlying code and configuration of the exchange platform, generate a series of automatically executable KTR configurations, SQL codes, XML configurations, scheduling configurations, etc., to achieve low-code development of data exchange. After the user inputs the initial format data through the API interface or WEB application and determines the target exchange scenario, the data exchange system proposed in this embodiment can realize the low-code automatic execution of the operation chain from data source collection, processing, and loading to the target database, thereby reducing the complexity of data exchange to a limited extent.

[0155] This embodiment discloses a development configuration subsystem including: a scene configuration module, a scene splitting module and a configuration generation module; the scene configuration module is used to generate a target processing object according to the data type of the scene encapsulation data; the scene configuration module is used to determine the target conversion task corresponding to the scene encapsulation data, and generate a target user data stream according to the target conversion task and the target processing object. The scene splitting module is used to screen the global splitting processor according to the target user data stream to obtain the target splitting processor; the scene splitting module is used to generate a target job data stream based on the target splitting processor, the target user data stream and the preset metadata library. The configuration generation module is used to determine the target exchange job according to the target user data stream; the configuration generation module is used to determine the target configuration processor based on the target job data stream; the configuration generation module is used to preset the configuration association between the target exchange job and the target configuration processor according to the preset metadata library to generate an exchange configuration file.

[0156] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0157] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0158] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0159] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0160] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An end-to-end scenario-based data exchange system, characterized in that: The system includes: a data preprocessing subsystem, a development configuration subsystem and a data synchronization subsystem; The data preprocessing subsystem is used to convert the initial format data into scene encapsulation data according to the target exchange scene corresponding to the initial format data when detecting that the source client sends a data exchange request; The development configuration subsystem is used to generate an exchange configuration file according to the preset metadata database and the scenario encapsulation data; The data synchronization subsystem is used to generate target format data based on the target file format corresponding to the target client and the exchange configuration file, and send the target format data to the target client.

2. The end-to-end scenario-based data exchange system according to claim 1, characterized in that: The data preprocessing subsystem is further used to perform preset parameter verification on the initial format data; The data preprocessing subsystem is further used to send the scene encapsulation data to the development configuration subsystem when the preset parameter verification passes.

3. The end-to-end scenario-based data exchange system according to claim 2, characterized in that: The data preprocessing subsystem includes a parameter verification module and an access detection module; The parameter checking module is used to convert the initial format data into scene encapsulation data according to the target exchange scene when detecting that the source client sends a data exchange request; The parameter verification module is further used to perform preset rule verification according to the target exchange scenario to obtain a rule verification result; The access detection module is used to perform a preset authority check according to the data exchange request when the rule check result is passed; The access detection module is further configured to send the scenario encapsulation data to the development configuration subsystem when the preset permission check passes.

4. The end-to-end scenario-based data exchange system according to claim 3, characterized in that: The parameter verification module includes: a scene analysis unit, a verification screening unit and a rule verification unit; The scene parsing unit is used to determine the target processing scene corresponding to the initial format data when detecting that the source client sends a data exchange request, and encapsulate the initial format data according to the target processing scene to obtain scene encapsulation data; The verification screening unit is used to screen the preset verification objects according to the target processing scenario to obtain the target verification rules; The rule verification unit is used to perform preset rule verification on the scene encapsulation data based on the target verification rule to obtain the rule verification result.

5. The end-to-end scenario-based data exchange system according to claim 4, characterized in that: The development configuration subsystem includes: a scene configuration module, a scene splitting module and a configuration generation module; The scene configuration module is used to perform preset object conversion according to the scene encapsulation data to generate a target user data stream; The scene splitting module is used to perform a preset splitting operation on the target user data stream to generate a target operation data stream; The configuration generation module is used to generate the exchange configuration file based on the target user data flow, the target job data flow and the preset metadata library.

6. The end-to-end scenario-based data exchange system according to claim 5, characterized in that: The scene configuration module is used to generate a target processing object according to the data type of the scene encapsulation data; The scene configuration module is used to determine the target conversion task corresponding to the scene encapsulation data, and generate the target user data flow according to the target conversion task and the target processing object.

7. The end-to-end scenario-based data exchange system according to claim 5, characterized in that: The scene splitting module is used to screen the global splitting processor according to the target user data flow to obtain the target splitting processor; The scene splitting module is used to generate the target operation data stream based on the target splitting processor, the target user data stream and the preset metadata library.

8. The end-to-end scenario-based data exchange system according to claim 5, characterized in that: The configuration generation module is used to determine a target switching operation according to the target user data flow; The configuration generation module is used to determine a target configuration processor based on the target job data flow; The configuration generation module is used to perform preset configuration association between the target exchange job and the target configuration processor according to the preset metadata database to generate the exchange configuration file.

9. The end-to-end scenario-based data exchange system according to claim 1, characterized in that: The system further comprises: a metadata processing subsystem; The metadata processing subsystem is used to obtain target table information from a target data source through a preset data probe when receiving a configuration extension request; The metadata processing subsystem is further used to store the target table information into the preset metadata database according to preset comparison rules.

10. The end-to-end scenario-based data exchange system according to claim 1, characterized in that: The data synchronization subsystem is further used to perform a preset grayscale conversion on the exchange configuration file to obtain a grayscale test operation; The data synchronization subsystem is further used to perform instantiation testing based on a preset production job and the grayscale test job when a grayscale test instruction is detected, and obtain a production job result and a grayscale job result; The data synchronization subsystem is further used to perform a preset caliber data comparison on the production operation result and the grayscale operation result to obtain a target exchange test result; The data synchronization subsystem is further configured to generate the target format data based on the target file format corresponding to the target client and the exchange configuration file when the target exchange test result is passed, and send the target format data to the target client.

Images