Data integration transmission method and system

Abstract

The embodiment of the invention discloses a data integration transmission method and system, and the method comprises the steps: responding to an integration request of a plurality of data sources, and determining the flow configuration corresponding to the integration request through a data integration field specific language; identifying the flow configuration through a job execution engine, generating a plurality of independent tasks based on the flow configuration, and executing each independent task through the job execution engine; and after the task is executed, outputting the integrated data. The problem that efficient data integrated transmission cannot be carried out on various data sources in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data integration transmission method and system. Background Art

[0002] With the vigorous development of future industries, it is crucial to build a data resource infrastructure platform that can achieve the integration of industrial chain, innovation chain, talent chain, capital chain, and policy chain (hereinafter referred to as the "five chains"). However, in the process of realizing the "five-chain integration", the existing data resource platform has many defects in data integration, such as: the data platforms of each chain are developed in a decentralized manner and usually do not use a unified transmission protocol. The integration of data from each chain by end users often requires manual configuration of the transmission protocol, and changes in the protocol on the data source end will lead to repeated end-to-end configurations. The data formats of different port sources are not unified, and end-to-end access requires manual access and conversion. The minimum observation unit and time granularity of each chain data are not the same, and there are often differences in the data frame structure. Different data must be aligned for fusion. The current data merging method mainly relies on manual intervention by experienced data warehouse engineers or data analysts, which is not only time-consuming and labor-intensive, but also amplifies the risk of misjudgment.

[0003] Therefore, there is an urgent need for a method for efficient data integration and transmission for multiple data sources. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data integration transmission method and system to solve the problem in the prior art that efficient data integration transmission cannot be performed for multiple data sources.

[0005] To achieve the above object, an embodiment of the present invention provides a data integration transmission method, which specifically includes:

[0006] In response to integration requests from multiple data sources, determining a process configuration corresponding to the integration request by using a data integration domain specific language;

[0007] Identifying the process configuration through a job execution engine, generating a plurality of separate tasks based on the process configuration, and executing each separate task through the job execution engine;

[0008] After the task is executed, the integrated data is output.

[0009] On the basis of the above technical solution, the present invention can also be improved as follows:

[0010] Furthermore, the process configuration includes data access and / or data extraction and / or data cleaning and / or data conversion and / or data integration and / or data modeling.

[0011] Furthermore, in response to the integration requests of the multiple data sources, determining the process configuration corresponding to the integration request by using the data integration domain specific language includes:

[0012] The process dependency and process execution order corresponding to the process configuration are determined through a data integration domain specific language.

[0013] Furthermore, executing each individual task by the job execution engine includes:

[0014] The job execution engine uses a scheduling mechanism to schedule the execution of each task based on process dependencies and process execution order.

[0015] Furthermore, the job execution engine uses a scheduling mechanism to schedule the execution of each task based on the process dependency and the process execution order, including:

[0016] Determine the access mode of the data source based on the data source type, and select the connection protocol based on the access mode to access different data sources through the data access adapter language component;

[0017] Extract the required data from different connected data sources through data extraction language components;

[0018] The required data is processed through data cleaning and conversion language components to obtain processed data;

[0019] Integrate the processed data from different data sources through data integration language components to obtain integrated data;

[0020] A data model is created through a data modeling language component, the data model is stored in a specified data warehouse, and integrated data is exported through a specified output path.

[0021] Furthermore, the method of extracting the required data from different connected data sources through the data extraction language component includes:

[0022] For data sources with large data volumes, the data access task is divided into multiple subtasks based on the data volume and executed in parallel.

[0023] Furthermore, the required data is processed by the data cleaning and conversion language component to obtain processed data, including:

[0024] For required data that needs to be updated in real time, the job execution engine uses a stream processing engine to process the required data in real time;

[0025] For the required data processing of large amounts of data, the job execution engine processes the required data in batch processing;

[0026] The cleaned required data is converted into the target format and stored in the specified storage system.

[0027] A data integration transmission system, comprising:

[0028] A process configuration module, used to respond to integration requests from multiple data sources and determine the process configuration corresponding to the integration request through a data integration domain specific language;

[0029] A job execution module, used for identifying the process configuration through a job execution engine, generating a plurality of separate tasks based on the process configuration, and executing each separate task through the job execution engine;

[0030] The integrated data output module is used to output integrated data after the task is completed.

[0031] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the computer program.

[0032] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0033] The embodiments of the present invention have the following advantages:

[0034] The data integration transmission method of the present invention can easily cope with various data source integration requirements by using a data integration domain specific language to determine the process configuration corresponding to the integration request. Whether it is structured data, unstructured data, or data sources of different formats and types, customized process configuration can be performed according to their uniqueness, greatly improving the adaptability of the method to various data source environments.

[0035] The job execution engine splits the process configuration into several separate tasks and executes them. This method enables the parallel processing of tasks. Compared with the traditional monolithic processing method, it can make better use of computing resources and improve execution efficiency. The independent execution of each task also helps to reduce mutual interference between tasks, reduce the probability of errors, and ensure the stability and efficiency of the entire integration process.

[0036] For users, they only need to initiate an integration request, and the system will automatically complete data integration and output integrated data according to the set process. This greatly simplifies the operation process of data integration and reduces the workload of manual intervention. Professionals no longer need to spend a lot of time writing complex codes or performing tedious manual configuration, which reduces labor costs and time costs. It is especially suitable for large-scale data integration projects or scenarios where data integration is frequently performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0038] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0039] Figure 1 It is a flow chart of the data integration transmission method of the present invention;

[0040] Figure 2 It is a structural diagram of the data integration transmission system of the present invention;

[0041] Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention.

[0042] The accompanying drawings are denoted as follows:

[0043] Process configuration module 10 , job execution module 20 , integrated data output module 30 , electronic device 40 , processor 401 , memory 402 , bus 403 . DETAILED DESCRIPTION

[0044] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Figure 1 FIG. 1 is a flow chart of an embodiment of the data integration transmission method of the present invention, as shown in FIG. Figure 1 As shown, a data integration transmission method provided by an embodiment of the present invention includes the following steps:

[0046] S101, in response to integration requests from multiple data sources, determining a process configuration corresponding to the integration request by using a data integration domain specific language;

[0047] Specifically: the process configuration includes data access and / or data extraction and / or data cleaning and / or data conversion and / or data integration and / or data modeling.

[0048] The process dependency and process execution order corresponding to the process configuration are determined through a data integration domain specific language.

[0049] The present invention defines the entire data integration process through a data integration domain specific language (DSL). DSL provides a concise, easy-to-understand and efficient language component, covering all aspects from data access, cleaning, conversion to integration and modeling, solving the cumbersome and error-prone manual configuration and data processing problems in the traditional data integration process.

[0050] S102, identifying the process configuration through the job execution engine, generating a plurality of separate tasks based on the process configuration, and executing each separate task through the job execution engine;

[0051] Specifically, the job execution engine uses a scheduling mechanism to schedule the execution of each task based on process dependencies and process execution order;

[0052] The job execution engine uses a scheduling mechanism to schedule the execution of each task based on process dependencies and process execution order, including:

[0053] Determine the access mode of the data source based on the data source type, and select the connection protocol based on the access mode to access different data sources through the data access adapter language component;

[0054] The data access adapter language component is used to unify and simplify data access operations from different data sources (such as files, databases, APIs, etc.). Through the data access adapter language component, users can easily configure the type, connection method and authentication information of the data source, and realize automatic access to the data source.

[0055] Command name and parameter rules:

[0056] connect_to: specifies the data source type and its connection parameters (such as file path, database connection information, API, URL, etc.).

[0057] Example command:

[0058] Ruby

[0059] connect_to:csv,file_path:"data.csv"

[0060] connect_to:api,url:"https: / / api.example.com",headers:

[0061] {"Authorization"=>"Bearer token"}

[0062] connect_to:database,db_type:"mysql",host:"localhost",username:"root",password:"password",database:"test_db"

[0063] Data access is the primary task of the job execution engine. The engine starts the adapter according to the connect_to command defined in the DSL and connects to different data sources through a unified data protocol. The specific access process is as follows:

[0064] The engine initializes the corresponding connector according to the data source type in the DSL command (such as CSV file, API, database, etc.). Each data source has a different access method, so the adapter will select the appropriate connection protocol based on the configuration.

[0065] The data access adapter unifies and simplifies data access operations from different data sources (such as files, databases, APIs, etc.). Through standardized interfaces, users can easily configure data sources, avoiding cumbersome configuration and format inconsistencies when accessing data from different sources.

[0066] Extract the required data from different connected data sources through data extraction language components;

[0067] The data extraction language component is used to extract the required data from various connected data sources. Users can specify the extracted fields, filtering conditions, and data conversion rules as needed.

[0068] Command name and parameter rules:

[0069] extract: Extract data from the specified data source.

[0070] fields: Specifies the fields to extract.

[0071] where: Optional filter condition.

[0072] Example command:

[0073] Ruby

[0074] extract from:csv,fields:[:name,:age],where:{age:">30"}extract from:api,fields:[:id,:status],where:{status:"active"}

[0075] After completing the connection initialization, the job execution engine starts to extract data from the specified data source. The data can be static (such as files) or dynamic (such as API responses, database query results).

[0076] For large-scale data sources, the job execution engine can divide the data access task into multiple subtasks according to the data volume and execute them in parallel to increase the data loading speed. At this time, the engine may also use message queues (such as Kafka and RabbitMQ) to implement parallel processing of tasks.

[0077] The present invention supports the combination of real-time data stream processing and batch data processing. This feature enables the system to efficiently process data of different time granularities and adapt to different data processing requirements. Real-time data processing is performed through a streaming engine, and batch data is processed through conventional data warehouse tools to ensure processing speed and accuracy.

[0078] The required data is processed through data cleaning and conversion language components to obtain processed data;

[0079] Data cleaning and transformation language components are used to clean and format the extracted data. Common operations include removing null values, data type conversion, field standardization, etc.

[0080] Command name and parameter rules:

[0081] transform: Transform the extracted data.

[0082] fields: Specifies the fields to be converted.

[0083] operation: specifies the conversion operation type, such as uppercase (convert to uppercase), trim (remove spaces), type_cast (data type conversion).

[0084] Example command:

[0085] Ruby

[0086] transform data,fields:[:age],type::integer

[0087] transform data,fields:[:name],operation::uppercase

[0088] transform data,fields:[:address],operation::trim

[0089] Data cleaning and conversion is an indispensable step in data integration. The job execution engine will perform data processing tasks according to the transform command defined in the DSL. The whole process includes data deduplication, format conversion, missing value processing, etc. The specific execution process is as follows:

[0090] The engine parses the cleaning rules according to the DSL definition, such as field conversion and missing value filling.

[0091] Real-time stream processing: For data that needs to be updated in real time, the job execution engine will use a stream processing engine (such as Apache Flink, Apache Kafka Streams) to process the data in real time. The data stream will be cleaned and converted according to the prescribed rules and output in real time.

[0092] Batch processing: For large amounts of data, the job execution engine will use batch processing to process data and perform cleaning tasks through data warehouses (such as Hive and Presto).

[0093] The cleaned data will be converted into the target format (such as JSON, Parquet) according to the rules defined by the DSL and stored in the specified storage system (such as HDFS, database).

[0094] The present invention provides flexible commands and parameters, and supports various common data processing operations, such as deduplication, type conversion, format standardization, etc. Users can complete complex data cleaning and conversion tasks through simple DSL syntax to ensure that the integrated data meets the predetermined quality standards.

[0095] Integrate the processed data from different data sources through data integration language components to obtain integrated data;

[0096] The data integration language component is used to merge data from different data sources. Through this component, users can merge different data sets together according to specified fields and merge rules (such as inner join, left join).

[0097] Command name and parameter rules:

[0098] integrate: Merge data from multiple data sources.

[0099] on: Specifies the field to be used when merging.

[0100] type: specifies the merge method (such as inner_join, left_join, right_join, etc.).

[0101] Example command:

[0102] Ruby

[0103] integrate from:csv,:api,on::id,type::inner_join

[0104] integrate from:database,:csv,on::user_id,type::left_join

[0105] Data integration is the process of merging data from different data sources. The job execution engine will integrate data from different ports according to the integrate command defined in the DSL. The integration process includes connecting different data sources, matching by specified fields, and removing duplicate data.

[0106] The job execution engine performs data merging operations based on the type of data source (such as database tables, data returned by APIs, CSV files, etc.). Common operations include inner join, left join, full outer join, etc.

[0107] Based on the type parameter in the DSL, the job execution engine determines how to merge. For example, when merging two datasets using an inner join, only records that exist in both datasets are retained.

[0108] The integrated data will be written to a designated storage system (such as a data warehouse or real-time database) for subsequent modeling and analysis.

[0109] Each processing step of the present invention (such as data extraction, cleaning, conversion, integration, etc.) has an independent component, and the user can flexibly define the parameters and rules of each step, and the system performs the corresponding tasks according to these rules. The modular design supports rapid expansion and flexible customization, and is suitable for various complex business scenarios.

[0110] A data model is created through a data modeling language component, the data model is stored in a specified data warehouse, and integrated data is exported through a specified output path.

[0111] The data modeling language component is used to define the data model and perform data warehouse storage or data export operations. Users can use this component to define the modeling storage type and model structure.

[0112] Command name and parameter rules:

[0113] model: defines the data model.

[0114] action (enumeration, required): specifies the type of action to perform. Common values ​​include:

[0115] create: Create a new model.

[0116] alter: Modify an existing model.

[0117] drop: Delete the model.

[0118] model_name (string, required): The name of the data model.

[0119] storage_type (enumeration, optional): specifies the storage type, which is only used in create operations. Common values ​​include relational, columnar, graph, etc.

[0120] model_structure (JSON structure, optional): model structure definition, only used in create or alter operations. Including tables, fields, data types, indexes, etc.

[0121] description (string, optional): description of the data model, only used in create operations.

[0122] operation (enumeration, optional): only used in alter operations, specifies the specific operation of modifying the model. Common values ​​include add (add columns), drop (delete columns), and modify (modify columns).

[0123] column_name (string, optional): Only used in alter operations, specifies the name of the column to be operated.

[0124] new_column_definition (JSON structure, optional): used only in alter operations, specifies the new column definition or modified column information.

[0125] Example command:

[0126] Ruby

[0127] model action:'create',model_name:'sales_data',storage_type:'relational',model_structure:'{"tables":[{"name":"sales_fact","columns":[{"name":"sales_amount","type":"DECIMAL"},{"name":"quantity","type":"INTEGER"}]}]}',description:'Sales data model for trackingtransactions andsales performance'modelaction:'alter',model_name:'sales_data',operation:'add',column_name:'discount',new_column_definition:'{"name":"discount","type":"DECIMAL"}'model action:'drop',model_name:'sales_data'

[0128] The job execution engine creates a data model based on the definition in the model command and stores it in the specified database or data warehouse.

[0129] The engine will optimize the model structure to ensure that it supports fast data query and efficient data storage. For example, the database will create appropriate indexes based on the model structure to improve query efficiency.

[0130] After the model is created, the data will be exported through the specified output path. Users can use the data for report generation, business analysis, or machine learning model training, etc.

[0131] S103, after the task is executed, the integrated data is output.

[0132] The present invention designs a data integration domain specific language (DSL) so that users can easily perform data integration operations without having a technical background. The DSL syntax is concise and intuitive, and users only need to configure it through simple commands, avoiding the complex coding work and high technical requirements in the traditional data integration process. Even non-technical personnel can easily complete tasks such as data access, cleaning, conversion, and integration, greatly reducing the technical threshold of data integration.

[0133] The DSL of the present invention helps users complete the entire data integration process in a concise configuration manner. Users only need to focus on business needs and implement operations such as data source access, data cleaning, conversion and integration through a few lines of configuration commands, avoiding cumbersome configuration steps and improving work efficiency. The design of DSL focuses on the expression of business logic, helping users quickly understand and use the tool.

[0134] The present invention supports automatic access and real-time processing of data sources, greatly reducing the need for manual intervention. The system acquires and processes data from different data sources in real time through automated processes, ensuring the efficiency and timeliness of data integration. Whether it is batch data processing or real-time stream processing, users can quickly implement it through unified DSL commands without worrying about the underlying technical details.

[0135] The DSL of the present invention not only meets the basic data integration requirements, but also supports flexible expansion and customization to adapt to different business scenarios and requirements. Users can adjust DSL commands according to actual conditions and expand new data source adapters or processing flows. The modular nature of DSL design enables the system to be continuously optimized and expanded in the future, maintaining good adaptability and maintainability.

[0136] The present invention uses DSL for configuration and management, which not only improves the readability and maintainability of the code, but also reduces errors and maintenance costs caused by technical difficulties. Users can easily modify and adjust the configuration to ensure the efficient operation of the data integration process, reducing the difficulty and cost of long-term maintenance.

[0137] Figure 2 FIG. 1 is a diagram showing an embodiment of the data integration transmission system of the present invention; Figure 2 As shown, a data integration transmission system provided by an embodiment of the present invention includes the following steps:

[0138] The process configuration module 10 is used to respond to integration requests from multiple data sources and determine the process configuration corresponding to the integration request through a data integration domain specific language; the process configuration includes data access and / or data extraction and / or data cleaning and / or data conversion and / or data integration and / or data modeling.

[0139] The process configuration module 10 is further used to determine the process dependency and process execution order corresponding to the process configuration through a data integration domain specific language.

[0140] A job execution module 20, configured to identify the process configuration through a job execution engine, generate a plurality of separate tasks based on the process configuration, and execute each separate task through the job execution engine;

[0141] The job execution module 20 is also used for:

[0142] The job execution engine uses a scheduling mechanism to schedule the execution of each task based on process dependencies and process execution order.

[0143] Determine the access mode of the data source based on the data source type, and select the connection protocol based on the access mode to access different data sources through the data access adapter language component;

[0144] The required data is extracted from different connected data sources through data extraction language components; for data sources with large data volumes, the data access task is divided into multiple subtasks based on the data volume and executed in parallel.

[0145] The required data is processed through data cleaning and conversion language components to obtain processed data;

[0146] For required data that needs to be updated in real time, the job execution engine uses a stream processing engine to process the required data in real time;

[0147] For the required data processing of large amounts of data, the job execution engine processes the required data in batch processing;

[0148] The cleaned required data is converted into the target format and stored in the specified storage system.

[0149] Integrate the processed data from different data sources through data integration language components to obtain integrated data;

[0150] A data model is created through a data modeling language component, the data model is stored in a specified data warehouse, and integrated data is exported through a specified output path.

[0151] The integrated data output module 30 is used to output the integrated data after the task is completed.

[0152] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 3 As shown, the electronic device 40 includes: a processor 401 (processor), a memory 402 (memory) and a bus 403;

[0153] The processor 401 and the memory 402 communicate with each other via the bus 403;

[0154] The processor 401 is used to call the program instructions in the memory 402 to execute the methods provided by the above-mentioned method embodiments, for example, including: responding to integration requests from multiple data sources, determining the process configuration corresponding to the integration request through a data integration domain-specific language; identifying the process configuration through a job execution engine, generating a number of separate tasks based on the process configuration, and executing each separate task through the job execution engine; after the task is executed, outputting the integrated data.

[0155] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: responding to integration requests from multiple data sources, determining a process configuration corresponding to the integration request through a data integration domain-specific language; identifying the process configuration through a job execution engine, generating a number of separate tasks based on the process configuration, and executing each separate task through the job execution engine; and outputting integrated data after the task is executed.

[0156] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0157] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0159] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A data integration transmission method, characterized in that: The data integration transmission method specifically includes: In response to integration requests from multiple data sources, determining a process configuration corresponding to the integration request by using a data integration domain specific language; Identifying the process configuration through a job execution engine, generating a plurality of separate tasks based on the process configuration, and executing each separate task through the job execution engine; After the task is executed, the integrated data is output.

2. The data integration transmission method according to claim 1, characterized in that: The process configuration includes data access and / or data extraction and / or data cleaning and / or data conversion and / or data integration and / or data modeling.

3. The data integration transmission method according to claim 1, characterized in that: The step of responding to the integration requests of multiple data sources and determining the process configuration corresponding to the integration requests by using a data integration domain specific language includes: The process dependency and process execution order corresponding to the process configuration are determined through a data integration domain specific language.

4. The data integration transmission method according to claim 3, characterized in that: The execution of each individual task by the job execution engine includes: The job execution engine uses a scheduling mechanism to schedule the execution of each task based on process dependencies and process execution order.

5. The data integration transmission method according to claim 4, characterized in that: The job execution engine uses a scheduling mechanism to schedule the execution of each task based on process dependencies and process execution order, including: Determine the access mode of the data source based on the data source type, and select the connection protocol based on the access mode to access different data sources through the data access adapter language component; Extract the required data from different connected data sources through data extraction language components; The required data is processed through data cleaning and conversion language components to obtain processed data; Integrate the processed data from different data sources through data integration language components to obtain integrated data; A data model is created through a data modeling language component, the data model is stored in a specified data warehouse, and integrated data is exported through a specified output path.

6. The data integration transmission method according to claim 5, characterized in that: The method of extracting the required data from different connected data sources through the data extraction language component includes: For data sources with large data volumes, the data access task is divided into multiple subtasks based on the data volume and executed in parallel.

7. The data integration transmission method according to claim 5, characterized in that: The required data is processed by the data cleaning and conversion language component to obtain processed data, including: For required data that needs to be updated in real time, the job execution engine uses a stream processing engine to process the required data in real time; For the required data processing of large amounts of data, the job execution engine processes the required data in batch processing; The cleaned required data is converted into the target format and stored in the specified storage system.

8. A data integration transmission system, characterized in that: include: A process configuration module, used to respond to integration requests from multiple data sources and determine the process configuration corresponding to the integration request through a data integration domain specific language; A job execution module, used for identifying the process configuration through a job execution engine, generating a plurality of separate tasks based on the process configuration, and executing each separate task through the job execution engine; The integrated data output module is used to output integrated data after the task is completed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images