Method and device for deriving multi-source heterogeneous data, readable storage medium and terminal

Abstract

A method, apparatus, readable storage medium, and terminal for exporting multi-source heterogeneous data are disclosed. The method includes: upon receiving a data export task, obtaining descriptive information of the data to be exported from the data export task. This descriptive information indicates the attribute information of the data to be exported and the various data sources from which it originates. The attribute information of the data to be exported is set based on preset metadata. A target execution engine is selected based on the various data sources from which the data to be exported originates and the current resource load of the execution engine. The data export task is converted into an executable task adapted to the target execution engine, and the executable task is distributed to the target execution engine. The target execution engine then performs data queries on the corresponding data sources based on the executable task. The export result is obtained based on the data query results and output. This solution enables the fusion and export of heterogeneous data from different data sources, improving the convenience of data export.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and apparatus for exporting multi-source heterogeneous data, a readable storage medium, and a terminal. Background Technology

[0002] With the continuous development of storage technology, more and more databases with different characteristics have emerged. As businesses grow and databases diversify, single storage solutions often cannot meet the needs of business development, and the coexistence of multiple storage solutions will become the norm. This will result in situations where the required data may come from different storage sources, i.e., from different data sources, and may involve interleaving, union, and difference operations on data from different data sources.

[0003] Because different data sources are independent of each other and have different data storage formats, data fusion cannot be performed during data export. Current technologies typically require data to be generated locally at each data source, and then the user must merge the data from each source according to their specific needs. This method of data export is quite cumbersome. Summary of the Invention

[0004] The technical problem solved by the embodiments of the present invention is that the existing methods for exporting multi-source heterogeneous data are rather cumbersome.

[0005] To address the aforementioned technical problems, this invention provides a method for exporting multi-source heterogeneous data, comprising: upon receiving a data export task, obtaining descriptive information of the data to be exported from the data export task, wherein the descriptive information of the data to be exported indicates the attribute information of the data to be exported and the various data sources from which it originates, and the attribute information of the data to be exported is set based on preset metadata; selecting a target execution engine based on the various data sources from which the data to be exported originates and the current resource load of the execution engine; converting the data export task into an executable task adapted to the target execution engine, and distributing the executable task to the target execution engine, wherein the target execution engine performs data queries on the corresponding data sources based on the executable task; and obtaining and outputting the export result based on the data query result returned by the target execution engine.

[0006] Optionally, selecting the target execution engine based on the various data sources from which the data to be exported comes and the current resource load of the execution engine includes: selecting the target execution engine based on the type of each data source from which the data to be exported comes, the current remaining resources of the execution engine, and the attribute information of the data to be exported.

[0007] Optionally, the metadata is generated in the following manner: obtaining the storage configuration information of each data source, wherein the storage configuration information is used to indicate the storage definition and structure definition of the data; establishing association information and processing rule information between the attribute information of each data and the storage definition and structure definition of the corresponding data in each data source; and obtaining the metadata based on the association information and the processing rule information.

[0008] Optionally, the attribute information of the data includes the fields of the data in each data source, the table information from which the data comes, the column information in the data table, and the relationship information between multiple tables.

[0009] Optionally, the method for exporting multi-source heterogeneous data further includes: after distributing the executable task to the target execution engine, if no data query result is received from the target execution engine, determining the failure type of the export task; estimating the retry duration based on the failure type of the export task; and after the retry duration is reached, re-distributing the executable task to the target execution engine.

[0010] Optionally, the step of redistributing the executable task to the target execution engine after the retry duration is reached includes: after the retry duration is reached, selecting a new execution engine as the target execution engine based on the failure type of the export task, the various data sources from which the data to be exported comes, and the current resource load of the execution engine; reconverting the data export task into an executable task adapted to the target execution engine, and redistributing the executable task to the target execution engine.

[0011] Optionally, converting the data export task into an executable task adapted to the target execution engine includes: determining the fields, table information, inter-table relationships, and processing rule information corresponding to each piece of data in the data to be exported based on the metadata and the data source from which the data to be exported originates; determining whether the processing rule information contains a corresponding user-defined function; if the processing rule information contains a corresponding user-defined function, generating an execution statement adapted to both the target execution engine and the corresponding data source based on the type of the target execution engine, the user-defined function, the fields, table information, and inter-table relationships of each piece of data in the data to be exported, and using the execution statement as the executable task; if the processing rule information does not contain a corresponding user-defined function, registering the user-defined function corresponding to the processing rule information, generating an execution statement adapted to both the target execution engine and the corresponding data source based on the registered user-defined function, the type of the target execution engine, the fields, table information, and inter-table relationships of each piece of data in the data to be exported, and using the execution statement as the executable task.

[0012] This invention also provides an apparatus for exporting multi-source heterogeneous data, comprising: an acquisition unit, configured to acquire descriptive information of data to be exported from a data export task when a data export task is received, wherein the descriptive information of the data to be exported indicates the attribute information of the data to be exported and the various data sources from which it originates, and the attribute information of the data to be exported is set based on preset metadata; a selection unit, configured to select a target execution engine based on the various data sources from which the data to be exported originates and the current resource load of the execution engine; a distribution unit, configured to convert the data export task into an executable task adapted to the target execution engine and distribute the executable task to the target execution engine, wherein the target execution engine performs data querying in the corresponding data source based on the executable task; and an output unit, configured to obtain and output the export result based on the data query result returned by the target execution engine.

[0013] This invention also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, performs the steps of any of the above-described methods for exporting multi-source heterogeneous data.

[0014] This invention also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of any of the above-described methods for exporting multi-source heterogeneous data when running the computer program.

[0015] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:

[0016] In this embodiment of the invention, when a data export task is received, descriptive information of the data to be exported is obtained from the data export task. This descriptive information indicates the attribute information of the data to be exported and the various data sources from which it originates. Since the attribute information of the data to be exported is set based on preset metadata, a target execution engine is selected according to the various data sources from which the data to be exported originates and the current resource load of the execution engine. The data export task is then converted into an executable task adapted to the target execution engine, and the executable task is distributed to the target execution engine. The target execution engine then performs data queries on the corresponding data sources based on the executable task. The export result is obtained and output based on the data query results returned by the target execution engine. Since the attribute information of the data to be exported obtained from the data export task is configured based on preset metadata, it helps users define data export tasks using a unified description language. Thus, when configuring the data export task, the differences between various data sources can be masked, eliminating the need for data synchronization between different sources. Data operations can be performed directly, and users do not need to concern themselves with the storage details of the underlying data sources, the specific execution engines, or the syntax differences and resource load of different engines. The optimal engine is automatically selected as the target execution engine. Furthermore, by converting the data export task into an executable task adapted to the target execution engine, the target execution engine can query data from various data sources. The export result is obtained based on the data query results returned by the target execution engine, thereby achieving the fusion and export of heterogeneous data from different data sources and improving the convenience of data export.

[0017] Furthermore, after the executable task is distributed to the target execution engine, if no data query result is received from the target execution engine, the export task failure type is determined; the retry duration is estimated based on the export task failure type; after the retry duration is reached, the executable task is re-distributed to the target execution engine. In this way, when an export task fails, data export is retried based on the retry duration determined according to the export task failure type, which improves the success rate of re-executed data export tasks while saving resource overhead. Attached Figure Description

[0018] Figure 1 This is a flowchart of a method for exporting multi-source heterogeneous data in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart of a metadata generation method according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the structure of a device for exporting multi-source heterogeneous data in an embodiment of the present invention. Detailed Implementation

[0021] As mentioned above, due to the different data storage formats of various data sources and the fact that each data source is independent, data fusion is not possible. Currently, when exporting data from each data source, only local data generation is possible. Users then need to fuse the data from each data source according to their actual requirements. For example, order data is stored in a primary data source (such as the data platform HIVE), and business marketing data based on these orders is stored in Alibaba Cloud PolarDB. The goal is to generate order information used in the business marketing data (the generated data contains attributes for both marketing and order information). The current common practice is to establish a data synchronization task to periodically synchronize business marketing data to HIVE, ensuring that business marketing data and order data are within the same data source. This is then done using syntax based on a single execution engine (HIVE / SPARK) to perform join operations within the same data source and generate the resulting data. However, this method of exporting multi-source heterogeneous data is quite cumbersome.

[0022] To address the aforementioned issues, in this embodiment of the invention, when a data export task is received, descriptive information of the data to be exported is obtained from the data export task. This descriptive information indicates the attribute information of the data to be exported and the various data sources from which it originates. Since the attribute information of the data to be exported is set based on preset metadata, a target execution engine is selected based on the various data sources from which the data to be exported originates and the current resource load of the execution engine. The data export task is then converted into an executable task adapted to the target execution engine, and the executable task is distributed to the target execution engine. The target execution engine then performs data queries on the corresponding data sources based on the executable task. The export result is obtained and output based on the data query results returned by the target execution engine. Since the attribute information of the data to be exported obtained from the data export task is configured based on preset metadata, it facilitates the use of a unified description language to define the data export task. Thus, when configuring the data export task, the differences between various data sources can be masked, eliminating the need for data synchronization between different sources. Data operations can be performed directly, and users do not need to concern themselves with the storage details of the underlying data source, the specific execution engine, or the syntax differences between different engines. The optimal engine is automatically selected as the target execution engine. Furthermore, by converting the data export task into an executable task adapted to the target execution engine, the target execution engine can query data from various data sources. The export result is obtained based on the data query results returned by the target execution engine, thereby achieving the fusion and export of heterogeneous data from different data sources and improving the convenience of data export.

[0023] To make the above-mentioned objectives, features and beneficial effects of the embodiments of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] This invention provides a method for exporting multi-source heterogeneous data. This method can be executed by a server, cloud platform, computer, etc., and is used to achieve the fusion and export of data from multiple data sources.

[0025] Reference Figure 1 The present invention provides a flowchart of a method for exporting multi-source heterogeneous data according to an embodiment of the present invention. The method for exporting multi-source heterogeneous data specifically includes the following steps.

[0026] Step 11: When a data export task is received, obtain the description information of the data to be exported from the data export task. The description information of the data to be exported is used to indicate the attribute information of the data to be exported and the various data sources from which it comes. The attribute information of the data to be exported is set based on preset metadata.

[0027] Step 12: Select the target execution engine based on the various data sources from which the data to be exported comes and the current resource load of the execution engine.

[0028] Step 13: Based on the metadata and the data source from which the data to be exported comes, the data export task is converted into an executable task adapted to the target execution engine, and the executable task is distributed to the target execution engine, which then performs data querying in the corresponding data source according to the executable task.

[0029] Step 14: Receive the data query results returned by the target execution engine and export the data query results.

[0030] In practice, the data export task received in step 11 can be obtained in a variety of ways. The following is an example of how the data export task is generated.

[0031] In some embodiments, a visual data export system is installed on the terminal, and configuration options can be displayed on the visual operation interface of the visual data export system. Users can select the data to be exported by operating the configuration options, and generate a data export task based on the selected data. Users can operate the configuration options using various suitable methods such as clicking and dragging.

[0032] In some embodiments, the visual interface can be configured with drag-and-drop operation units built based on source data and drag-and-drop assembly operation units. The location information of the data source is used as a selection criterion, and the tables of each data source are used as the smallest drag-and-drop operation units. Each drag-and-drop operation unit can also utilize field information from metadata and the transformation rules supported by each field. The drag-and-drop assembly operation unit configures the relationship between the selected unit and the newly added unit, selects the data columns to be exported and their corresponding transformation rules, adds filtering conditions, and after assembly, obtains the data export task and submits it for execution. Upon receiving the execution request, the configured table units, relationships, exported columns and processing rules, and filtering conditions are validated based on the metadata definition. After successful validation, a suitable execution engine is selected as the target execution engine, taking into account the location information of the submitted table units and the resource utilization of available execution engines. After the target execution engine is selected, the corresponding engine's converter is obtained, and the data export task is converted into an executable task adapted to the target execution engine. Furthermore, after submitting the executable task to the corresponding target execution engine, the task status can be monitored.

[0033] For example, configuration options are identified by field names. Users can select the data to be exported by dragging and dropping the required field names, and the field names will serve as attribute information for the data to be exported.

[0034] The visual interface can display all the data sources corresponding to the user, allowing the user to select the data source to export the data. The data source to export the data can be used as the attribute information of the data to be exported.

[0035] The display method of data sources varies in different application scenarios. Taking e-commerce as an example, different e-commerce platforms may use different data sources for data storage. Data sources can be displayed using the e-commerce platform's identifier or the store name on the e-commerce platform. For example, a user may have stores on e-commerce platforms A, B, and C. The data source for e-commerce platform A is 'a', for platform B it's 'b', and for platform C it's 'c'. The visual interface displays the stores on e-commerce platforms A, B, and C. There is a relationship between the stores on each e-commerce platform and their corresponding data sources. When a user selects an e-commerce store, the corresponding data source can be retrieved based on the selected store. If the user selects field name 'D' and e-commerce store 'C', the attribute information of the data to be exported can include field name 'D' and data source 'c'. Furthermore, the attribute information of the data to be exported can also include the user identifier. The user identifier is used to identify different users and is uniquely associated with each user. Based on the user identifier, the data corresponding to that user can be retrieved from the data source.

[0036] In other embodiments, user-input text can be received, and the text can be parsed to obtain a data export task based on the parsing results. Users can describe the task using natural language, which further simplifies user operations and enhances the flexibility and ease of task configuration.

[0037] In some other embodiments, user-input voice data can be received, and the voice data can be parsed to obtain a data export task based on the parsing results.

[0038] In this embodiment of the invention, the metadata is used to describe the storage definition and structure definition of data in various data sources. Typically, data sources store data in a row-and-column manner, that is, in a table manner. Therefore, the storage definition and structure definition can refer to the storage definition and structure definition of the table.

[0039] Each data point is configured with fields, and metadata can be used to describe the field name, the table it belongs to, and the relationships between the tables in each data source. Thus, even if different data sources have different storage or structural definitions, metadata can be used to associate and integrate the data storage and structural definitions of each data source, facilitating a unified format for display to users. This eliminates the need for users to focus on the differences between data sources, effectively shielding them from these discrepancies.

[0040] For example, if the attribute information of the data to be exported is field name D, according to the metadata description of field name D, we can know that field name D is d1 in data source a, belonging to table e, and field name D is d2 in data source b, belonging to table f.

[0041] In practical implementation, data sources can include relational databases such as MySQL, SQL Server, and Oracle, as well as PolarDB, the Hadoop Distributed File System (HDFS), HBase, and ADB. ADB is a database from Alibaba Cloud, short for AnalyticDB, which is good at query analysis in Online Analytical Processing (OLAP) scenarios.

[0042] In some embodiments, metadata can be generated in the following manner. The following is in conjunction with... Figure 2 The flowchart of the given metadata generation method illustrates the generation of metadata, which may include steps 21 to 23 below.

[0043] Step 21: Obtain the storage configuration information of each data source. The storage configuration information is used to indicate the storage definition and structure definition.

[0044] Storage definition can refer to the storage type of the data source.

[0045] Table structure definition refers to the definition of the table structure, which typically includes defining the table name, field names, data types, length, etc. A table can include one or more fields, and each field can contain multiple elements, such as field name, data type, length, precision, etc.

[0046] Step 22: Establish the association information and processing rules between each metadata and the corresponding data in each data source, including storage definitions, structure definitions, and corresponding data.

[0047] Step 23: Obtain the metadata based on the association information and the processing rule information.

[0048] The attribute information of the data includes the field names of the data in each data source, the table information from which the data comes, the column information in the data table, and the relationship information between multiple tables.

[0049] In some embodiments, metadata generation rules can be pre-configured to extract structured data from various data sources. The extracted data is then correlated and integrated with the data from each data source based on the pre-configured rules. By extracting structured data from each data source, a list of data source IP addresses can be provided during subsequent data export, a data asset list can be established, the required data to be exported can be received, and the storage location can be returned. This lays the foundation for masking storage differences between data sources, eliminating the need for users to worry about whether the data to be exported is within a single data source.

[0050] For example, each field name in the metadata is associated with its corresponding field names in various data sources. Similarly, for multiple fields in each data source that have a joint calculation relationship, the joint calculation between these fields is transformed according to the calculation rules and the relationships between the field names, resulting in a joint calculation between fields that conforms to the metadata generation rules. The processing rule information can include the joint calculation transformation relationships.

[0051] In one specific implementation of step 12, the target execution engine can be selected based on the various data sources from which the data to be exported comes and the current resource load of the execution engine. Specifically, the target execution engine is selected based on the type of each data source from which the data to be exported comes, the current remaining resources of the execution engine, and the attribute information of the data to be exported.

[0052] Different types of data sources require different execution engines, and different types of data to be exported require different execution engines. There may be multiple execution engines, and a single data source may support one or more execution engines. When determining the current execution engine, considering the type of each data source, the attribute information of the data to be exported, and the remaining resources of the execution engine, a more suitable target execution engine can be selected.

[0053] In some embodiments, the execution engine is selected by comprehensively considering various factors, such as the data transfer cost due to different storage locations, the current resource load of the engine (task execution queue length or current execution engine resource utilization), the appropriate execution engine for the data size queried by the task, and the business-mandated execution engine. Among these, the business-mandated engine has the highest priority.

[0054] The execution engine can be HIVE, SPARK, etc. HIVE is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. SPARK is a fast and general-purpose execution engine designed specifically for large-scale data processing.

[0055] There can be one or more target execution engines. When there are multiple target execution engines, they can run in parallel, meaning they can execute relatively independently and query data from their respective data sources.

[0056] In some non-limiting embodiments, the type of data source, the attribute information of the data to be exported, and the selection weight of the execution engine corresponding to the current remaining resources of the execution engine can be pre-configured, and the target execution engine is selected according to the selection weight.

[0057] In practice, a relationship can be established between the data source and the corresponding execution engine. When selecting the target execution engine, it can be chosen from the execution engines associated with the data source.

[0058] Furthermore, the selection weight varies depending on the current remaining amount of the execution engine; the smaller the current remaining amount, the smaller the selection weight. The current remaining amount can be either the remaining percentage of the execution engine's resources or the remaining amount of resources.

[0059] In one specific implementation of step 13, the fields, table information, inter-table relationships, and processing rule information corresponding to each piece of data in the data to be exported can be determined based on metadata and the data source from which the data to be exported originates. It is then determined whether the processing rule information contains a corresponding user-defined function. If the processing rule information contains a corresponding user-defined function, an execution statement adapted to both the target execution engine and the corresponding data source is generated based on the type of the target execution engine, the user-defined function, the fields, table information, and inter-table relationships of each piece of data in the data to be exported, and this execution statement is used as the executable task. If the processing rule information does not contain a corresponding user-defined function, the user-defined function corresponding to the processing rule information is registered. Based on the registered user-defined function, the type of the target execution engine, the fields, table information, and inter-table relationships of each piece of data in the data to be exported, an execution statement adapted to both the target execution engine and the corresponding data source is generated, and this execution statement is used as the executable task.

[0060] In practice, the processing rule information can describe which tables are used for field processing, the intersection, union, and difference relationships between the tables used, and which fields are extracted from which tables.

[0061] Processing rule information can be obtained from user-defined processing rules. A visual rule management system can be configured to manage the rule engine. For some commonly used rules, pre-defined rules can be built in, and their function names can be exposed in the visual operation interface of the rule management system for easy selection by users. Since there are many processing rules, it is difficult to enumerate them all; therefore, dynamic scripts can be used to define the rules.

[0062] Step 13 allows the table's storage and structure definitions to be retrieved from the metadata. These definitions are then converted into mapping execution logic for different execution engines, generating temporary logical datasets within the execution engines. These temporary logical datasets are then used to manipulate data in the corresponding data sources. Compared to directly using the native engines of various data sources, which requires creating different mapping tables for different data sources, specifying storage location information, and defining mapping relationships between fields, this invention's data export method enables cross-data source data export without secondary development. It is simple to use and develop, thus lowering the barrier to entry for data export.

[0063] In the specific implementation of step 14, the data query results are stored as a result set and exported as a file.

[0064] The result set can be exported to a preset storage device, file system, or database, so that users can use the data outside the data source.

[0065] In some embodiments, the data export task may further include processing rule information, which integrates the data query results returned by one or more target execution engines according to the processing rule information to obtain the export result.

[0066] As described above, when a data export task is received, the descriptive information of the data to be exported is obtained from the data export task. This descriptive information indicates the attribute information of the data to be exported and the various data sources from which it originates. Since the attribute information of the data to be exported is set based on preset metadata, a target execution engine is selected based on the various data sources from which the data to be exported originates and the current resource load of the execution engine. The data export task is then converted into an executable task adapted to the target execution engine, and the executable task is distributed to the target execution engine. The target execution engine then performs data queries on the corresponding data sources based on the executable task. The export result is obtained and output based on the data query results returned by the target execution engine. Since the attribute information of the data to be exported obtained from the data export task is configured based on preset metadata, it facilitates the use of a unified description language to define the data export task. Thus, when configuring the data export task, the differences between various data sources can be masked, eliminating the need for data synchronization between different sources. Data operations can be performed directly, and users do not need to concern themselves with the storage details of the underlying data sources, the specific execution engines, or the syntax differences and resource load of different engines. The optimal engine is automatically selected as the target execution engine. Furthermore, by converting the data export task into an executable task adapted to the target execution engine, the target execution engine can query data from various data sources. The export result is obtained based on the data query results returned by the target execution engine, thus enabling the simultaneous export of heterogeneous data from multiple different data sources, improving the convenience of data export for users. In addition, selecting the target execution engine based on the various data sources from which the data to be exported originates and the current resource load of the execution engine can improve the execution speed of the task, enhance resource utilization, and ensure system stability.

[0067] In specific implementation, after the executable task is distributed to the target execution engine, if no data query result is received from the target execution engine, the export task failure type is determined; the retry time is estimated based on the export task failure type; after the retry time is reached, the executable task is re-distributed to the target execution engine.

[0068] In some embodiments, when an export task fails, a fault code is returned. The fault code can be used to determine the type of export task failure. Different fault codes represent different types of export task failures.

[0069] In some implementations, different failure types of export tasks are configured with matching retry durations. This allows sufficient time for troubleshooting by determining the matching retry duration based on the failure type of the export task. This increases the probability that the target execution engine is in a normal state when the executable task is redistributed to it, thereby improving the data export success rate and efficiency.

[0070] In other embodiments, the cause of the export task failure is determined based on the type of export task failure, the time required to resolve the cause of the export task failure is estimated, and the retry time is estimated based on the estimated time required to resolve the cause of the export task failure.

[0071] Furthermore, the cause of the export task failure is determined based on the failure type, and the time required to resolve the failure is estimated. Based on the data sources from which the data to be exported originates and the current resource load of the execution engine, it is determined whether other suitable execution engines exist.

[0072] If no other suitable execution engine exists, the estimated retry time is calculated based on the estimated time required to resolve the cause of the export task failure.

[0073] If other suitable execution engines exist, the queuing time for the data export task to be executed by the other suitable execution engines and the queuing time for the data export task to be executed by the target execution engine are estimated. If the estimated queuing time for the data export task to be executed by the other suitable execution engines is less than the sum of the estimated time required to resolve the cause of the export task failure and the queuing time for the data export task to be executed by the target execution engine, then the other matching execution engine is selected as the target execution engine, and the executable task is distributed to the target execution engine.

[0074] In some embodiments, after the retry duration is reached, an execution engine is reselected as the target execution engine based on the failure type of the export task, the various data sources from which the data to be exported comes, and the current resource load of the execution engine; the data export task is reconstructed into an executable task adapted to the target execution engine, and the executable task is distributed to the target execution engine.

[0075] When the export task fails due to an execution engine unavailability error, the system determines whether there are other suitable execution engines based on the data sources from which the data to be exported originates and the current resource load of the execution engine. If other suitable execution engines exist, one is selected as the target execution engine, and the executable task is distributed to that target execution engine. If no other suitable execution engines exist, a maximum retry interval is set as the retry duration, and an execution engine availability monitoring task is initiated. Once the execution engine recovers, the list of tasks to be executed can be restored immediately. If the retry duration is too long, the target execution engine is selected again. By monitoring the execution engine status and configuring the maximum retry interval, the system can prevent tasks from being permanently unable to execute due to the engine status monitoring task seemingly freezing, and it can also detect newly added execution engines to ensure that tasks can be executed as quickly as possible.

[0076] When the export task fails due to an insufficient resource error, the system determines whether there are other suitable execution engines, such as those with more abundant resources, based on the data sources from which the data to be exported originates and the current resource load of the execution engine. If other suitable execution engines exist, one is selected as the target execution engine, and the executable task is distributed to that target execution engine, i.e., retrying immediately. If no other suitable execution engines exist, the executable task is placed in a task queue waiting for execution. At this point, the task priority is set based on the number of times it has been executed and the task submission time. The waiting queue will schedule tasks based on priority, executing higher-priority tasks first.

[0077] If the export task fails due to other errors, it will be retried on the current execution engine, with a set interval (related to the number of times it has been executed). If it still fails after the set number of attempts, the system will determine whether there are other suitable execution engines based on the data sources from which the data to be exported comes and the current resource load of the execution engine. If other suitable execution engines exist, a target execution engine will be selected from them, and the data export task will be converted into an executable task adapted to the target execution engine and distributed to the target execution engine. If no other available execution engines exist, the system will continue to retry on the current execution engine.

[0078] Furthermore, a maximum number of retries can be set for failed data export tasks. If the maximum number of retries is not reached, and other suitable execution engines exist, the current resource load of the selected engine will be prioritized, choosing one with higher resource availability as the target computing engine. If the maximum number of retries is reached, and other suitable execution engines exist, an unused execution engine will be prioritized as the target computing engine.

[0079] In other implementations, after the retry duration is reached, the executable task can be directly redistributed to the target execution engine.

[0080] In practice, due to the lag in resource consumption by tasks, the resource load is low when an executable task is submitted, resulting in n executable tasks being submitted. However, as these tasks execute, resource usage increases, eventually becoming insufficient to support the simultaneous execution of n tasks, leading to resource deadlock. To avoid this, each execution engine is abstracted into n resource slots. By default, an executable task occupies one resource slot during execution. As tasks fail and are retried, the number of resource slots occupied by the task is increased based on the number of retried errors. The more retries, the more resource slots are occupied. By combining the remaining resource slots of each execution engine to determine the target execution engine, resource deadlock can be effectively avoided.

[0081] In practice, upon receiving a data export task, the task can be validated to determine if the fields conform to the set field configuration rules. If the validation passes, step 11 is executed. If the validation fails, the reason for the failure is returned, allowing users to intuitively understand why the validation failed and facilitating modifications to the data export task.

[0082] This invention also provides a device for exporting multi-source heterogeneous data. Figure 3 This is a schematic diagram of the structure of a multi-source heterogeneous data export device according to an embodiment of the present invention, combined with... Figure 3The multi-source heterogeneous data export device 30 includes: an acquisition unit 31, used to acquire description information of the data to be exported from the data export task when a data export task is received, the description information of the data to be exported being used to indicate the attribute information of the data to be exported and the various data sources from which it comes, the attribute information of the data to be exported being set based on preset metadata; a selection unit 32, used to select a target execution engine according to the various data sources from which the data to be exported comes and the current resource load of the execution engine; a distribution unit 33, used to convert the data export task into an executable task adapted to the target execution engine, and distribute the executable task to the target execution engine, the target execution engine performing data query in the corresponding data source according to the executable task; and an export unit 34, used to obtain and output the export result based on the data query result returned by the target execution engine.

[0083] In specific implementation, the working principle and process of the multi-source heterogeneous data export device 30 can be found in the description of the multi-source heterogeneous data export method in the above embodiments, and will not be repeated here.

[0084] This invention also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, performs the steps of any of the above-described methods for exporting multi-source heterogeneous data.

[0085] This invention also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of any of the above-described methods for exporting multi-source heterogeneous data when running the computer program.

[0086] It should be noted that the sequence number of each step in this embodiment does not represent a limitation on the execution order of each step.

[0087] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in any computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0088] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for exporting multi-source heterogeneous data, characterized in that, For e-commerce scenarios, different e-commerce platforms can use different data sources for data storage. The export method includes: When a data export task is received, the descriptive information of the data to be exported is obtained from the data export task. The descriptive information of the data to be exported is used to indicate the attribute information of the data to be exported and the various data sources from which it comes. The attribute information of the data to be exported is set based on preset metadata. The metadata is generated in the following way: obtaining the storage configuration information of each data source, which is used to indicate the storage definition and structure definition of the data; establishing association information and processing rule information between the attribute information of each data and the corresponding data storage definition and structure definition in each data source; obtaining the metadata based on the association information and the processing rule information; the attribute information of the data includes the fields of the data in each data source, the table information from which the data comes, the column information of the data table, and the association information between multiple tables; the processing rule information describes which tables are used for field processing, the intersection, union, and difference relationships between the tables, and which fields are extracted from which tables. Select the target execution engine based on the various data sources from which the data to be exported comes and the current resource load of the execution engine; Based on the metadata and the data source from which the data to be exported comes, the data export task is converted into an executable task adapted to the target execution engine, and the executable task is distributed to the target execution engine, which then performs data querying in the corresponding data source according to the executable task. The exported results are obtained and output based on the data query results returned by the target execution engine. The step of converting the data export task into an executable task adapted to the target execution engine includes: Based on the metadata and the data source from which the data to be exported comes, determine the fields, table information, inter-table relationships, and processing rules for each piece of data in the data to be exported. Determine whether the processing rule information has a corresponding user-defined function; If the processing rule information has a corresponding user-defined function, an execution statement that is compatible with the target execution engine and the corresponding data source is generated based on the type of the target execution engine, the user-defined function, the fields of each data in the data to be exported, the table information, and the relationship information between the tables. The execution statement is then used as the executable task. If the processing rule information does not have a corresponding user-defined function, register the user-defined function corresponding to the processing rule information. Based on the registered user-defined function, the type of the target execution engine, the fields of each data in the data to be exported, the table information, and the relationship information between the tables, generate an execution statement that is compatible with the target execution engine and the corresponding data source, and use the execution statement as the executable task. The data export task also includes processing rule information. Obtaining the export result based on the data query results returned by the target execution engine includes: integrating the data query results returned by one or more target execution engines according to the processing rule information to obtain the export result.

2. The method for exporting multi-source heterogeneous data as described in claim 1, characterized in that, The step of selecting a target execution engine based on the various data sources from which the data to be exported originates and the current resource load of the execution engine includes: The target execution engine is selected based on the type of each data source from which the data to be exported comes, the current remaining resources of the execution engine, and the attribute information of the data to be exported.

3. The method for exporting multi-source heterogeneous data as described in claim 1, characterized in that, Also includes: After the executable task is distributed to the target execution engine, if no data query result is received from the target execution engine, the export task failure type is determined. Estimate retry time based on the failure type of the exported task; After the retry duration is reached, the executable task is redistributed to the target execution engine.

4. The method for exporting multi-source heterogeneous data as described in claim 3, characterized in that, The step of redistributing the executable task to the target execution engine after the retry duration has been reached includes: After the retry duration is reached, the execution engine is reselected as the target execution engine based on the failure type of the export task, the data sources from which the data to be exported comes, and the current resource load of the execution engine. The data export task is reconstructed into an executable task adapted to the target execution engine, and the executable task is distributed to the target execution engine.

5. A device for exporting multi-source heterogeneous data, characterized in that, For e-commerce scenarios, different e-commerce platforms can use different data sources for data storage. The export device includes: The acquisition unit is used to acquire description information of the data to be exported from the data export task when a data export task is received. The description information of the data to be exported is used to indicate the attribute information of the data to be exported and the various data sources from which it comes. The attribute information of the data to be exported is set based on preset metadata. The metadata is generated in the following way: acquiring the storage configuration information of each data source, which is used to indicate the storage definition and structure definition of the data; establishing association information and processing rule information between the attribute information of each data and the corresponding data storage definition and structure definition in each data source; obtaining the metadata based on the association information and the processing rule information; the attribute information of the data includes the fields of the data in each data source, the table information from which the data comes, the column information of the data table, and the association information between multiple tables; the processing rule information describes which tables are used for field processing, the intersection, union, and difference relationships between the tables, and which fields are extracted from which tables. The selection unit is used to select the target execution engine based on the various data sources from which the data to be exported comes and the current resource load of the execution engine; The distribution unit is used to convert the data export task into an executable task adapted to the target execution engine, and distribute the executable task to the target execution engine, so that the target execution engine can perform data query in the corresponding data source according to the executable task; The output unit is used to obtain and output the export result based on the data query result returned by the target execution engine; The distribution unit is configured to determine, based on the metadata and the data source from which the data to be exported originates, the fields, table information, inter-table relationships, and processing rule information corresponding to each piece of data in the data to be exported; determine whether the processing rule information contains a corresponding user-defined function; if the processing rule information contains a corresponding user-defined function, generate an execution statement that is compatible with both the target execution engine and the corresponding data source based on the type of the target execution engine, the user-defined function, the fields, table information, and inter-table relationships of each piece of data in the data to be exported, and use the execution statement as the executable task; if the processing rule information does not contain a corresponding user-defined function, register the user-defined function corresponding to the processing rule information, and generate an execution statement that is compatible with both the target execution engine and the corresponding data source based on the registered user-defined function, the type of the target execution engine, the fields, table information, and inter-table relationships of each piece of data in the data to be exported, and use the execution statement as the executable task. The data export task also includes processing rule information. The output unit is used to integrate the data query results returned by one or more target execution engines according to the processing rule information to obtain the export result.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the method for deriving multi-source heterogeneous data as described in any one of claims 1 to 4.

7. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the method for deriving multi-source heterogeneous data according to any one of claims 1 to 4.

Images