Multi-source data element fusion integration scheduling system and method based on data weaving

Abstract

The application discloses a multi-source data element fusion integrated scheduling system and method based on data weaving, relates to the technical field of data analysis, and comprises the following steps: constructing a data element directory; parsing a required data element set, fusion logic and maximum execution time, traversing the data element directory, semantically matching and locating the storage positions of each data element, and calculating real-time accessibility indexes; judging whether to generate a logical execution plan; when it is judged that the logical execution plan is generated, a data operation node sequence is parsed, each data operation node is specified with an operation type and a bound data element, computing resources are allocated, and the execution time of each data operation node is estimated; each data operation node is executed in turn, the actual execution state is recorded, and automatic retry is performed when a data operation node fails; and if all data operation nodes are successfully executed, a fused data element result set is output, and the application realizes multi-source data element fusion integrated scheduling.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, specifically to a multi-source data element fusion and integration scheduling system and method based on data weaving. Background Technology

[0002] As enterprises deepen their digital transformation, the integration of multi-source heterogeneous data elements has become an important part of data analysis and business intelligent decision-making. Under the data weaving architecture, the efficient and reliable scheduling of data elements from different data sources, and the realization of automated extraction, transformation and loading, are important challenges currently facing the field of data integration.

[0003] However, existing data integration and scheduling methods still have the following problems in practical applications: First, scheduling decisions lack proactive awareness of the real-time status of data sources. Traditional scheduling systems typically use fixed-time triggering or manual configuration, which cannot dynamically perceive the load changes and data timeliness of each data source. This leads to scheduling being initiated even when the data source is busy or the data has expired, resulting in task execution timeouts or inaccurate results, reducing the success rate of scheduling and resource utilization. Second, the scheduling process lacks sufficient intelligence and adaptive threshold mechanisms. Most existing technologies rely on preset fixed thresholds to determine whether a data source is available, which is difficult to adapt to the dynamic changes of different data sources. This can easily lead to overly strict or lenient scheduling, increasing the burden of manual parameter tuning. In addition, the fault tolerance during task execution is limited. There is a lack of automatic retry and interrupt replay mechanisms for node-level failures. Once a data operation node fails, the entire task fails, and there is a lack of recovery strategies. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-source data element fusion and integration scheduling system and method based on data weaving to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-source data element fusion and integration scheduling method based on data weaving, the method comprising the following steps: Actively collect metadata from multi-source heterogeneous data to construct a unified data element catalog; Upon receiving the data fusion and integration task, the required set of data elements, fusion logic, and maximum execution time are parsed out. The data element directory is traversed and the storage location of each data element is located by semantic matching. Real-time accessibility indicators are calculated. The accessibility metrics for each required data element are compared with the accessibility threshold to determine whether a logical execution plan should be generated. When determining the generated logical execution plan, the sequence of data operation nodes is parsed out. Each data operation node specifies the operation type and the data elements it is bound to. Computing resources are allocated according to the order of the data operation nodes, and the execution time of each data operation node is estimated. Each data operation node is executed sequentially, the actual execution status is recorded, and automatic retry is performed when a data operation node fails to execute. If all data operation nodes execute successfully, the merged data element result set will be output.

[0006] Active metadata collection from multi-source heterogeneous data to construct a unified data element catalog involves the following steps: Send metadata collection requests to two or more heterogeneous data sources, each data source D. n Return its active metadata, which includes connection information, data table structure, and data update frequency. The connection information is used to access the data source, and the data table structure contains the table name of each table and the field names and data types contained in the table. n = 1, 2, 3, ..., N, where N represents the total number of data sources. All returned active metadata is aggregated, and all fields are parsed from each data table structure. Each field is uniquely identified by its "data source identifier, table name, and field name," and a data element node is generated for each field. All data element nodes constitute a unified data element catalog. The m-th data element node is denoted as F. m Each data feature node includes a data source identifier, table name, field name, and data type, where m = 1, 2, 3, ..., M, and M represents the total number of data feature nodes.

[0007] Upon receiving the data fusion and integration task, the required set of data elements, fusion logic, and maximum execution time are parsed out. The data element directory is traversed, and the storage location of each data element is located through semantic matching. Real-time accessibility indicators are calculated. Specific steps include: Upon receiving a data fusion and integration task request, task parameters are parsed from the request. These parameters include the set of data elements R required by the task, the fusion logic sequence, and the maximum allowed execution time, where R = {F1, F2, ..., F...} k ,...,F K}, F k This represents the k-th data element node in the constructed data element catalog, where K represents the number of elements required for the task. The fusion logic sequence includes connection, aggregation, filtering, and mapping. Based on the set of data elements R required for the task, traverse the data element directory and locate each data element node F through semantic matching. k The physical storage location; For each data feature node F kCalculate its real-time accessibility metric A(F) k ,t)=α×L(F k ,t)+β×U(F k ,t), where L(F k ,t) represents data element node F k The load factor of the data source at time t, U(F) k ,t)=min(1,t-t1(F k )) / ΔT max ), t1(F k ) represents data element node F k The latest update time of the corresponding data, ΔT max This represents the preset maximum allowable data delay, with the same unit as t. α and β are preset weights.

[0008] The accessibility metrics for each required data element are compared with the accessibility threshold to determine whether a logical execution plan should be generated. Specific steps include: Each required data element node F k Accessibility index A(F) k ,t) is compared with the preset accessibility threshold A0. If all data element nodes F k All satisfy A(F) k If ,t)≤A0, then the scheduling conditions are met, a logical execution plan P is generated, and subsequent steps are entered. In this process, the accessibility index values ​​of all scheduled data element nodes in the period before the current scheduling time are sorted in ascending order, and the Xth percentile is taken as the current accessibility threshold. X can be selected as 80, and the period can be selected as the past 24 hours. If any data element node F exists k A(F) is not satisfied k If ,t)≤A0, then return an insufficient resource message to the requester and list the data element nodes that do not meet the conditions, prompting the user to choose an alternative data source or reduce the constraint requirements.

[0009] When determining the generated logical execution plan, the sequence of data operation nodes is parsed out. Each data operation node specifies the operation type and its bound data elements. Computational resources are allocated according to the order of the data operation nodes, and the execution time of each data operation node is estimated. The specific steps include: Based on the generated logical execution plan P, the sequence of data operation nodes contained in the plan is parsed out, denoted as O={o1,o2,…,o…} i ,...,o n Each data operation node o i This includes the operation type and its associated data element node F. k , where o iThis represents the i-th data operation node parsed from the logical execution plan P, where n represents the total number of nodes. The operation types include source data extraction, feature transformation and association, and target loading. Allocate computing resources according to the order of data operation nodes, and record each data operation node. i The estimated execution time T1(o i )=D(o i ) / R1, where D(o i ) represents the data operation node o i R1 represents the amount of data to be processed, and R1 represents the amount of computing resources allocated to this data operation node.

[0010] Each data operation node is executed sequentially, its actual execution status is recorded, and automatic retry is performed if a data operation node fails. Specific steps include: During scheduling, each data operation node is triggered sequentially. i Record the actual execution time T0(o) of each data operation node. i ) and state S(o i ), where state S(o i This includes success and failure; If state S(o i If the test fails, an automatic retry mechanism is initiated, with the number of retries denoted as Ry. The number of retries is determined when Ry ≤ R. max If the data operation node is re-executed, the task is marked as failed and an interrupt replay is performed. Where R... max This indicates the preset maximum number of retries.

[0011] If all data operation nodes execute successfully, the merged data element result set will be output. The specific steps include: If all data operation nodes execute successfully, the fused data element result set will be output to the target location, which is specified by the target loading node in the data operation nodes, including a database or file, and the metadata lineage information will be updated, recording the data source, transformation operation and output path of this scheduling. If there are failures and retries are exhausted, an error report is returned to the task submitter, which lists the failed data operation nodes.

[0012] A multi-source data element fusion and integration scheduling system based on data weaving. This system includes: The system includes a metadata acquisition module, a task parsing module, a scheduling decision module, and an execution output module. The output of the metadata acquisition module is connected to the input of the task parsing module. The output of the task parsing module is connected to the input of the scheduling decision module. The output of the scheduling decision module is connected to the input of the execution output module.

[0013] The metadata acquisition module is used to collect active metadata from multi-source heterogeneous data and construct a unified data element catalog. The task parsing module includes a parameter parsing unit, a positioning unit, and an indicator calculation unit; The parameter parsing unit is used to receive the fusion task, parse the required data element set, fusion logic, and maximum execution time; the positioning unit is used to traverse the directory and semantically match to locate the physical storage location of each data element; the index calculation unit is used to calculate the accessibility index of each data element based on the load rate and data timeliness.

[0014] The scheduling decision module includes a threshold management unit, a comparison and judgment unit, and a plan generation unit; The threshold management unit is used to dynamically determine the accessibility threshold based on historical scheduling records; the comparison and judgment unit compares each indicator with the threshold to determine whether to generate a logical execution plan; the plan generation unit is used to generate a logical execution plan when the conditions are met, otherwise it returns a resource shortage prompt. The execution output module includes a node parsing unit, an execution scheduling unit, and a result output unit; The node parsing unit is used to parse the logical plan into a sequence of data operation nodes, specifying the operation type and binding elements; the execution scheduling unit is used to allocate resources in sequence, estimate the execution time, execute nodes in sequence and record the status, and automatically retry when it fails; the result output unit is used to output the fusion result set and update the lineage information when successful, and return an error report when it fails.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. By actively collecting metadata from multi-source heterogeneous data, a unified data element catalog is constructed, and accessibility indicators are calculated based on the real-time load rate and data timeliness of the data source where each data element is located. Unlike the scheduling methods that are triggered at fixed times or manually configured in the existing technology, this invention can avoid initiating scheduling when the data source is busy or the data is expired, thereby improving resource utilization. 2. Based on the accessibility threshold dynamically determined from historical scheduling records, the indicators of each required data element are compared with the threshold. Only when all conditions are met is a logical execution plan generated. Unlike existing technologies that rely on static thresholds or manual parameter tuning, this invention can adaptively adjust the scheduling admission conditions according to changes in the data source status. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the multi-source data element fusion and integration scheduling method based on data weaving, as described in this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, this invention provides a technical solution: a multi-source data element fusion and integration scheduling method based on data weaving, which includes the following steps: Actively collect metadata from multi-source heterogeneous data to construct a unified data element catalog; Upon receiving the data fusion and integration task, the required set of data elements, fusion logic, and maximum execution time are parsed out. The data element directory is traversed and the storage location of each data element is located by semantic matching. Real-time accessibility indicators are calculated. The accessibility metrics for each required data element are compared with the accessibility threshold to determine whether a logical execution plan should be generated. When determining the generated logical execution plan, the sequence of data operation nodes is parsed out. Each data operation node specifies the operation type and the data elements it is bound to. Computing resources are allocated according to the order of the data operation nodes, and the execution time of each data operation node is estimated. Each data operation node is executed sequentially, the actual execution status is recorded, and automatic retry is performed when a data operation node fails to execute. If all data operation nodes execute successfully, the merged data element result set will be output.

[0019] Active metadata collection from multi-source heterogeneous data to construct a unified data element catalog involves the following steps: Send metadata collection requests to two or more heterogeneous data sources, each data source D. n Return its active metadata, which includes connection information, data table structure, and data update frequency. The connection information is used to access the data source, and the data table structure contains the table name of each table and the field names and data types contained in the table. n = 1, 2, 3, ..., N, where N represents the total number of data sources. All returned active metadata is aggregated, and all fields are parsed from each data table structure. Each field is uniquely identified by its "data source identifier, table name, and field name," and a data element node is generated for each field. All data element nodes constitute a unified data element catalog. The m-th data element node is denoted as F. m Each data feature node includes a data source identifier, table name, field name, and data type, where m = 1, 2, 3, ..., M, and M represents the total number of data feature nodes.

[0020] Upon receiving the data fusion and integration task, the required set of data elements, fusion logic, and maximum execution time are parsed out. The data element directory is traversed, and the storage location of each data element is located through semantic matching. Real-time accessibility indicators are calculated. Specific steps include: Upon receiving a data fusion and integration task request, task parameters are parsed from the request. These parameters include the set of data elements R required by the task, the fusion logic sequence, and the maximum allowed execution time, where R = {F1, F2, ..., F...} k ,...,F K}, F k This represents the k-th data element node in the constructed data element catalog, where K represents the number of elements required for the task. The fusion logic sequence includes connection, aggregation, filtering, and mapping. Based on the set of data elements R required for the task, traverse the data element directory and locate each data element node F through semantic matching. k The physical storage location; For each data feature node F k Calculate its real-time accessibility metric A(F) k ,t)=α×L(F k ,t)+β×U(F k ,t), where L(F k ,t) represents data element node F k The load factor of the data source at time t, U(F) k ,t)=min(1,t-t1(F k )) / ΔT max ), t1(F k ) represents data element node F k The latest update time of the corresponding data, ΔT max This represents the preset maximum allowable data delay, with the same unit as t. α and β are preset weights.

[0021] The accessibility metrics for each required data element are compared with the accessibility threshold to determine whether a logical execution plan should be generated. Specific steps include: Each required data element node F k Accessibility index A(F) k ,t) is compared with the preset accessibility threshold A0. If all data element nodes F k All satisfy A(F) kIf ,t)≤A0, then the scheduling conditions are met, a logical execution plan P is generated, and subsequent steps are entered. In this process, the accessibility index values ​​of all scheduled data element nodes in the period before the current scheduling time are sorted in ascending order, and the Xth percentile is taken as the current accessibility threshold. X can be selected as 80, and the period can be selected as the past 24 hours. If any data element node F exists k A(F) is not satisfied k If ,t)≤A0, then a resource shortage message is returned to the requester, listing the data element nodes that do not meet the conditions, and prompting the user to choose an alternative data source or reduce the constraint requirements.

[0022] When determining the generated logical execution plan, the sequence of data operation nodes is parsed out. Each data operation node specifies the operation type and its bound data elements. Computational resources are allocated according to the order of the data operation nodes, and the execution time of each data operation node is estimated. The specific steps include: Based on the generated logical execution plan P, the sequence of data operation nodes contained in the plan is parsed out, denoted as O={o1,o2,…,o…} i ,...,o n Each data operation node o i This includes the operation type and its associated data element node F. k , where o i This represents the i-th data operation node parsed from the logical execution plan P, where n represents the total number of nodes. The operation types include source data extraction, feature transformation and association, and target loading. Allocate computing resources according to the order of data operation nodes, and record each data operation node. i The estimated execution time T1(o i )=D(o i ) / R1, where D(o i ) represents the data operation node o i R1 represents the amount of data to be processed, and R1 represents the amount of computing resources allocated to this data operation node.

[0023] Each data operation node is executed sequentially, its actual execution status is recorded, and automatic retry is performed if a data operation node fails. Specific steps include: During scheduling, each data operation node is triggered sequentially. i Record the actual execution time T0(o) of each data operation node. i ) and state S(o i ), where state S(o i This includes success and failure; If state S(o i If the test fails, an automatic retry mechanism is initiated, with the number of retries denoted as Ry. The number of retries is determined when Ry ≤ R.max If the data operation node is re-executed, the task is marked as failed and an interrupt replay is performed. Where R... max This indicates the preset maximum number of retries.

[0024] If all data operation nodes execute successfully, the merged data element result set will be output. The specific steps include: If all data operation nodes execute successfully, the fused data element result set will be output to the target location, which is specified by the target loading node in the data operation nodes, including a database or file, and the metadata lineage information will be updated, recording the data source, transformation operation and output path of this scheduling. If there are failures and retries are exhausted, an error report is returned to the task submitter, which lists the failed data operation nodes.

[0025] In Example 1: Metadata collection requests are sent to the first database and the second database respectively. Different databases return corresponding return information. Each field is parsed from these return information and data element nodes are generated with "data source identifier, table name, and field name" as unique identifiers. All nodes together constitute a unified data element catalog. The upper-layer application submits a data fusion and integration task, requiring the fields generated in the past five minutes to be correlated and output to a distributed file, with a maximum execution time of two minutes. The task request is used to parse out the required set of data elements, the fusion logic sequence, and the maximum execution time. The fusion logic includes joining, aggregation, filtering, and mapping, and records within the past five minutes are filtered out. The system traverses the data element catalog and locates the physical storage location of these data elements through semantic matching. At 10:00 AM on a certain weekday, it is detected in real time that the load rate of the first database is at a high level, while the load rate of the second database is relatively stable. A maximum allowable data delay is preset, such as five minutes. Based on this information, a real-time accessibility index is calculated for each data element. The higher the load rate and the older the data, the larger the index value. To determine whether scheduling is currently suitable, the accessibility index values ​​of all scheduled data element nodes over the past 24 hours were reviewed, sorted from smallest to largest, and the value at the 80th position was taken as the current accessibility threshold. This threshold reflects the upper limit of accessibility allowed by most successful scheduling over the past day. After calculation, the data elements on the first database exceeded the threshold due to excessive load, while the data elements on the second database met the requirements. Since not all required data elements met the requirements, scheduling was deemed temporarily unsuitable. A resource shortage message was then returned to the upper-layer application, explicitly indicating that the data source containing a certain field in the first database was currently under high load. The user was advised to wait for a while before trying again, or to choose another backup database with lower load as an alternative data source. The upper-layer application accepted the suggestion, switched to the backup database, and resubmitted the task. At this time, the accessibility indicators of all required data elements did not exceed the dynamic threshold, and the scheduling conditions were met, generating a logical execution plan. The logical execution plan is parsed into a sequence of data operation nodes. Computational resources are allocated to each node according to the node order, and the execution time is estimated based on the expected amount of data to be processed by each node. During the scheduling process, each data operation node is triggered sequentially. The first two extraction nodes were executed successfully, although the actual execution time was slightly longer than expected but still within an acceptable range. When the third associated node was executed, it failed on the first attempt due to temporary memory shortage. The automatic retry mechanism was immediately activated, and the node was re-executed after a short wait. The second attempt was successful. The fourth loading node also successfully wrote the results to the target file. After all nodes were executed successfully, the fused dataset was output to the specified location, and the metadata lineage information was automatically updated. The data source, the transformation operations performed, and the output path of this scheduling were recorded for future auditing and problem tracing.

[0026] A multi-source data element fusion and integration scheduling system based on data weaving. This system includes: The system includes a metadata acquisition module, a task parsing module, a scheduling decision module, and an execution output module. The output of the metadata acquisition module is connected to the input of the task parsing module. The output of the task parsing module is connected to the input of the scheduling decision module. The output of the scheduling decision module is connected to the input of the execution output module.

[0027] The metadata acquisition module is used to collect active metadata from multi-source heterogeneous data and construct a unified data element catalog. The task parsing module includes a parameter parsing unit, a positioning unit, and an indicator calculation unit; The parameter parsing unit is used to receive the fusion task, parse the required data element set, fusion logic, and maximum execution time; the positioning unit is used to traverse the directory and semantically match to locate the physical storage location of each data element; the index calculation unit is used to calculate the accessibility index of each data element based on the load rate and data timeliness.

[0028] The scheduling decision module includes a threshold management unit, a comparison and judgment unit, and a plan generation unit; The threshold management unit is used to dynamically determine the accessibility threshold based on historical scheduling records; the comparison and judgment unit compares each indicator with the threshold to determine whether to generate a logical execution plan; the plan generation unit is used to generate a logical execution plan when the conditions are met, otherwise it returns a resource shortage prompt. The execution output module includes a node parsing unit, an execution scheduling unit, and a result output unit; The node parsing unit is used to parse the logical plan into a sequence of data operation nodes, specifying the operation type and binding elements; the execution scheduling unit is used to allocate resources in sequence, estimate the execution time, execute nodes in sequence and record the status, and automatically retry when it fails; the result output unit is used to output the fusion result set and update the lineage information when successful, and return an error report when it fails.

[0029] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A multi-source data element fusion and integration scheduling method based on data weaving, characterized in that: The method includes the following steps: Actively collect metadata from multi-source heterogeneous data to construct a unified data element catalog; Upon receiving the data fusion and integration task, the required set of data elements, fusion logic, and maximum execution time are parsed out. The data element directory is traversed and the storage location of each data element is located by semantic matching. Real-time accessibility indicators are calculated. Upon receiving a data fusion and integration task request, task parameters are parsed from the request. These parameters include the set of data elements R required by the task, the fusion logic sequence, and the maximum allowed execution time, where R = {F1, F2, ..., F...} k ,...,F K }, F k This represents the k-th data element node in the constructed data element catalog, where K represents the number of elements required for the task. The fusion logic sequence includes connection, aggregation, filtering, and mapping. Based on the set of data elements R required for the task, traverse the data element directory and locate each data element node F through semantic matching. k The physical storage location; For each data feature node F k Calculate its real-time accessibility metric A(F) k ,t)=α×L(F k ,t)+β×U(F k ,t), where L(F k ,t) represents data element node F k The load factor of the data source at time t, U(F) k ,t)=min(1,t-t1(F k )) / ΔT max ), t1(F k ) represents data element node F k The latest update time of the corresponding data, ΔT max This represents the preset maximum allowable data delay, where α and β are preset weights. The accessibility metrics for each required data element are compared with the accessibility threshold to determine whether a logical execution plan should be generated. When determining the generated logical execution plan, the sequence of data operation nodes is parsed out. Each data operation node specifies the operation type and the data elements it is bound to. Computing resources are allocated according to the order of the data operation nodes, and the execution time of each data operation node is estimated. Each data operation node is executed sequentially, the actual execution status is recorded, and automatic retry is performed when a data operation node fails to execute. If all data operation nodes execute successfully, the merged data element result set will be output.

2. The multi-source data element fusion and integration scheduling method based on data weaving according to claim 1, characterized in that: Active metadata collection from multi-source heterogeneous data to construct a unified data element catalog involves the following steps: Send metadata collection requests to two or more heterogeneous data sources, each data source D. n Return its active metadata, which includes connection information, data table structure, and data update frequency. The connection information is used to access the data source, and the data table structure contains the table name of each table and the field names and data types contained in the table. n = 1, 2, 3, ..., N, where N represents the total number of data sources. All returned active metadata is aggregated, and all fields are parsed from the data table structure. Each field is uniquely identified by its "data source identifier, table name, and field name," and a data element node is generated for each field. All data element nodes constitute a unified data element catalog. The m-th data element node is denoted as F. m Each data feature node includes a data source identifier, table name, field name, and data type, where m = 1, 2, 3, ..., M, and M represents the total number of data feature nodes.

3. The multi-source data element fusion and integration scheduling method based on data weaving according to claim 2, characterized in that: The accessibility metrics for each required data element are compared with the accessibility threshold to determine whether a logical execution plan should be generated. Specific steps include: Each required data element node F k Accessibility index A(F) k ,t) is compared with the preset accessibility threshold A0. If all data element nodes F k All satisfy A(F) k If ,t)≤A0, then the scheduling condition is met, a logical execution plan P is generated, and the process proceeds to the next step; If any data element node F exists k A(F) is not satisfied k If ,t)≤A0, then return an insufficient resource message to the requester and list the data element nodes that do not meet the conditions, prompting the user to choose an alternative data source or reduce the constraint requirements.

4. The multi-source data element fusion and integration scheduling method based on data weaving according to claim 3, characterized in that: When determining the generated logical execution plan, the sequence of data operation nodes is parsed out. Each data operation node specifies the operation type and its bound data elements. Computational resources are allocated according to the order of the data operation nodes, and the execution time of each data operation node is estimated. The specific steps include: Based on the generated logical execution plan P, the sequence of data operation nodes contained in the plan is parsed out, denoted as O={o1,o2,…,o…} i ,...,o n Each data operation node o i This includes the operation type and its associated data element node F. k , where o i This represents the i-th data operation node parsed from the logical execution plan P, where n represents the total number of nodes. The operation types include source data extraction, feature transformation and association, and target loading. Allocate computing resources according to the order of data operation nodes, and record each data operation node. i The estimated execution time T1(o i )=D(o i ) / R1, where D(Oᵢ) represents the data operation node o i R1 represents the amount of data to be processed, and R1 represents the amount of computing resources allocated to this data operation node.

5. The multi-source data element fusion and integration scheduling method based on data weaving according to claim 4, characterized in that: Each data operation node is executed sequentially, its actual execution status is recorded, and automatic retry is performed if a data operation node fails. Specific steps include: During scheduling, each data operation node is triggered sequentially. i Record the actual execution time T0(o) of each data operation node. i ) and state S(o i ), where state S(o i This includes success and failure; If state S(o i If the test fails, an automatic retry mechanism is initiated, with the number of retries denoted as Ry. The number of retries is determined when Ry ≤ R. max If the data operation node is re-executed, the task is marked as failed and an interrupt replay is performed. Where R... max This indicates the preset maximum number of retries.

6. The multi-source data element fusion and integration scheduling method based on data weaving according to claim 5, characterized in that: If all data operation nodes execute successfully, the merged data element result set will be output. The specific steps include: If all data operation nodes execute successfully, the fused data element result set will be output to the target location, which is specified by the target loading node in the data operation nodes, including a database or file, and the metadata lineage information will be updated, recording the data source, transformation operation and output path of this scheduling. If there are failures and retries are exhausted, an error report is returned to the task submitter, which lists the failed data operation nodes.

7. A multi-source data element fusion and integration scheduling system based on data weaving, applied to the multi-source data element fusion and integration scheduling method based on data weaving as described in any one of claims 1-6, characterized in that: The system includes: The system includes a metadata acquisition module, a task parsing module, a scheduling decision module, and an execution output module. The output of the metadata acquisition module is connected to the input of the task parsing module. The output of the task parsing module is connected to the input of the scheduling decision module. The output of the scheduling decision module is connected to the input of the execution output module.

8. The multi-source data element fusion and integration scheduling system based on data weaving according to claim 7, characterized in that: The metadata acquisition module is used to collect active metadata from multi-source heterogeneous data and construct a unified data element catalog. The task parsing module includes a parameter parsing unit, a positioning unit, and an indicator calculation unit; The parameter parsing unit is used to receive the fusion task, parse the required data element set, fusion logic, and maximum execution time; the positioning unit is used to traverse the directory and semantically match to locate the physical storage location of each data element; the index calculation unit is used to calculate the accessibility index of each data element based on the load rate and data timeliness.

9. The multi-source data element fusion and integration scheduling system based on data weaving according to claim 8, characterized in that: The scheduling decision module includes a threshold management unit, a comparison and judgment unit, and a plan generation unit; The threshold management unit is used to dynamically determine the accessibility threshold based on historical scheduling records; the comparison and judgment unit compares each indicator with the threshold to determine whether to generate a logical execution plan; the plan generation unit is used to generate a logical execution plan when the conditions are met, otherwise it returns a resource shortage prompt. The execution output module includes a node parsing unit, an execution scheduling unit, and a result output unit; The node parsing unit is used to parse the logical plan into a sequence of data operation nodes, specifying the operation type and binding elements; the execution scheduling unit is used to allocate resources in sequence, estimate the execution time, execute nodes in sequence and record the status, and automatically retry when it fails; the result output unit is used to output the fusion result set and update the lineage information when successful, and return an error report when it fails.

Images