A cloud integration data fusion method for multi-project collaboration

Abstract

The application discloses a kind of multi-project collaborative cloud integration data fusion methods, it is related to big data processing technical field, the method includes: obtaining the project data requirement of multiple target projects, and the target level corresponding to each project data is parsed and determined, combining characteristic dimension level rule and multiple project data requirements and associated target level, construct data fusion directed acyclic graph, with the output of multiple target projects as terminal node, solve the connected subgraph of all project data requirements and the minimum total fusion cost, as optimal fusion path, according to the topological order distribution of optimal fusion path each fusion operation is executed, obtains final fusion result.The technical problem that the existing data fusion method does not consider multi-project parallel scene, leading to cloud resource waste, high operating cost is solved.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, specifically to a cloud-based integrated data fusion method for multi-project collaboration. Background Technology

[0002] With the development of cloud computing and distributed collaborative technologies, scenarios where multiple teams and projects operate in parallel are becoming increasingly common. Different projects often have a large number of overlapping data requirements for the same original data sources, requiring repeated cleaning, feature extraction, and fusion calculation operations to be performed on the same original data.

[0003] However, repetitive fusion operations consume significant cloud computing and storage resources, leading to resource waste and increased overall operating costs. Furthermore, when different projects independently perform data fusion, it's difficult to share and reuse intermediate fusion results. Inconsistent big data processing standards can also cause discrepancies in fusion results, affecting data consistency across multiple projects and increasing collaboration costs. Existing data fusion methods are mostly designed for the needs of single projects, failing to consider the sharing and optimization of overlapping needs in parallel multi-project scenarios. Therefore, they are ill-suited to the low-cost, consistent data fusion requirements of multi-entity collaboration in the cloud. Summary of the Invention

[0004] This application provides a cloud-integrated data fusion method for multi-project collaboration, which solves the technical problem that existing data fusion methods do not consider multi-project parallel scenarios, resulting in wasted cloud resources and high operating costs.

[0005] The technical solution to the above-mentioned technical problems in this application is as follows: This application provides a cloud-based integrated data fusion method for multi-project collaboration, the method comprising: Obtain project data requirements from multiple target projects, and parse the project data requirements in conjunction with preset feature dimension hierarchy rules to determine the target hierarchy corresponding to each project data. Combining the aforementioned feature dimension hierarchy rules with the data requirements and associated target levels of multiple projects, a data fusion directed acyclic graph is constructed layer by layer from bottom to top. Using the outputs of multiple target projects as terminal nodes, the optimal fusion path is obtained by solving the connected subgraph that covers all project data requirements and has the minimum total fusion cost in the directed acyclic graph of the data fusion. Each fusion operation is executed in a distributed manner according to the topological order of the optimal fusion path to obtain the final fusion result.

[0006] Preferably, the project data requirements of multiple target projects are obtained, and the project data requirements are parsed in conjunction with preset feature dimension hierarchy rules to determine the target hierarchy corresponding to each project data, including: Receive project data requirement declarations submitted by multiple target projects, combine domain prior knowledge to perform feature tracing of the project data requirements of each target project, obtain the corresponding requirement features, and determine the original data sources corresponding to multiple requirement features; Map the required features of multiple target projects to a preset feature dimension level to determine the target level of each project's data; The feature dimension hierarchy consists of multiple feature levels arranged in ascending order of information abstraction, and the set of required features corresponding to each feature level constitutes a strictly nested superset chain.

[0007] Preferably, the project data requirements for each target project are traced by features to obtain corresponding requirement features, and the original data sources corresponding to multiple requirement features are determined, including: By combining prior domain knowledge, the project data requirement declaration for each target project is decomposed item by item to obtain the corresponding requirement features, and the set of original fields on which each requirement feature depends is determined. Based on the original field set, the original data source corresponding to each of the aforementioned requirement features is determined and used as the source data dependency set for the corresponding target project.

[0008] Preferably, combining the aforementioned feature dimension hierarchy rules with multiple project data requirements and associated target hierarchies, a data fusion directed acyclic graph is constructed layer by layer from bottom to top, including: Initialize the source node based on multiple original data sources; For each feature level, select the required features that are referenced by at least one higher feature level, and generate all feasible fusion operation types for each required feature to obtain a candidate fusion operation set; Calculate the fusion cost for each candidate fusion operation in the candidate fusion operation set; Based on the fusion cost, nodes with the same input feature set and overlapping output feature sets are identified. Pruning operations are performed, and the nodes are merged into multiple common intermediate fusion nodes that can be shared and reused by multiple target projects. The output feature set of the common intermediate fusion node is the union of the output feature sets of multiple merged nodes. Iteratively obtain the common intermediate fusion nodes after pruning operations at multiple feature levels and the original nodes corresponding to the required features, and construct the data fusion directed acyclic graph layer by layer from bottom to top.

[0009] Preferably, calculating the fusion cost of each candidate fusion operation in the candidate fusion operation set includes: The execution weight of each candidate fusion operation is determined, wherein the execution weight is composed of a weighted average of a data size component, a cross-source complexity component, and an operation type component; Determine the amount of feature change caused by each candidate fusion operation, wherein the amount of feature change is based on the rate of structural change between the input feature set and the output feature set before and after the fusion operation; The execution weight is multiplied by the feature variation to obtain the fusion cost for each candidate fusion operation.

[0010] Preferably, the process of iteratively obtaining common intermediate fusion nodes after pruning operations at multiple feature levels and the original nodes corresponding to the required features, and constructing the data fusion directed acyclic graph layer by layer from bottom to top, further includes: Define the formal structure of the data fusion directed acyclic graph, where graph nodes represent a requirement feature, identified by the input feature set and the data instance set as a pair, directed edges represent fusion operations, and edge weights are the fusion costs corresponding to the fusion operations; Based on the formal structure, and combining the common intermediate fusion nodes retained after pruning at each feature level, the original nodes corresponding to the required features, and the fusion operation, an internal subgraph corresponding to each feature level is established. Following the level of feature hierarchy, inter-layer directed edges are established between the output nodes of lower-level internal subgraphs and the input nodes of adjacent higher-level internal subgraphs, and the fusion cost of the corresponding fusion operation is used as the edge weight to obtain the data fusion directed acyclic graph. For each target project, the shallowest node that satisfies all the requirements of the target project is determined in the data fusion directed acyclic graph, and a connection edge is established from the shallowest node to the corresponding output sink node of the target project.

[0011] More preferably, using the outputs of multiple target projects as terminal nodes, the optimal fusion path is obtained by solving for the connected subgraph in the directed acyclic graph of the data fusion that covers all project data requirements and minimizes the total fusion cost, including: Based on the target level of the target projects from low to high, a greedy sharing identification is performed on multiple target projects to obtain the fusion sub-path that can be reused by multiple projects; Cost-aware path merging is performed on multiple fusion sub-paths to obtain the path combination with the minimum total cost, which is taken as the optimal fusion path. In the optimal fusion path, multiple target projects share a prefix in the fusion sub-path.

[0012] More preferably, multiple target projects are greedily shared and identified in ascending order of their target level to obtain fusion sub-paths that can be reused by multiple projects, including: For each target project, traverse the existing shared sub-paths corresponding to each common intermediate fusion node in the data fusion directed acyclic graph, and filter the existing shared sub-paths with the largest intersection coverage with the target project's demand feature set. If the intersection coverage exceeds a preset coverage threshold, the existing shared sub-path is reused, and the difference path between the output node of the existing shared sub-path and the uncovered requirement features of the target project is supplemented, and the fused sub-path is output. If the intersection coverage does not exceed the preset coverage threshold, a new fusion sub-path is established starting from the source node of the data fusion directed acyclic graph.

[0013] More preferably, cost-aware path merging is performed on multiple fusion sub-paths to obtain the path combination with the smaller total cost, thereby obtaining the optimal fusion path, including: The fused sub-paths are paired up, and for each pair of fused sub-paths, the nearest common ancestor node of the two is found in the data fusion directed acyclic graph. The nearest common ancestor node is the common predecessor node of the two fused sub-paths with the largest depth in the data fusion directed acyclic graph. Calculate the merge cost, wherein the merge cost includes the shared prefix portion cost from the source node to the nearest common ancestor node and the costs of the two branch paths from the nearest common ancestor node to the target project output nodes corresponding to the two merged sub-paths; Calculate the separation cost, where the separation cost is the sum of the costs of the two fused sub-paths running independently; Compare the merger cost with the separation cost: If the merging cost is less than the separation cost, then the merging operation is performed, merging the paths of the two paths before the nearest common ancestor node into a shared prefix, and establishing branches after the nearest common ancestor node respectively, and updating the data fusion directed acyclic graph; If the merge cost is greater than or equal to the separation cost, the merge is not performed.

[0014] This application provides one or more technical solutions, which have at least the following technical effects or advantages: This application provides a cloud-based integrated data fusion method for multi-project collaboration. First, it performs unified analysis and hierarchical positioning of the data requirements of multiple target projects. Combining preset feature dimension hierarchical rules, it constructs a directed acyclic graph (DAG) for data fusion from bottom to top. This method can clarify the overlapping relationships of requirements among multiple projects in advance, extracting the overlapping fusion operations of multiple projects into shareable common intermediate nodes, avoiding redundant fusion process construction. Second, by solving for the optimal fusion path with the goal of minimizing the total fusion cost, it maximizes the sharing of reusable fusion sub-paths while satisfying the data requirements of all projects, reducing redundant fusion calculations and storage overhead, and lowering the overall cloud operating cost. Finally, it uniformly executes distributed fusion operations based on the constructed optimal fusion path. All projects share a unified intermediate fusion result, ensuring the standard consistency of multi-project output data, reducing the cost of multi-project collaboration, and adapting to real-world scenarios of multi-entity parallel collaboration in the cloud.

[0015] Through the above technical solution, this application hierarchically sorts out the data requirements of multiple projects based on the feature dimension hierarchy, constructs a shareable and reusable data fusion graph structure, and solves the optimal fusion path with the goal of minimizing the total fusion cost, and finally realizes integrated fusion under multi-project collaboration, effectively reducing repeated fusion operations, reducing cloud resource consumption, ensuring the consistency of fused data among multiple projects, and reducing the cost of multi-project collaboration. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a cloud-based integrated data fusion method for multi-project collaboration provided in an embodiment of this application. Figure 2 This is a flowchart illustrating the calculation of fusion cost in a cloud-based integrated data fusion method for multi-project collaboration provided in an embodiment of this application. Figure 3 This is a flowchart illustrating the process of finding the optimal fusion path in a cloud-based integrated data fusion method for multi-project collaboration provided in this application embodiment. Detailed Implementation

[0018] This application provides a cloud-integrated data fusion method for multi-project collaboration, which addresses the technical problem that existing data fusion methods do not consider multi-project parallel scenarios, resulting in wasted cloud resources and high operating costs.

[0019] Example 1, as Figure 1 As shown in the figure, this application provides a cloud-based integrated data fusion method for multi-project collaboration, including: S10: Obtain the project data requirements of multiple target projects, and combine them with preset feature dimension hierarchy rules to parse the project data requirements and determine the target hierarchy corresponding to each project data. In this embodiment of the application, the project data requirements of the target project are first obtained, and then, in combination with the preset feature dimension hierarchy rules that are divided from low to high according to the level of information abstraction, the requirements of each project are analyzed and matched in terms of features to determine the target hierarchy corresponding to the data required by each project.

[0020] Specifically, step S10 in the method includes: Receive project data requirement declarations submitted by multiple target projects, combine domain prior knowledge to perform feature tracing of the project data requirements of each target project, obtain the corresponding requirement features, and determine the original data sources corresponding to multiple requirement features; Map the required features of multiple target projects to a preset feature dimension level to determine the target level of each project's data; The feature dimension hierarchy consists of multiple feature levels arranged in ascending order of information abstraction, and the set of required features corresponding to each feature level constitutes a strictly nested superset chain.

[0021] In this embodiment of the application, firstly, the project data requirement declarations submitted by multiple target projects are received. Then, the project data requirements of each target project are decomposed into features based on domain prior knowledge to obtain the corresponding requirement features and determine the set of original fields on which each requirement feature depends. Here, domain prior knowledge refers to the correspondence between features and original data sources that are pre-summarized based on business domain data processing experience. This correspondence is achieved through data annotation, and subsequent tracing can be completed based on the preset correspondence.

[0022] Specifically, we first collect data requirement documents, corresponding original data source samples, and data processing logs from multiple historical projects. From the above data, we extract the requirement description text and the set of original fields actually used in the past. We then label each sample and organize the correspondence of the labels into a domain prior knowledge base.

[0023] Furthermore, the annotation is based on the pattern matching regularity mapping rules: when the demand features contain keywords such as "total amount," "sum," and "aggregate," the corresponding numeric fields in the original data source are subjected to SUM aggregation; when they contain keywords such as "average," "mean value," and "average price," AVG aggregation is performed; when they contain keywords such as "quantity," "number," and "number of transactions," COUNT aggregation is performed; when they contain keywords such as "independent," "duplicate," and "different," COUNT(DISTINCT) is performed to count duplicates; when they contain keywords such as "maximum," "highest," and "most," MAX aggregation is performed; when they contain keywords such as "minimum," "lowest," and "least," MIN aggregation is performed; when they contain keywords such as "percentage," "ratio," and "ratio," the ratio of the conditional count to the total count is calculated; when they contain keywords such as "year-on-year," "month-on-month," and "compared to last month," the growth rate of (current value - same period value) / same period value is calculated; when they contain keywords such as "last N days," "this month," and "this quarter," time window filtering conditions are added.

[0024] The lower levels correspond to the original business details, such as basic user attributes and transaction history features; the middle levels correspond to single-dimensional statistical features, such as monthly transaction amount statistics; and the higher levels correspond to multi-dimensional fusion features, such as user risk scoring features. The set of high-level requirement features includes all lower-level requirement features. In specific annotation, each annotated sample corresponds to one annotation record. Each annotation record contains the requirement feature keywords, the unique identifier of the original data source that the feature directly depends on, and the corresponding original fields. One requirement feature corresponds to one or more annotation records.

[0025] For example, taking the "total sales" requirement feature as an example, when a new project submits a requirement statement for "total sales in a certain region in a certain month", the requirement feature "total sales" is decomposed, and the corresponding relationship in the domain prior knowledge base is used to locate the sales transaction log table in the original data source, and the corresponding original amount field is extracted as the original field set on which the requirement feature depends.

[0026] Secondly, all the identified demand features are mapped to pre-defined feature dimension levels to complete the hierarchical positioning of the data required for each target project. In this embodiment, the feature dimension levels are arranged from low to high according to the degree of information abstraction. The lower level corresponds to the original data level, and the higher level corresponds to the feature level derived after processing. The higher the level, the higher the degree of information abstraction. The demand feature sets of each level form a strictly nested superset chain, that is, all demand features of the higher level are derived from the processed features of the lower level.

[0027] For example, the feature dimension hierarchy is arranged from low to high according to the level of information abstraction, including: the original field layer, which is used to represent atomic fields directly extracted from the data source; the cleaning and alignment layer, which is used to represent features after format unification, noise reduction and type standardization; the association and fusion layer, which is used to represent composite features after cross-data source key-value association; the semantic aggregation layer, which is used to represent features derived from domain semantic level statistics; and the decision support layer, which is used to represent high-level indicator features for business decision-making.

[0028] Specifically, for each target project, the project data requirements are traced by features to obtain the corresponding requirement features, and the original data sources corresponding to multiple requirement features are determined, including: By combining prior domain knowledge, the project data requirement declaration for each target project is decomposed item by item to obtain the corresponding requirement features, and the set of original fields on which each requirement feature depends is determined. Based on the original field set, the original data source corresponding to each of the aforementioned requirement features is determined and used as the source data dependency set for the corresponding target project.

[0029] In this embodiment, firstly, based on prior domain knowledge, each requirement feature extracted from the project data requirement declaration is decomposed one by one to obtain the lowest-level original data field on which the feature depends, thus obtaining a complete set of original fields. Then, based on the data source identifier to which the original field set belongs, the original data source corresponding to each requirement feature is located by reverse tracing, and the complete source data dependency set of the corresponding target project is obtained, providing a basic dependency relationship for the subsequent layered construction of the fusion diagram.

[0030] In summary, the project data requirement declaration is first decomposed into features. Following the reverse decomposition path from the target requirement to the original data, the underlying original fields are obtained by tracing back level by level. Then, the corresponding original data source is determined based on the storage location of the original fields. All original data sources together form the source data dependency set corresponding to the target project.

[0031] Step S10 above can complete the analysis of the requirements characteristics and hierarchical positioning of multiple projects, clarify the basic dependencies for the subsequent layered construction of a shared and integrated structure, and avoid the construction of redundant integration nodes due to unclear requirements.

[0032] S20: Combining the feature dimension hierarchy rules and the data requirements and associated target levels of multiple projects, construct a data fusion directed acyclic graph from bottom to top. In this embodiment, starting from the lowest level of the feature dimension hierarchy, all levels are traversed layer by layer from bottom to top. For all features in each level that are dependent on the current project requirements, corresponding fusion nodes are created. Each fusion node generates dependency relationships based on its features and connects to all child nodes in the next lower level that generate the dependencies of that feature. Finally, all fusion nodes and connection edges together form a complete directed acyclic graph of data fusion. The bottom layer is the source node corresponding to the original data source, and the top layer is the output sink node corresponding to the requirements of each target project.

[0033] Specifically, step S20 in the method includes: Initialize the source node based on multiple original data sources; For each feature level, select the required features that are referenced by at least one higher feature level, and generate all feasible fusion operation types for each required feature to obtain a candidate fusion operation set; Calculate the fusion cost for each candidate fusion operation in the candidate fusion operation set; Based on the fusion cost, nodes with the same input feature set and overlapping output feature sets are identified. Pruning operations are performed, and the nodes are merged into multiple common intermediate fusion nodes that can be shared and reused by multiple target projects. The output feature set of the common intermediate fusion node is the union of the output feature sets of multiple merged nodes. Iteratively obtain the common intermediate fusion nodes after pruning operations at multiple feature levels and the original nodes corresponding to the required features, and construct the data fusion directed acyclic graph layer by layer from bottom to top.

[0034] In this embodiment of the application, firstly, based on multiple original data sources, corresponding source nodes are initialized. Each source node corresponds to an atomic field of an original data source. Specifically, the initialization of the source node is completed by reading the metadata information of the original data source to create the node. Each source node stores the data storage address and basic attribute description of the corresponding atomic field.

[0035] Secondly, all feature levels are traversed in ascending order. For the current level, all feature requirements that are referenced by at least one higher-level feature requirement or directly included in the target project requirements are selected to avoid generating redundant nodes that will not be used later. Then, for each selected feature requirement, all feasible fusion operation types that can generate the feature are generated based on the lower-level input features it depends on, and the corresponding candidate fusion operation set is obtained.

[0036] Then, the fusion cost corresponding to each candidate fusion operation is calculated, and this fusion cost is used to complete the optimal path selection with the goal of minimizing the total cost.

[0037] Subsequently, multiple candidate fusion nodes with the same input feature set are pruned and merged. If the output features of multiple nodes have an intersection, they are merged into a common intermediate fusion node that can be shared and reused by all corresponding target projects. After merging, the output feature set of this common node is the union of the output feature sets of all merged nodes. This avoids multiple projects generating multiple duplicate fusion nodes for the same input. The pruning operation refers to removing multiple independent duplicate fusion nodes and retaining only the merged common shared node, thereby reducing the generation of redundant nodes, reducing the size of the directed acyclic graph of data fusion, and simplifying the subsequent optimal path solution process.

[0038] Finally, starting from the lowest level source node, the processed common intermediate fusion nodes and the original nodes of the required features that do not need to be merged are added to the graph structure layer by layer upwards according to the feature dependencies, thus completing the construction of the bottom-up data fusion directed acyclic graph.

[0039] In summary, after processing all feature levels layer by layer in this manner, all the remaining source nodes, common intermediate fusion nodes, and required output nodes can be connected by edges according to their dependencies, thus completely constructing a data fusion directed acyclic graph containing shared nodes from bottom to top.

[0040] Furthermore, such as Figure 2 As shown, the fusion cost of each candidate fusion operation in the candidate fusion operation set is calculated, including: The execution weight of each candidate fusion operation is determined, wherein the execution weight is composed of a weighted average of a data size component, a cross-source complexity component, and an operation type component; Determine the amount of feature change caused by each candidate fusion operation, wherein the amount of feature change is based on the rate of structural change between the input feature set and the output feature set before and after the fusion operation; The execution weight is multiplied by the feature variation to obtain the fusion cost for each candidate fusion operation.

[0041] In this embodiment of the application, firstly, the execution weight of each candidate fusion operation is determined, which is obtained by weighted calculation of the data size component, the cross-source complexity component and the operation type component. The data size component is determined based on the logarithm of the number of data records involved in the fusion operation, the cross-source complexity component is determined based on the number of original data sources involved in the fusion operation, and the operation type component is determined based on the baseline cost of the operation type to which the fusion operation belongs.

[0042] Furthermore, the data size component corresponds to the total amount of input data that needs to be processed in this fusion operation; the larger the data size, the higher the component value. The cross-source complexity component corresponds to the number of different original data sources from which the input features come; the higher the degree of data source dispersion, the higher the component value. The operation type component corresponds to the computational complexity of the fusion operation itself; the more complex the computational logic, the higher the component value. By summing the three components according to the preset weight ratio, the execution weight of the candidate fusion operation can be obtained. For example, the preset weights are 0.4 for the data size component, 0.3 for the cross-source complexity component, and 0.3 for the operation type component. The weight ratios can be dynamically adjusted according to the actual cloud operation statistics. The preset weights can be adaptively adjusted according to the degree of influence of different components on the overall fusion cost during actual operation. If the resource overhead of cross-source access is higher in actual operation, the weight ratio of the cross-source complexity component can be appropriately increased.

[0043] Secondly, the differences in size and structure between the input and output feature sets of the candidate fusion operation are statistically analyzed to calculate the feature variation. The larger the feature variation, the higher the processing cost of the fusion operation. Specifically, it can be measured by the structural variation rate between the input and output feature sets before and after the fusion operation. For example, the structural variation rate is the ratio of the number of elements in the symmetric difference between the input and output feature sets to the sum of the number of elements in the input and output feature sets, i.e., structural variation rate = (|input feature set Δ output feature set|) / (|input feature set| + |output feature set|), where the numerator is the number of elements in the symmetric difference between the input and output feature sets, and the denominator is the sum of the number of elements in the input and output feature sets. The normalized structural variation rate is obtained in this way and used as the feature variation of this candidate fusion operation.

[0044] The number of elements in the input feature set is used to calculate the structural change rate, which is then directly used as the feature change amount.

[0045] Finally, by multiplying the execution weight by the feature change, the fusion cost of a single candidate fusion operation can be obtained. This cost can accurately reflect the resource consumption level of the fusion operation in the actual cloud operation, providing a reliable cost metric for subsequent optimal path solving.

[0046] For example, if the calculated execution weight is 0.56 and the input-output structure change rate, i.e. the feature change amount, is 0.8, the final fusion cost of this candidate fusion operation is 0.56 × 0.8 = 0.448.

[0047] Specifically, the process involves iteratively obtaining common intermediate fusion nodes from multiple feature levels after pruning operations, along with the original nodes corresponding to the required features, and constructing the data fusion directed acyclic graph layer by layer from bottom to top. This also includes: Define the formal structure of the data fusion directed acyclic graph, where graph nodes represent a requirement feature, identified by the input feature set and the data instance set as a pair, directed edges represent fusion operations, and edge weights are the fusion costs corresponding to the fusion operations; Based on the formal structure, and combining the common intermediate fusion nodes retained after pruning at each feature level, the original nodes corresponding to the required features, and the fusion operation, an internal subgraph corresponding to each feature level is established. Following the level of feature hierarchy, inter-layer directed edges are established between the output nodes of lower-level internal subgraphs and the input nodes of adjacent higher-level internal subgraphs, and the fusion cost of the corresponding fusion operation is used as the edge weight to obtain the data fusion directed acyclic graph. For each target project, the shallowest node that satisfies all the requirements of the target project is determined in the data fusion directed acyclic graph, and a connection edge is established from the shallowest node to the corresponding output sink node of the target project.

[0048] In this embodiment, firstly, a formal structure of a directed acyclic graph for data fusion is predefined, in which all graph nodes are identified by a tuple of "input feature set + data instance set", each node corresponds to a specific required feature, the directed edges between nodes correspond to the fusion operation required for feature generation, and the weight of the edge is directly set as the fusion cost corresponding to the fusion operation.

[0049] Furthermore, based on the defined formal structure, for each feature level, all nodes retained after pruning at this level are combined with the internal dependency fusion relationships between nodes to construct an internal subgraph corresponding to each feature level. In the internal subgraph, the nodes are the fusion operation nodes within that level, and the directed edges are the data flow dependencies between the fusion operations within that level.

[0050] Then, in order of feature hierarchy from low to high, dependency connections between adjacent levels are established sequentially. Inter-level directed edges are established between the output nodes of the lower-level internal subgraph and the input nodes of the higher-level internal subgraph that depend on the output nodes. The fusion cost of the corresponding fusion operation is set as the weight of the edge. After traversing all levels, the complete overall data fusion directed acyclic graph can be obtained.

[0051] Finally, for each target project, the entire directed acyclic graph of the data fusion is traversed to find the lowest-level node that satisfies all the requirements of the target project, i.e., the shallowest node. Connection edges are directly drawn from the shallowest node and connected to the output sink node corresponding to the target project to complete the construction of the entire graph.

[0052] Through the above steps S20, the construction of a directed acyclic graph for shared data fusion under multi-project collaboration can be completed. By pruning and merging common intermediate fusion nodes, the redundant construction of repeated fusion nodes is effectively reduced, and the overall size of the graph structure is reduced. At the same time, the operation path is pre-evaluated through fusion cost, which provides a quantitative basis for solving the optimal fusion path in the future. This not only realizes the sharing and reuse of common fusion nodes among multiple projects, but also avoids meaningless redundant computational resource consumption.

[0053] S30: Using the outputs of multiple target projects as terminal nodes, find the connected subgraph in the directed acyclic graph of the data fusion that covers all project data requirements and has the minimum total fusion cost, and use it as the optimal fusion path. In this embodiment, the output sink node of each target project is used as the terminal node. The entire directed acyclic graph of data fusion is traversed, all dependent node paths that meet the requirements of the project are traced back, the path coverage of all projects is integrated, the total fusion cost of all feasible connected subgraphs is calculated, and the connected subgraph with the minimum total fusion cost that can cover all data requirements of all target projects is selected. This connected subgraph is the optimal fusion path for the final data fusion.

[0054] Furthermore, since redundant nodes have been pruned and merged during the construction of the directed acyclic graph and intermediate fusion nodes are shared, the optimal fusion path obtained at this time can reuse existing fusion calculation results to the greatest extent, avoid repeating the same fusion operation between multiple projects, and minimize the total computing resource consumption in the cloud from the overall perspective.

[0055] Specifically, step S30 in the method includes: Based on the target level of the target projects from low to high, a greedy sharing identification is performed on multiple target projects to obtain the fusion sub-path that can be reused by multiple projects; Cost-aware path merging is performed on multiple fusion sub-paths to obtain the path combination with the minimum total cost, which is taken as the optimal fusion path. In the optimal fusion path, multiple target projects share a prefix in the fusion sub-path.

[0056] In this embodiment, the target projects are processed sequentially from low to high target levels. A greedy sharing identification is performed on each target project, prioritizing the identification of fusion sub-paths that can be directly reused in the existing graph structure. These fusion sub-paths have been included in the path range of other target projects and can directly meet some or all of the feature requirements of the current project. The shareable and reusable fusion sub-paths are directly retained without the need to regenerate new fusion paths.

[0057] Specifically, greedy sharing identification refers to starting from the target requirement node of the current target project and tracing the dependent nodes backward from top to bottom. When tracing back to an existing shared node that has already been included in the optimal path by a processed project, the downward tracing of that branch is stopped, and the fusion sub-path above the existing shared node is directly shared and reused. No further downward tracing is continued to generate new exclusive dependent paths. If the target projects are at the same level, they are sorted according to the total amount of data requirement of the target projects from smallest to largest and processed in sequence. This ensures that projects with less demand complete the sharing identification first, avoiding the situation where large demand projects occupy a large number of nodes, leaving small demand projects unable to split and reuse enough shared paths, resulting in a high overall cost. After all branches have been traced, all reusable fusion sub-paths of the current target project are obtained.

[0058] Secondly, after completing the greedy sharing identification of a single project, all identified reusable fusion sub-paths are integrated. Based on the goal of minimizing the total fusion cost, paths with overlapping coverage are merged and adjusted to eliminate duplicate path segments. Finally, the path combination that can cover all data requirements of all target projects and minimize the total fusion cost is obtained. This combination is the final optimal fusion path. In the optimal fusion path, multiple target projects will share fusion sub-paths with the same prefix, maximizing the reuse of fusion computing under multi-project collaboration.

[0059] Specifically, based on the target project's target level from low to high, a greedy sharing identification is performed on multiple target projects to obtain fusion sub-paths that can be reused by multiple projects, including: For each target project, traverse the existing shared sub-paths corresponding to each common intermediate fusion node in the data fusion directed acyclic graph, and filter the existing shared sub-paths with the largest intersection coverage with the target project's demand feature set. If the intersection coverage exceeds a preset coverage threshold, the existing shared sub-path is reused, and the difference path between the output node of the existing shared sub-path and the uncovered requirement features of the target project is supplemented, and the fused sub-path is output. If the intersection coverage does not exceed the preset coverage threshold, a new fusion sub-path is established starting from the source node of the data fusion directed acyclic graph.

[0060] In this embodiment of the application, firstly, for each target project, the current shared sub-paths corresponding to all common intermediate fusion nodes that have been identified by other projects in the directed acyclic graph of data fusion are traversed, and the intersection coverage rate of the features contained in the current shared sub-paths and the current target project's requirement feature set is calculated one by one, and the current shared sub-path with the largest intersection coverage rate is selected.

[0061] For example, the intersection coverage rate = the number of elements in the intersection of the current target project requirement feature set and the existing shared sub-path feature set ÷ the total number of elements in the current target project requirement feature set. This can be used to measure the degree to which the shared sub-path meets the current project requirements.

[0062] Secondly, the maximum intersection coverage obtained by filtering is compared with the preset coverage threshold. The preset coverage threshold is an experience value set by the developers based on actual application scenarios and is dynamically adjusted according to the actual requirements for reusability. For example, the value range can be [0.6, 0.9].

[0063] Specifically, if the intersection coverage exceeds the preset coverage threshold, it means that the shared sub-path has covered most of the requirements of the current project. The existing shared sub-path can be reused directly. Only an additional difference path needs to be added from the output node of the shared sub-path to the uncovered requirements of the target project. Finally, it is integrated with the shared sub-path to obtain the fused sub-path corresponding to the current project.

[0064] If the intersection coverage does not exceed the preset coverage threshold, it means that the existing shared sub-path has too low a coverage of the current project's needs and has little reuse value. In this case, you can start directly from the bottom source node of the data fusion directed acyclic graph and build the fusion sub-path exclusive to the current target project from scratch.

[0065] Furthermore, cost-aware path merging is performed on multiple fusion sub-paths to obtain the path combination with the smaller total cost, thereby obtaining the optimal fusion path, including: The fused sub-paths are paired up, and for each pair of fused sub-paths, the nearest common ancestor node of the two is found in the data fusion directed acyclic graph. The nearest common ancestor node is the common predecessor node of the two fused sub-paths with the largest depth in the data fusion directed acyclic graph. Calculate the merge cost, wherein the merge cost includes the shared prefix portion cost from the source node to the nearest common ancestor node and the costs of the two branch paths from the nearest common ancestor node to the target project output nodes corresponding to the two merged sub-paths; Calculate the separation cost, where the separation cost is the sum of the costs of the two fused sub-paths running independently; Compare the merger cost with the separation cost: If the merging cost is less than the separation cost, then the merging operation is performed, merging the paths of the two paths before the nearest common ancestor node into a shared prefix, and establishing branches after the nearest common ancestor node respectively, and updating the data fusion directed acyclic graph; If the merge cost is greater than or equal to the separation cost, the merge is not performed.

[0066] In the embodiments of this application, such as Figure 3 As shown, all identified fusion sub-paths are paired up. For each pair of fusion sub-paths, the nearest common ancestor node of the two paths is first found in the entire directed acyclic graph of data fusion. This node is the one with the largest depth among all common predecessor nodes of the two fusion sub-paths, that is, the common node closest to the two target output nodes.

[0067] Secondly, calculate the total cost of the two schemes for the pair of fusion sub-paths, namely the total fusion cost of the merging scheme and the total separation cost of the separation scheme. The total merging cost is the total fusion cost of the shared prefix part from the source node of the graph to the nearest common ancestor node, plus the total fusion cost of the two branch paths from the nearest common ancestor node to the output nodes of the two target projects. The total merging cost of the separation scheme is obtained by simply adding the independent total fusion costs of the two fusion sub-paths.

[0068] Then, the merge cost and the separation cost are compared. If the merge cost is less than the separation cost, it means that merging shared prefix paths can reduce the total resource consumption. In this case, the merge operation is performed, merging the paths before the nearest common ancestor node into a shared prefix path, retaining only the two independent branches after the nearest common ancestor node, and updating the entire path structure. If the merge cost is not less than the separation cost, it means that merging cannot reduce the total consumption, and the two paths are kept independent. After traversing all pairwise fused sub-paths, all paths that can reduce the total cost are merged. The final path combination is the optimal fused path with the minimum total fusion cost.

[0069] For example, if the merging sub-paths of two target projects start from the source node and pass through three common predecessor nodes with depths of 1, 2, and 3, the common predecessor node with depth 3 is closest to the two target output nodes, which is the nearest common ancestor node of the two merging sub-paths. The total cost of the shared prefix from the source node to this nearest common ancestor node is calculated to be 0.32, the branch cost from this node to the first target output node is 0.16, and the branch cost to the second target output node is 0.18, for a total merging cost of 0.32 + 0.16 + 0.18 = 0.66. However, when the two paths run independently, the total cost of the first merging sub-path is 0.48, the total cost of the second is 0.52, and the total separation cost is 0.48 + 0.52 = 1.0. In this case, the merging cost of 0.66 is less than the separation cost of 1.0, so the merging operation is performed, sharing the path before the nearest common ancestor node and only retaining the independent branches after the node.

[0070] The above step S30 can complete the optimal fusion path planning under multi-project collaboration. Under the premise of ensuring that all project data requirements are met, the common fusion nodes and fusion sub-paths are shared to the greatest extent, avoiding repeated execution of the same fusion operation, minimizing the total computing resource consumption of cloud data fusion from the global level, and realizing the efficient utilization of cloud resources under multi-project collaboration.

[0071] S40: Execute each fusion operation in a distributed manner according to the topological order of the optimal fusion path to obtain the final fusion result.

[0072] In this embodiment, according to the topology sorting obtained by the optimal fusion path, each fusion operation is distributed to multiple distributed computing nodes in the cloud for parallel execution. Each fusion operation is calculated only once on a shared intermediate node. After the calculation is completed, the result is cached in the cloud storage space for all subsequent fusion operations that depend on the output of this node to read and call directly. It is not necessary for each project to repeat the same calculation process. After all fusion operations are completed, the final fusion result required by the project can be obtained from the output sink node corresponding to each target project.

[0073] By using the above step S40, it can be ensured that each fusion operation is performed only once, and the calculation results are shared with all projects that depend on the output of this node. This avoids redundant calculations and makes full use of the parallel computing capabilities of multiple nodes based on the cloud-based distributed architecture.

[0074] In summary, compared with existing technologies, this application constructs a unified shared directed acyclic graph for data fusion by pre-pruning and merging common intermediate fusion nodes among multiple projects. Then, through greedy sharing identification and cost-aware path merging, it solves for the optimal fusion path with the minimum total fusion cost. Finally, it executes the fusion operation in a distributed manner according to the path topology. This achieves the sharing and reuse of common fusion results under multi-project collaboration at the global level, solving the problems of high redundancy of repeated calculations and serious waste of cloud resources in traditional multi-project data fusion. In cloud fusion scenarios with multiple projects running concurrently, it can effectively reduce the total computing resource consumption and improve the execution efficiency of cloud data fusion.

[0075] In summary, the embodiments of this application have at least the following technical effects: This application provides a cloud-based integrated data fusion method for multi-project collaboration. First, it performs unified analysis and hierarchical positioning of the data requirements of multiple target projects. Combining preset feature dimension hierarchical rules, it constructs a directed acyclic graph (DAG) for data fusion from bottom to top. This method can clarify the overlapping relationships of requirements among multiple projects in advance, extracting the overlapping fusion operations of multiple projects into shareable common intermediate nodes, avoiding redundant fusion process construction. Second, by solving for the optimal fusion path with the goal of minimizing the total fusion cost, it maximizes the sharing of reusable fusion sub-paths while satisfying the data requirements of all projects, reducing redundant fusion calculations and storage overhead, and lowering the overall cloud operating cost. Finally, it uniformly executes distributed fusion operations based on the constructed optimal fusion path. All projects share a unified intermediate fusion result, ensuring the standard consistency of multi-project output data, reducing the cost of multi-project collaboration, and adapting to real-world scenarios of multi-entity parallel collaboration in the cloud.

[0076] Through the above technical solution, this application hierarchically sorts out the data requirements of multiple projects based on the feature dimension hierarchy, constructs a shareable and reusable data fusion graph structure, and solves the optimal fusion path with the goal of minimizing the total fusion cost, and finally realizes integrated fusion under multi-project collaboration, effectively reducing repeated fusion operations, reducing cloud resource consumption, ensuring the consistency of fused data among multiple projects, and reducing the cost of multi-project collaboration.

[0077] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0078] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cloud-based integrated data fusion method for multi-project collaboration, characterized in that, include: Obtain project data requirements from multiple target projects, and parse the project data requirements in conjunction with preset feature dimension hierarchy rules to determine the target hierarchy corresponding to each project data. Combining the aforementioned feature dimension hierarchy rules with the data requirements and associated target levels of multiple projects, a data fusion directed acyclic graph is constructed layer by layer from bottom to top. Using the outputs of multiple target projects as terminal nodes, the optimal fusion path is obtained by solving the connected subgraph that covers all project data requirements and has the minimum total fusion cost in the directed acyclic graph of the data fusion. The fusion operations are executed in a distributed manner according to the topological order of the optimal fusion path to obtain the final fusion result.

2. The cloud-based integrated data fusion method for multi-project collaboration as described in claim 1, characterized in that, Obtain project data requirements from multiple target projects, and parse these requirements using pre-defined feature dimension hierarchy rules to determine the target hierarchy corresponding to each project data point, including: Receive project data requirement declarations submitted by multiple target projects, combine domain prior knowledge to perform feature tracing of the project data requirements of each target project, obtain the corresponding requirement features, and determine the original data sources corresponding to multiple requirement features; Map the required features of multiple target projects to a preset feature dimension level to determine the target level of each project's data; The feature dimension hierarchy consists of multiple feature levels arranged in ascending order of information abstraction, and the set of required features corresponding to each feature level constitutes a strictly nested superset chain.

3. The cloud-based integrated data fusion method for multi-project collaboration as described in claim 2, characterized in that, For each target project, the project data requirements are traced by features to obtain the corresponding requirement features, and the original data sources corresponding to multiple requirement features are determined, including: By combining prior domain knowledge, the project data requirement declaration for each target project is decomposed item by item to obtain the corresponding requirement features, and the set of original fields on which each requirement feature depends is determined. Based on the original field set, the original data source corresponding to each of the aforementioned requirement features is determined and used as the source data dependency set for the corresponding target project.

4. The cloud-based integrated data fusion method for multi-project collaboration as described in claim 1, characterized in that, Combining the aforementioned feature dimension hierarchy rules, and integrating multiple project data requirements with associated target hierarchies, a data fusion directed acyclic graph is constructed layer by layer from bottom to top, including: Initialize the source node based on multiple original data sources; For each feature level, select the required features that are referenced by at least one higher feature level, and generate all feasible fusion operation types for each required feature to obtain a candidate fusion operation set; Calculate the fusion cost for each candidate fusion operation in the candidate fusion operation set; Based on the fusion cost, nodes with the same input feature set and overlapping output feature sets are identified. Pruning operations are performed, and the nodes are merged into multiple common intermediate fusion nodes that can be shared and reused by multiple target projects. The output feature set of the common intermediate fusion node is the union of the output feature sets of multiple merged nodes. Iteratively obtain the common intermediate fusion nodes after pruning operations at multiple feature levels and the original nodes corresponding to the required features, and construct the data fusion directed acyclic graph layer by layer from bottom to top.

5. The cloud-based integrated data fusion method for multi-project collaboration as described in claim 4, characterized in that, Calculating the fusion cost for each candidate fusion operation in the candidate fusion operation set includes: The execution weight of each candidate fusion operation is determined, wherein the execution weight is composed of a weighted average of a data size component, a cross-source complexity component, and an operation type component; Determine the amount of feature change caused by each candidate fusion operation, wherein the amount of feature change is based on the rate of structural change between the input feature set and the output feature set before and after the fusion operation; The execution weight is multiplied by the feature variation to obtain the fusion cost for each candidate fusion operation.

6. The cloud-based integrated data fusion method for multi-project collaboration as described in claim 4, characterized in that, Iteratively obtaining common intermediate fusion nodes after pruning multiple feature levels and the original nodes corresponding to the required features, and constructing the data fusion directed acyclic graph layer by layer from bottom to top, also includes: Define the formal structure of the data fusion directed acyclic graph, where graph nodes represent a requirement feature, identified by the input feature set and the data instance set as a pair, directed edges represent fusion operations, and edge weights are the fusion costs corresponding to the fusion operations; Based on the formal structure, and combining the common intermediate fusion nodes retained after pruning at each feature level, the original nodes corresponding to the required features, and the fusion operation, an internal subgraph corresponding to each feature level is established. Following the level of feature hierarchy, inter-layer directed edges are established between the output nodes of lower-level internal subgraphs and the input nodes of adjacent higher-level internal subgraphs, and the fusion cost of the corresponding fusion operation is used as the edge weight to obtain the data fusion directed acyclic graph. For each target project, the shallowest node that satisfies all the requirements of the target project is determined in the data fusion directed acyclic graph, and a connection edge is established from the shallowest node to the corresponding output sink node of the target project.

7. The cloud-based integrated data fusion method for multi-project collaboration as described in claim 1, characterized in that, Using the outputs of multiple target projects as terminal nodes, the optimal fusion path is determined by finding the connected subgraph in the directed acyclic graph that covers all project data requirements and minimizes the total fusion cost. This path includes: Based on the target level of the target projects from low to high, a greedy sharing identification is performed on multiple target projects to obtain the fusion sub-path that can be reused by multiple projects; Cost-aware path merging is performed on multiple fusion sub-paths to obtain the path combination with the minimum total cost, which is taken as the optimal fusion path. In the optimal fusion path, multiple target projects share a prefix in the fusion sub-path.

8. The cloud-based integrated data fusion method for multi-project collaboration as described in claim 7, characterized in that, Based on the target project's target level from low to high, a greedy sharing identification is performed on multiple target projects to obtain fusion sub-paths that can be reused by multiple projects, including: For each target project, traverse the existing shared sub-paths corresponding to each common intermediate fusion node in the data fusion directed acyclic graph, and filter the existing shared sub-paths with the largest intersection coverage with the target project's demand feature set. If the intersection coverage exceeds a preset coverage threshold, the existing shared sub-path is reused, and the difference path between the output node of the existing shared sub-path and the uncovered requirement features of the target project is supplemented, and the fused sub-path is output. If the intersection coverage does not exceed the preset coverage threshold, a new fusion sub-path is established starting from the source node of the data fusion directed acyclic graph.

9. The cloud-based integrated data fusion method for multi-project collaboration as described in claim 7, characterized in that, The optimal fusion path is obtained by performing cost-aware path merging on multiple fusion sub-paths to obtain the path combination with the smaller total cost, including: The fused sub-paths are paired up, and for each pair of fused sub-paths, the nearest common ancestor node of the two is found in the data fusion directed acyclic graph. The nearest common ancestor node is the common predecessor node of the two fused sub-paths with the largest depth in the data fusion directed acyclic graph. Calculate the merge cost, wherein the merge cost includes the shared prefix portion cost from the source node to the nearest common ancestor node and the costs of the two branch paths from the nearest common ancestor node to the target project output nodes corresponding to the two merged sub-paths; Calculate the separation cost, where the separation cost is the sum of the costs of the two fused sub-paths running independently; Compare the merger cost with the separation cost: If the merging cost is less than the separation cost, then the merging operation is performed, merging the paths of the two paths before the nearest common ancestor node into a shared prefix, and establishing branches after the nearest common ancestor node respectively, and updating the data fusion directed acyclic graph; If the merge cost is greater than or equal to the separation cost, the merge is not performed.

Images