Multi-cloud data analysis method and system based on cloud platform
By constructing a unified metadata catalog and interactive topology map, combined with a four-dimensional evaluation matrix and hierarchical task processing, the problems of data silos and privacy compliance in multi-cloud environments are solved, and efficient and secure cross-cloud data analysis is achieved.
Patent Information
- Application Number
- CN202511060164.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-11-07
AI Technical Summary
Existing technologies in multi-cloud environments suffer from problems such as data silos, fragmented management, fragmented metadata, high privacy compliance risks, increased task execution costs, and difficulty in meeting privacy requirements.
By constructing a unified metadata catalog and interactive topology graph, global governance of multi-cloud data is achieved; a four-dimensional evaluation matrix is used to dynamically select computing nodes, and sensitive and non-sensitive tasks are processed in a hierarchical manner. Combined with federated learning and differential privacy protection, cross-cloud computing and privacy security are achieved.
It enables intelligent governance of multi-cloud data, reduces task execution costs and privacy compliance risks, improves resource utilization and privacy compliance coverage, and meets real-time analysis and privacy security requirements.
Smart Images
Figure CN120910140A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of data processing, and particularly relates to a multi-cloud data analysis method and system based on a cloud platform. BACKGROUND
[0002] With the acceleration of enterprise digital transformation, multi-cloud architecture has become the mainstream choice; however, the problem of data islandization and management fragmentation in hybrid cloud environment is highlighted: heterogeneous storage systems have format differences, and cross-cloud data flow needs to be manually converted; compliance requirements such as GDPR and HIPAA increase the sensitivity of data geographical location, and traditional solutions lack a global compliance view. At the same time, new loads such as AI training and real-time analysis require higher cross-cloud collaboration, and edge computing scenarios need to consider low latency and privacy security.
[0003] In the prior art, cloud-native API interoperability is poor, and there is a lack of automatic field mapping mechanism, which requires manual maintenance of data dictionary, causing cross-cloud query semantic ambiguity; traditional solutions select computing nodes based on static strategies, ignoring real-time network delay and dynamic load, resulting in increased task execution cost; sensitive data processing lacks fine-grained control, federated learning and conventional ETL tasks are executed together, and the differential privacy injection timing is lagging, which cannot meet the privacy requirements, and the audit traceability coverage is insufficient.
[0004] In the prior art, there are problems of meta-data fragmentation, rigid scheduling, and privacy compliance risks. SUMMARY
[0005] (I) Technical problems solved In view of the problems in the related art, the present application provides a multi-cloud data analysis method based on a cloud platform to overcome the above technical problems existing in the prior art.
[0006] (II) Technical solutions To solve the above technical problems, the present application is realized by the following technical solutions: S1, configure a multi-cloud environment and initialize to obtain a structured meta-data list and a heterogeneous data mode mapping table; S2, based on the structured meta-data list and the heterogeneous data mode mapping table, construct a unified meta-data directory and an interactive topology graph; S3, collect real-time multi-modal request data; based on the unified meta-data directory and the interactive topology graph, perform semantic analysis, blood relation analysis processing on the real-time multi-modal request data to obtain a standardized task descriptor; S4, based on the standardized task descriptor, evaluate the network, cost, compliance and load of the cloud nodes of the cloud platform to obtain an optimal cloud node; S5, input the sensitive task in the standardized task descriptor to the optimal cloud node, and obtain an encrypted gradient update package through federated learning fragmentation; field conversion, calculation of encrypted aggregation processing are performed on the non-sensitive task in the standardized task descriptor, and an encrypted intermediate calculation result set is obtained; S6, the encrypted gradient update package and the encrypted intermediate calculation result set are aggregated, calculated and differentially private protection processed, and a desensitization data set meeting differential privacy is obtained; S7, after S6 is ended, the completed task is destroyed, and the cloud platform resource is released; The application realizes global governance of multi-cloud data by automatically constructing a unified metadata directory, and combines interactive topology graph labeling with compliance labels; task descriptors are generated based on multi-modal request analysis, and a four-dimensional evaluation matrix is innovatively used for dynamic optimization of computing nodes; sensitive tasks and non-sensitive tasks are processed in stages, noise is injected after fusion of the results to realize mathematically verifiable differential privacy protection, and resources are automatically recycled; the multi-cloud data island problem is solved, the metadata alignment accuracy is improved, the cost and over-budget risk are reduced through dynamic scheduling, and the privacy compliance coverage and resource utilization are improved.
[0007] Preferably, the S1 comprises the following steps: S11, configure a network security policy, input a cloud platform account credential, configure cross-account access permission through a cloud service provider console, and use an OpenSSL tool to establish a TLS 1.3 encrypted channel; set a role-based access control permission matrix to obtain an encrypted authentication credential library and a secure connection endpoint list; S12, set a target cloud node from the secure connection endpoint list to obtain a target cloud node set; deploy a lightweight collection agent on the target cloud node in the target cloud node set, and automatically scan the storage structure of the encrypted authentication credential library through a cloud-native API to obtain a structured metadata list and a heterogeneous data mode mapping table; The application establishes a security baseline through a TLS 1.3 encrypted channel and an RBAC permission matrix, deploys a lightweight agent to automatically scan multi-cloud heterogeneous storage, and generates a structured metadata list containing an encrypted state and format conversion rule; cross-cloud authentication credential zero-contact management is realized, the metadata collection efficiency is improved, and the risk of manual configuration errors is eliminated.
[0008] Preferably, the S2 comprises the following steps: S21, columnar storage structures in the structured metadata list are parsed, separators and encoding formats are detected, and nested fields are extracted using an Apache Parquet tool, a CSV Sniffer and a JSONPath processor respectively to obtain parsed structured metadata; S22, based on the heterogeneous data mode mapping table and the field meaning of the parsed structured metadata, type conversion rules are established by semantic similarity algorithm matching, combined with the target data type, and special fields in the structured metadata list are marked, to obtain a cross-cloud field mapping dictionary; S23, based on the cross-cloud field mapping dictionary and cloud platform infrastructure information, a unified SQL interface is created using virtualization technology, encapsulating the differences in underlying storage, and transparent data access is realized, to obtain a unified metadata directory view; An interactive data topology map is obtained by determining the physical location of data through IP geolocation, labeling compliance tags based on storage area, and integrating cloud service provider compliance certification status. The application automatically parses multi-source data structure through intelligent semantic alignment technology, generates a cross-cloud field mapping dictionary, encapsulates heterogeneous storage into a unified SQL interface based on virtualization technology, realizes transparent data access, and forms an interactive topology map by labeling compliance tags combined with IP geolocation. Eliminate manual mapping errors, improve query efficiency, and reduce compliance audit time.
[0009] Preferably, the S3 comprises the following steps; S31, collect SQL query statements, machine learning task descriptions, and stream processing requests to obtain real-time multi-modal request data; S32, receive real-time multi-modal request data through a RESTful API gateway, and use a protocol converter to standardize the input format of real-time multi-modal request data to obtain standardized real-time multi-modal request data; The physical location of the standardized real-time multi-modal request data is obtained by querying the unified metadata directory through the logical table name, and the field mapping rule is loaded to obtain the request data with physical location markers; S33, parse the cross-cloud association in the SQL statement in the request data with physical location markers, identify the data dependency of the machine learning task in the standardized real-time multi-modal request data, extract the window parameters of the stream processing task in the standardized real-time multi-modal request data, and obtain structured task metadata; The risk report is generated by binding the compliance level of the unified metadata directory, the data flow path of the structured task metadata is labeled on the interactive data topology map, and the task dependency is constructed to obtain the data lineage analysis result; S34, synthesize the structured task metadata and data lineage analysis result to obtain a standardized task descriptor; The application receives a multi-modal request through a unified API gateway, automatically binds a physical position and a field mapping rule, parses cross-cloud association and data dependency, generates a risk report in combination with a compliance label, dynamically marks a data blood relationship path on a topology graph and outputs a standardized task descriptor, improves multi-source request processing efficiency and task scheduling accuracy, and reduces compliance audit time consumption through data flow path visualization.
[0010] Preferably, the S4 comprises the following steps: S41, a cross-cloud delay data threshold is set, cross-cloud delay data is obtained according to a target position in the standardized task descriptor in combination with an interactive data topology graph, cloud nodes with cross-cloud delay data greater than the cross-cloud delay data threshold are removed as candidate cloud nodes, a network score matrix is constructed based on the cross-cloud delay data of the candidate cloud nodes, and the network score matrix comprises the candidate cloud nodes, delay of the candidate cloud nodes and network scores; S42, a cloud platform price API is collected, the cost of the candidate cloud nodes in the network score matrix is calculated in combination with the standardized task descriptor, a cost score matrix is obtained, the cost score matrix comprises the candidate cloud nodes, cloud node unit price, estimated cost and cost score; S43, the candidate cloud nodes are scored based on compliance labels of the unified metadata directory, and a compliance score matrix is obtained; S44, load data of the candidate nodes is obtained based on the interactive data topology graph, and the candidate nodes are scored based on the load data, and a load score matrix is obtained; S45, the network score matrix, the cost score matrix, the compliance score matrix and the load score matrix are fused, and a four-dimensional evaluation matrix is obtained, and the four-dimensional score matrix comprises network scores, cost scores, compliance scores and load scores of each candidate cloud node; S46, weights of each score in the four-dimensional score matrix are set, a weight set is obtained, a comprehensive score of each candidate cloud node is calculated based on the four-dimensional evaluation matrix in combination with the weight set, a final score set is obtained, a cloud node corresponding to the highest score in the final score set is selected, and an optimal cloud node is obtained; The application builds a four-dimensional dynamic evaluation model through innovation, filters high-delay nodes through a delay threshold, calculates cost scores by integrating cloud pricing APIs, generates compliance scores by binding compliance labels, generates load scores by real-time monitoring of node loads, and outputs an optimal node after weighted fusion, thereby improving task scheduling accuracy and response speed of delay-sensitive tasks and reducing comprehensive cost and over-budget risks.
[0011] Preferably, the S5 comprises the following steps: S51, input sensitive tasks in the standardized task descriptor to the optimal cloud node, adopt federated learning fragmentation for the sensitive tasks, perform local training at each edge node, only return encrypted gradient parameters, and obtain an encrypted gradient update package; S52, transmit data of non-sensitive tasks in the standardized task descriptor through a special channel, perform field conversion using a field mapping rule, and perform format standardization to obtain a converted intermediate data set; S53, split the converted intermediate data set according to the MapReduce principle to obtain a task fragment set, select optimal computing cloud nodes from an interactive data topology graph to obtain a computing cloud node set, and distribute the fragments in the task fragment set to the cloud nodes in the computing cloud node set; The cloud nodes in the computing cloud node set perform calculation on the fragments in the task fragment set, and perform transparent encryption and cross-cloud aggregation to obtain an encrypted intermediate calculation result set; The present application realizes task hierarchical processing mechanism; sensitive tasks are trained in situ at edge nodes through federated learning fragmentation, and only encrypted gradients are returned; after field conversion, non-sensitive tasks are dispatched to optimal nodes for homomorphic encryption aggregation based on MapReduce fragmentation; the risk of sensitive data leakage is reduced to zero, the processing speed of non-sensitive tasks is improved, the cross-cloud computing cost is reduced, and the dual privacy requirement is met.
[0012] Preferably, the federated learning in S51 is realized by the following steps: S511, collect historical execution index data; the historical execution index data includes task execution logs, cloud platform billing API, and precision evaluation reports; Extract the key indicators of the historical execution index data, including actual delay vs. estimated delay, actual cost vs. estimated cost, and result precision, to obtain a historical task execution index triple; S512, based on the historical task execution index triple, use reinforcement learning combined with a multi-objective genetic algorithm to optimize and update the reward function weight coefficient to obtain an updated scheduling strategy; The present application dynamically optimizes the federated learning scheduling strategy based on the historical task index triple using reinforcement learning and a multi-objective genetic algorithm; reduces the model training delay prediction error and precision fluctuation rate, and saves resource cost.
[0013] Preferably, S6 includes the following steps: S61, use the Paillier homomorphic encryption algorithm to aggregate the encrypted gradient update package and the encrypted intermediate calculation result set to obtain an encrypted global intermediate calculation result set; Perform average operation on the encrypted global intermediate calculation result set to obtain an aggregated encrypted result set; S62, decrypt the aggregated encrypted result set to obtain a decrypted aggregated result, decrypt the aggregated result injected with Laplace noise, and verify differential privacy to obtain a desensitized data set satisfying differential privacy; S63, labeling data sources in the desensitized data set satisfying differential privacy, and generating a multi-dimensional interactive chart with additional cost consumption analysis to obtain an intelligent analysis report with verifiable traceability; The present application fuses encrypted gradients and intermediate results through a Paillier algorithm, realizes differential privacy protection by injecting Laplace noise, labels data sources, and generates a multi-dimensional traceability report, thereby achieving mathematically verifiable privacy security, improving audit efficiency, and reducing violation risks.
[0014] Preferably, the S7 comprises the following steps: S71, identifying a temporary instance of a completed task, calling a cloud platform API to immediately destroy the instance, and obtaining a resource release confirmation report; S72, setting an idle time threshold; detecting that the idle time is greater than the idle time threshold, executing a graceful shutdown to release the connection, and obtaining a database connection state log; The present application automatically identifies and destroys completed task instances, and monitors idle connection timeout thresholds to perform graceful shutdown, thereby reducing resource release delays and improving connection pool utilization.
[0015] A multi-cloud data analysis system based on a cloud platform is used to implement the above-mentioned multi-cloud data analysis method based on a cloud platform, and comprises a multi-cloud environment initialization and metadata collection module, a unified metadata directory and topology construction module, a multi-modal request analysis and blood relationship analysis module, a dynamic scheduling optimization decision module, a secure task dispatching and federated computing module, a privacy protection aggregation and traceability module, and a resource recycling and state monitoring module. The multi-cloud environment initialization and metadata collection module is used to establish a secure cross-cloud connection and extract heterogeneous data metadata; by configuring network security strategies and automatic deployment tools, credential management and node access on multiple platforms are realized; a lightweight agent is deployed on the target node to scan the storage structure, and a structured metadata list containing storage types, encryption states and format conversion rules and a heterogeneous data mode mapping table are generated; The unified metadata directory and topology construction module is used to analyze raw metadata and construct a global view; data modes are analyzed by using Parquet tools, CSV Sniffer and other technologies, and a cross-cloud field mapping dictionary is established by using a semantic similarity algorithm; based on this, a virtualized SQL interface is created, physical storage differences are encapsulated, a unified metadata directory is formed, and a logical table and physical location mapping relationship is labeled; by combining IP geolocation and compliance authentication, an interactive data topology graph is generated to realize visualization of data location and compliance; The multi-modal request analysis and blood relation analysis module is used for processing diversified computing requests and tracking data blood relations; SQL queries, machine learning tasks and stream processing requests are received through an API gateway, and after standardization, physical location and field mapping rules are obtained by association with a UMC; task dependency relationships are analyzed, data flow paths are marked on a topology graph, and risk reports and data blood relation analysis results are generated in combination with compliance labels, and finally, a standardized task descriptor containing complete context is output; The dynamic scheduling optimization decision module is used for evaluating optimal task execution nodes; network delay, cost, compliance and real-time load in four dimensions are calculated according to the task descriptor and the topology graph, and a four-dimensional evaluation matrix is constructed; the optimal cloud node with the highest comprehensive score is output by weighted fusion of the scores of each dimension; The secure task dispatching and federated computing module is used for realizing differentiated execution of sensitive and non-sensitive tasks; sensitive tasks are processed in fragments on edge nodes through federated learning, and only encrypted gradient parameters are returned; non-sensitive tasks are dispatched to optimal computing nodes in fragments according to the MapReduce principle after field conversion, and transparent encryption and cross-cloud aggregation are performed; the federated learning fragmentation strategy is optimized by reinforcement learning, and the delay, cost and accuracy weights are dynamically adjusted; The privacy protection aggregation and traceability module is used for guaranteeing the security and auditability of results; Paillier homomorphic encryption is used to aggregate gradient update packages and intermediate calculation results, and after decryption, Laplace noise is injected to realize differential privacy protection; data source cloud platforms are labeled for desensitized data sets, and multi-dimensional interactive charts containing cost consumption analysis are generated, forming an intelligent analysis report with verifiable traceability; The resource recycling and state monitoring module is used for managing the life cycle of resources; temporary instances of completed tasks are actively identified, cloud APIs are called to immediately destroy them and generate release reports; database connection states are continuously monitored, and connections exceeding the idle threshold are gracefully closed, and connection state logs are recorded to guarantee resource utilization.
[0016] (Three) beneficial effects The present application has the following beneficial effects: The present application takes metadata intelligent alignment, dynamic optimization scheduling, hierarchical privacy protection and resource closed-loop recycling as the core chain, solves the three major problems of data fragmentation, inefficient scheduling and compliance vulnerability in the multi-cloud scenario; in the strong supervision fields of financial risk control and medical research, the cross-cloud data analysis period is reduced, and at the same time, the data flow is ensured to comply with privacy protection.
[0017] The application solves the problem of multi-cloud data island, realizes global data intelligent management, and solves the problem of metadata split of heterogeneous storage systems through automatic collection agent and cross-cloud field mapping dictionary; the unified metadata directory abstracts physical storage into a virtual table, improves the cross-cloud query response speed and field semantic alignment accuracy; the interactive topology graph dynamically labels the data physical location and compliance label, improves the compliance audit efficiency, and eliminates the data flow error caused by metadata ambiguity. The application dynamically optimizes resource scheduling and reduces comprehensive cost; the application creates a four-dimensional evaluation matrix decision model; based on real-time delay threshold filtering high-delay nodes, the influence of cross-country transmission jitter is avoided, the cloud platform price API is integrated to dynamically calculate the optimal unit price, the real-time load scoring mechanism is used to preferentially select idle resources, and the risk area is automatically avoided through binding labels, the model reduces the task execution cost and reduces the risk of exceeding the budget. The hierarchical privacy computing pipeline of the application meets the mathematically verifiable security guarantee, processes in situ at the edge node through federated learning fragmentation, only returns encrypted gradients, solves the problem of data leakage, uses homomorphic encryption to aggregate intermediate results, supports average value calculation and other operations in ciphertext state, realizes differential privacy through Laplace noise injection, improves privacy compliance coverage, and realizes accurate positioning of the data source and cost consumption of each result. The application reduces idle resource waste through resource recycling automation, reduces release delay by actively destroying temporary instances of completed tasks, and improves measured resource utilization by monitoring database connection idle threshold and automatically closing timeout connections.
[0018] Of course, any product implementing the application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.
[0020] Figure 1 A flowchart of a multi-cloud data analysis method based on a cloud platform of the application; Figure 2 A module diagram of a multi-cloud data analysis system based on a cloud platform of the application. DETAILED DESCRIPTION
[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0022] In the description of the present application, it should be understood that the terms "opening", "upper", "lower", "top", "middle", "inner" and the like indicate the orientation or positional relationship, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the components or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.
[0023] Embodiment one: Please refer to Figure 1 The present application discloses a multi-cloud data analysis method based on a cloud platform, comprising the following steps: S1, configuring a multi-cloud environment and initializing to obtain a structured metadata list and a heterogeneous data mode mapping table; The S1 comprises the following steps: S11, configuring a network security policy, inputting cloud platform account credentials (API key / access key), configuring cross-account access permissions through a cloud service provider console, establishing a TLS 1.3 encryption channel using an OpenSSL tool, setting a role-based access control (RBAC) permission matrix, obtaining an encrypted authentication credential library (such as AWS, Azure and Aliyun), and a secure connection endpoint list (including SSL certificate fingerprints); In a public cloud, automatic deployment is performed through CloudFormation / Terraform; in a private cloud, batch installation is performed using an Ansible script; and in an edge node, a Docker containerized collector is deployed; S12, setting a target cloud node from the secure connection endpoint list to obtain a target cloud node set; deploying a lightweight collection agent on the target cloud node in the target cloud node set, and automatically scanning the storage structure of the encrypted authentication credential library through a cloud-native API to obtain a structured metadata list (including storage type / encryption status / access mode) and a heterogeneous data mode mapping table (JSON / Parquet / CSV format conversion rules); For AWS S3 identification, the object header information in the bucket is scanned, the storage category is parsed, and the server-side encryption status is extracted; for Azure Blob identification, the Blob hierarchical structure is analyzed, and the metadata attributes are read; for Aliyun OSS identification, the storage redundancy type is detected and the lifecycle rule configuration is obtained; S2, constructing a unified metadata catalog and an interactive topology graph based on the structured metadata list and the heterogeneous data schema mapping table; The S2 includes the following steps: S21, using Apache Parquet tools, CSV Sniffer, and JSONPath processors respectively to parse columnar storage structures in the structured metadata list, detect separators and encoding formats, and extract nested fields, to obtain parsed structured metadata; S22, based on the heterogeneous data schema mapping table and the field meanings of the parsed structured metadata, establishing type conversion rules through semantic similarity algorithm matching, combining target data types, and marking special fields in the structured metadata list, to obtain a cross-cloud field mapping dictionary; S23, based on the cross-cloud field mapping dictionary and cloud platform infrastructure information, using virtualization technology to create a unified SQL interface, encapsulating underlying storage differences (such as S3 path → virtual database table), and implementing transparent data access (without sensing physical location), to obtain a unified metadata catalog (UMC) view; as follows, Logical table name Source cloud platform Physical location Compliance level Field mapping status user_profile AWS us-east-1 HIPAA Aligned device_logs Azure westeurope GDPR Partial conversion Determine the data physical location through IP geolocation, label compliance tags based on storage regions (such as GDPR / HIPAA), and integrate cloud service provider compliance certification status, to obtain an interactive data topology graph; S3, collecting real-time multi-modal request data; based on the unified metadata catalog and the interactive topology graph, performing semantic analysis and blood relationship analysis on the real-time multi-modal request data, to obtain standardized task descriptors; The S3 includes the following steps: S31, collecting SQL query statements (such as cross-cloud joint queries), machine learning task descriptions (such as TensorFlow training jobs), and stream processing requests (such as Kafka real-time analysis), to obtain real-time multi-modal request data; S32, receiving real-time multi-modal request data through a RESTful API gateway, using a protocol converter to standardize the input format of the real-time multi-modal request data, to obtain standardized real-time multi-modal request data; Query the unified metadata catalog through the logical table name, obtain the physical location of the standardized real-time multi-modal request data, and load the field mapping rules, to obtain request data with physical location markers; S33, analyze the cross-cloud association in the SQL statement in the request data with physical location markers (such as: edge.sensor_data→edge node), identify the data dependency of the machine learning task in the standardized real-time multi-modal request data (such as: training set location), extract the window parameters of the stream processing task in the standardized real-time multi-modal request data, and obtain the structured task metadata; Bind the compliance level of the unified metadata directory to generate a risk report, mark the data flow path of the structured task metadata on the interactive data topology graph, and construct the task dependency to obtain the data bloodline analysis result; S34, synthesize the structured task metadata and the data bloodline analysis result to obtain the standardized task descriptor; S4, based on the standardized task descriptor, network, cost, compliance and load evaluation of the cloud nodes of the cloud platform, and obtain the optimal cloud node; The S4 includes the following steps: S41, set the cross-cloud delay data threshold; obtain the cross-cloud delay data according to the target location in the standardized task descriptor combined with the interactive data topology graph; remove the cloud nodes with cross-cloud delay data greater than the cross-cloud delay data threshold, and obtain the candidate cloud nodes; construct a network score matrix based on the cross-cloud delay data of the candidate cloud nodes; the network score matrix includes the candidate cloud nodes, the delay and the network score of the candidate cloud nodes; S42, collect the cloud platform price API, combine the standardized task descriptor, calculate the cost of the candidate cloud nodes in the network score matrix, and obtain the cost score matrix; the cost score matrix; the cost score matrix includes the candidate cloud nodes, the unit price, the estimated cost and the cost score; the cost matrix is as follows, candidate cloud node unit price estimated cost score Tokyo 0.12 0.96 0.85 Frankfurt 0.15 1.20 0.75 Singapore 0.18 1.44 0.65 S43, score the candidate cloud nodes based on the compliance labels of the unified metadata directory to obtain the compliance score matrix; S44, obtain the load data of the candidate nodes based on the interactive data topology graph; score the candidate nodes based on the load data to obtain the load score matrix; for example, get the current load of AWS Tokyo: 68% → score = 1-0.68 = 0.32, detect the load of Azure Frankfurt: 52% → score = 1-0.52 = 0.48; S45, fuse the network score matrix, the cost score matrix, the compliance score matrix and the load score matrix to obtain a four-dimensional evaluation matrix; the four-dimensional score matrix includes the network score, the cost score, the compliance score and the load score of each candidate cloud node; S46, set a weight for each score in the four-dimensional score matrix to obtain a weight set; calculate the comprehensive score of each candidate cloud node based on the four-dimensional evaluation matrix and the weight set to obtain a final score set; select the cloud node corresponding to the highest score in the final score set to obtain an optimal cloud node; S5, input the sensitive task in the standardized task descriptor to the optimal cloud node, and obtain an encrypted gradient update package through federated learning fragmentation; perform field conversion, calculation of encrypted aggregation processing on the non-sensitive task in the standardized task descriptor to obtain an encrypted intermediate calculation result set; The S5 includes the following steps: S51, input the sensitive task in the standardized task descriptor to the optimal cloud node, and perform local training on each edge node by using federated learning fragmentation, and only return the encrypted gradient parameters to obtain an encrypted gradient update package; The federated learning is implemented by the following steps: S511, collect historical execution index data; the historical execution index data includes task execution logs (actual time consumption), cloud platform billing APIs (actual resource cost) and precision evaluation reports; Extract the key indicators of the historical execution index data, including actual delay vs. estimated delay, actual cost vs. estimated cost and result precision, to obtain a historical task execution index triple (delay, cost and precision); S512, based on the historical task execution index triple, use reinforcement learning combined with a multi-objective genetic algorithm to optimize and update the reward function weight coefficient to obtain an updated scheduling strategy; S52, transmit data through a dedicated channel for the non-sensitive task in the standardized task descriptor, perform field conversion using a field mapping rule, and perform format standardization to obtain a converted intermediate data set; S53, split the converted intermediate data set according to the MapReduce principle to obtain a task fragment set; select optimal computing cloud nodes from an interactive data topology graph to obtain a computing cloud node set; distribute the fragments in the task fragment set to the cloud nodes in the computing cloud node set; The cloud nodes in the computing cloud node set calculate the fragments in the task fragment set, and perform transparent encryption and cross-cloud aggregation to obtain an encrypted intermediate calculation result set; S6, aggregate, calculate and perform differential privacy protection processing on the encrypted gradient update package and the encrypted intermediate calculation result set to obtain a differential privacy satisfying desensitized data set; The S6 includes the following steps: S61, using a Paillier homomorphic encryption algorithm to aggregate the encrypted gradient update package and the encrypted intermediate calculation result set to obtain an encrypted global intermediate calculation result set; Performing an average operation on the encrypted global intermediate calculation result set to obtain an aggregated encrypted result set; S62, decrypting the aggregated encrypted result set to obtain a decrypted aggregated result; injecting Laplace noise into the decrypted aggregated result and verifying differential privacy to obtain a differential privacy satisfying desensitization data set; S63, labeling the data source cloud platform in the differential privacy satisfying desensitization data set, and generating a multi-dimensional interactive chart with additional cost consumption analysis to obtain a verifiable traceable intelligent analysis report; S7, after S6, destroying the completed task and releasing the cloud platform resources; The S7 includes the following steps: S71, identifying the temporary instance of the completed task, calling the cloud platform API to immediately destroy the instance, and obtaining a resource release confirmation report (instance ID / release time); S72, setting an idle time threshold; detecting that the idle time is greater than the idle time threshold, performing graceful shutdown to release the connection, and obtaining a database connection state log.
[0024] Embodiment two: Please refer to Figure 2 A multi-cloud data analysis system based on a cloud platform, for realizing the above-mentioned multi-cloud data analysis method based on a cloud platform, comprising a multi-cloud environment initialization and metadata acquisition module, a unified metadata directory and topology construction module, a multi-modal request analysis and blood relationship analysis module, a dynamic scheduling optimization decision module, a security task dispatching and federal computing module, a privacy protection aggregation and traceability module, and a resource recycling and state monitoring module. The multi-cloud environment initialization and metadata acquisition module is used to establish a secure cross-cloud connection and extract heterogeneous data metadata; by configuring network security policies and automatic deployment tools, credential management and node access of multiple platforms are realized; a lightweight agent is deployed on the target node, the storage structure is scanned, and a structured metadata list containing storage types, encryption status and format conversion rules and a heterogeneous data mode mapping table are generated; The unified metadata catalog and topology building module are used to parse raw metadata and build a global view; data patterns are parsed using Parquet tools, CSV Sniffer, and other technologies, and a cross-cloud field mapping dictionary is established through a semantic similarity algorithm; a virtualized SQL interface is created based on this, encapsulating physical storage differences to form a unified metadata catalog, and marking logical table and physical location mapping relationships; an interactive data topology map is generated by combining IP geolocation and compliance certification, enabling visualization of data location and compliance; The multi-modal request parsing and bloodline analysis module is used to process diversified computing requests and track data bloodlines; SQL queries, machine learning tasks, and stream processing requests are received through an API gateway, standardized, and associated with the UMC to obtain physical location and field mapping rules; task dependency relationships are parsed, data flow paths are marked on the topology map, and risk reports and data bloodline analysis results are generated in combination with compliance labels, and finally a standardized task descriptor containing complete context is output; The dynamic scheduling optimization decision module is used to evaluate optimal task execution nodes; a four-dimensional evaluation matrix is constructed by calculating network delay, cost, compliance, and real-time load in four dimensions based on the task descriptor and topology map; the optimal cloud node with the highest comprehensive score is output by weighting and fusing the scores of each dimension; The secure task dispatching and federated computing module is used to implement differentiated execution of sensitive and non-sensitive tasks; sensitive tasks are processed in shards on edge nodes through federated learning, and only encrypted gradient parameters are returned; non-sensitive tasks are converted by fields and dispatched to optimal computing nodes according to the MapReduce principle, performing transparent encryption and cross-cloud aggregation; federated learning shard strategies are optimized using reinforcement learning, dynamically adjusting delay, cost, and accuracy weights; The privacy protection aggregation and traceability module is used to ensure the security and auditability of the results; Paillier homomorphic encryption is used to aggregate gradient update packages and intermediate calculation results, and after decryption, Laplace noise is injected to achieve differential privacy protection; desensitized data sets are labeled with data source cloud platforms to generate multi-dimensional interactive charts containing cost consumption analysis, forming an intelligent analysis report with verifiable traceability; The resource recycling and state monitoring module is used to manage resource lifecycles; temporary instances of completed tasks are actively identified, cloud APIs are called to immediately destroy them, and release reports are generated; database connection states are continuously monitored, and connections exceeding the idle threshold are gracefully closed, and connection state logs are recorded to ensure resource utilization.
[0025] In the description of the specification, reference to "one embodiment", "an example", "a specific example" or the like means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The appearances of the phrases "in one embodiment", "an example", "a specific example" or the like in various places in the specification are not necessarily referring to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0026] The preferred embodiments of the application disclosed above are only to help explain the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the specification. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and use the application.
Claims
1. A cloud platform-based multi-cloud data analysis method, characterized in that, The method comprises the following steps: S1, configuring a multi-cloud environment and initializing to obtain a structured metadata list and a heterogeneous data mode mapping table; S2, constructing a unified metadata directory and an interactive topology graph based on the structured metadata list and the heterogeneous data mode mapping table; S3, collecting real-time multi-modal request data; Based on the unified metadata directory and the interactive topology graph, the real-time multi-modal request data is subjected to semantic analysis, blood analysis processing, and a standardized task descriptor is obtained; S4, based on the standardized task descriptor, network, cost, compliance and load evaluation of the cloud nodes of the cloud platform are carried out, and the optimal cloud node is obtained; S5, the sensitive task in the standardized task descriptor is input into the optimal cloud node, and the encrypted gradient update package is obtained through federated learning fragmentation; the non-sensitive task in the standardized task descriptor is subjected to field conversion, calculation and encryption aggregation processing, and the encrypted intermediate calculation result set is obtained; S6, the encrypted gradient update package and the encrypted intermediate calculation result set are aggregated, calculated and subjected to differential privacy protection processing, and a differential privacy desensitization data set is obtained; S7, after S6, the completed task is destroyed, and the cloud platform resources are released.
2. The multi-cloud data analysis method based on a cloud platform according to claim 1, characterized in that, The S1 comprises the following steps: S11, configuring a network security policy, inputting a cloud platform account credential, configuring a cross-account access permission through a cloud service provider console, establishing a TLS1.3 encryption channel, setting a role-based access control permission matrix, obtaining an encrypted authentication credential library and a secure connection endpoint list; S12, setting a target cloud node from the secure connection endpoint list to obtain a target cloud node set; deploying a lightweight collection agent on the target cloud node in the target cloud node set, and automatically scanning the storage structure of the encrypted authentication credential library through a cloud native API to obtain a structured metadata list and a heterogeneous data mode mapping table.
3. The multi-cloud data analysis method based on a cloud platform according to claim 1, characterized in that, The S2 comprises the following steps: S21, using Apache Parquet tools, CSV Sniffer and JSONPath processors to parse columnar storage structures in the structured metadata list, detect delimiters and encoding formats, and extract nested fields, respectively, to obtain parsed structured metadata; S22, based on the field meanings of the parsed structured metadata and the heterogeneous data mode mapping table, matching through a semantic similarity algorithm, establishing type conversion rules in combination with target data types, and marking special fields in the structured metadata list, to obtain a cross-cloud field mapping dictionary; S23, based on the cross-cloud field mapping dictionary and cloud platform infrastructure information, using virtualization technology to create a unified SQL interface, encapsulating underlying storage differences, and realizing transparent data access, to obtain a unified metadata directory view; An interactive data topology graph is obtained by determining the physical location of data through IP geolocation, labeling compliance tags based on storage areas, and integrating cloud service provider compliance certification status.
4. The multi-cloud data analysis method based on a cloud platform according to claim 1, characterized in that, The S3 comprises the following steps: S31, collecting SQL query statements, machine learning task descriptions and stream processing requests to obtain real-time multi-modal request data; S32, receive real-time multi-modal request data through the RESTful API gateway, use a protocol converter to standardize the real-time multi-modal request data into a format, and obtain standardized real-time multi-modal request data; Obtain the physical location of the standardized real-time multi-modal request data by querying the unified metadata directory through the logical table name, and load the field mapping rule to obtain the request data with a physical location marker; S33, parse the cross-cloud association in the SQL statement in the request data with a physical location marker, identify the data dependency of the machine learning task in the standardized real-time multi-modal request data, and extract the window parameters of the stream processing task in the standardized real-time multi-modal request data, to obtain structured task metadata; Bind the compliance level of the unified metadata directory to generate a risk report, mark the data flow path of the structured task metadata on the interactive data topology graph, and construct task dependency to obtain data lineage analysis results; S34, synthesize the structured task metadata and the data lineage analysis results to obtain a standardized task descriptor.
5. The multi-cloud data analysis method based on a cloud platform according to claim 1, characterized in that The S4 includes the following steps: S41, set a cross-cloud delay data threshold; obtain cross-cloud delay data according to the target location in the standardized task descriptor combined with the interactive data topology graph; remove the cloud nodes with cross-cloud delay data greater than the cross-cloud delay data threshold to obtain candidate cloud nodes; and construct a network score matrix based on the cross-cloud delay data of the candidate cloud nodes; the network score matrix includes the candidate cloud nodes, the delay of the candidate cloud nodes, and the network score; S42, collect cloud platform price APIs, and calculate the cost of the candidate cloud nodes in the network score matrix combined with the standardized task descriptor to obtain a cost score matrix; the cost score matrix; the cost score matrix includes the candidate cloud nodes, the unit price of the cloud nodes, the estimated cost, and the cost score; S43, score the candidate cloud nodes based on the compliance labels of the unified metadata directory to obtain a compliance score matrix; S44, obtain load data of the candidate nodes based on the interactive data topology graph; score the candidate nodes based on the load data to obtain a load score matrix; S45, fuse the network score matrix, the cost score matrix, the compliance score matrix, and the load score matrix to obtain a four-dimensional evaluation matrix; the four-dimensional score matrix includes the network score, the cost score, the compliance score, and the load score of each candidate cloud node; S46, set weights for each score in the four-dimensional score matrix to obtain a weight set; calculate the comprehensive score of each candidate cloud node based on the four-dimensional evaluation matrix combined with the weight set to obtain a final score set; select the cloud node corresponding to the highest score in the final score set to obtain the optimal cloud node.
6. The multi-cloud data analysis method based on a cloud platform according to claim 1, characterized in that, The S5 includes the following steps: S51, input the sensitive task in the standardized task descriptor to the optimal cloud node, use federated learning fragmentation for the sensitive task, perform local training on each edge node, and only return encrypted gradient parameters to obtain an encrypted gradient update package; S52, transmit data for the non-sensitive task in the standardized task descriptor through a dedicated channel, use a field mapping rule to convert fields, and perform format standardization to obtain a converted intermediate data set; S53, split the converted intermediate data set according to the principle of MapReduce to obtain a task fragment set; select the optimal computing cloud node from the interactive data topology graph to obtain a computing cloud node set; distribute the fragments in the task fragment set to the cloud nodes in the computing cloud node set; The cloud nodes in the computing cloud node set calculate the fragments in the task fragment set, and perform transparent encryption and cross-cloud aggregation to obtain an encrypted intermediate calculation result set.
7. The multi-cloud data analysis method based on a cloud platform according to claim 6, characterized in that, The federated learning in S51 is implemented by the following steps: S511, collect historical execution index data; the historical execution index data includes task execution logs, cloud platform billing API, and precision evaluation reports; Extract the key indicators of the historical execution index data to obtain a historical task execution index triple; S512, based on the historical task execution index triple, use reinforcement learning combined with a multi-objective genetic algorithm to optimize and update the reward function weight coefficient to obtain an updated scheduling strategy. 8.The cloud platform based multi-cloud data analysis method of claim 1, wherein, S6 includes the following steps: S61, use the Paillier homomorphic encryption algorithm to aggregate the encrypted gradient update package and the encrypted intermediate calculation result set to obtain an encrypted global intermediate calculation result set; Perform average operation on the encrypted global intermediate calculation result set to obtain an aggregated encrypted result set; S62, decrypt the aggregated encrypted result set to obtain a decrypted aggregation result; inject Laplace noise into the decrypted aggregation result and verify differential privacy to obtain a desensitized data set that satisfies differential privacy; S63 label the data source cloud platform in the desensitized data set that satisfies differential privacy, and generate a multi-dimensional interactive chart with additional cost consumption analysis to obtain an intelligent analysis report that can be verified and traced. 9.The cloud platform based multi-cloud data analysis method of claim 1, wherein, S7 includes the following steps: S71, identify the temporary instance of the completed task, call the cloud platform API to immediately destroy the instance, and obtain a resource release confirmation report; S72, set an idle time threshold; detect that the idle time is greater than the idle time threshold, perform graceful shutdown to release the connection, and obtain a database connection state log.
10. A cloud platform based multi-cloud data analytics system, characterized in that, A multi-cloud data analysis method based on a cloud platform is implemented as claimed in any one of claims 1-9, the system comprising a multi-cloud environment initialization and metadata collection module, a unified metadata directory and topology construction module, a multi-modal request analysis and blood relationship analysis module, a dynamic scheduling optimization decision module, a secure task dispatching and federated computing module, a privacy protection aggregation and tracing module, and a resource recycling and state monitoring module.