Enterprise cloud asset global data integration management method based on data center

Abstract

The invention discloses an enterprise cloud asset global data integration management method based on a data center station, and aims to solve the problems of data source dispersion, complex management, data islanding and the like, and the method comprises the following steps: automatically identifying and classifying various data sources of an enterprise through an evolutionary algorithm, and generating a dynamic data source directory; based on the directory, a self-adaptive data access strategy is generated, unified access of structured, semi-structured and unstructured data and data synchronization of a cross-cloud platform are realized, a unified data model is constructed in a data platform, and unified mapping of data of different sources is realized through a semantic mapping technology. In the data transmission process, the optimal path set is adopted to ensure high efficiency and stability of data transmission. According to the method, the received data is dynamically treated through a self-adaptive strategy, a cross-cloud data virtualization layer is constructed based on the treated data, transparent management of data resources is realized, and data analysis is performed by adopting real-time processing and a differential privacy enhancement algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of enterprise data management, and in particular to an enterprise cloud asset global data integration management method based on a data middle platform. Background Art

[0002] In the current information age, the amount of data generated and collected by enterprises in the course of business operations has exploded. With the popularization of cloud computing technology, more and more enterprises choose to deploy data and applications on multiple cloud platforms to achieve flexible resource scheduling and cost optimization. However, in a multi-cloud environment, enterprises face many challenges, mainly including decentralized data sources, isolated data, increased management complexity, and data security and privacy protection.

[0003] In the existing technology, enterprise data management in a multi-cloud environment faces several key defects and challenges. First, the problem of scattered and isolated data sources is particularly prominent. In a multi-cloud environment, enterprise data is usually distributed on multiple different cloud platforms. The architecture and data format between different cloud platforms are quite different, resulting in poor interoperability between data sources. Traditional solutions often have difficulty in automatically identifying and classifying these scattered data sources, which leads to difficulties in data integration and the inability to form a global data view. This data isolation state seriously limits the unified management and efficient use of data by enterprises, hindering data-driven business innovation and decision optimization. The difficulty of data access and synchronization has also become an important challenge faced by enterprises in a multi-cloud environment. Existing technologies usually rely on manual or semi-automatic methods to access data of different structures to the data middle platform, which is a cumbersome and error-prone process. At the same time, the data synchronization mechanism across cloud platforms is complex, and it is difficult to ensure real-time synchronization of data between different cloud platforms, which easily leads to data inconsistency and redundancy problems, thereby affecting the real-time and accuracy of enterprise business. In this case, it is difficult for enterprises to support rapid business decisions through real-time data flow analysis.

[0004] In terms of building a unified data model, due to the complex and heterogeneous data sources in a multi-cloud environment, existing technologies have difficulty in handling the differences between different data structures, especially the mapping of semi-structured and unstructured data is often not accurate enough. This makes it difficult for the construction of data models to meet the needs of enterprises for multi-dimensional data analysis and decision-making, limiting the role of the data center in a multi-cloud environment. Low data transmission efficiency is also a common problem in a multi-cloud environment. Due to the complexity of the data transmission path and the huge amount of data, it is difficult for existing technologies to select the optimal transmission path during cross-cloud data transmission, resulting in low data transmission efficiency. Data loss or delay may also occur during the transmission process, affecting the continuity and stability of the enterprise's business, which is particularly unfavorable for business scenarios that require real-time data support. In terms of data governance, existing technologies often lack sufficient flexibility and dynamism. When facing dynamically changing data sources in a multi-cloud environment, traditional data governance solutions usually have fixed governance processes, which are difficult to dynamically adjust according to the characteristics of the data source and business priorities, resulting in unsatisfactory governance results and ineffective data quality assurance. Such a governance model cannot meet the needs of modern enterprises for high-quality data, which in turn affects the analysis and use of data.

[0005] In terms of virtualization and transparent management, enterprises in multi-cloud environments face huge challenges in unified management and transparent access to data resources on different cloud platforms. When building a cross-cloud data virtualization layer, existing technologies usually lack a unified representation and management mechanism for different data sources, resulting in enterprises often needing to rely on independent interfaces of each cloud platform when using and managing data resources in a multi-cloud environment. This not only increases the complexity of management, but also reduces the efficiency of data access. Data security and privacy protection issues are particularly prominent in a multi-cloud environment. With the increasingly stringent laws and regulations on data privacy protection, enterprises need to take more stringent privacy protection measures during data processing and analysis. However, when implementing differential privacy protection, existing technologies often face the contradiction between data accuracy and privacy protection strength, making it difficult to achieve comprehensive privacy protection in a multi-cloud environment. At the same time, data is vulnerable to security threats and sensitive information leakage during cross-platform transmission and storage, which poses severe challenges to enterprise data security and compliance management.

[0006] Therefore, how to provide an enterprise cloud asset global data integration management method based on a data middle platform is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0007] One purpose of the present invention is to propose a data-based enterprise cloud asset global data integration management method. The present invention uses an evolutionary data source identification algorithm, a heterogeneous ensemble learning algorithm, and a hyperdimensional evolution algorithm to automatically identify, classify, access, and manage various data sources of an enterprise in a multi-cloud environment, and build a cross-cloud data virtualization layer to support real-time data processing and differential privacy protection. The present invention describes in detail the key steps of data integration, governance, transmission, virtualization, and privacy protection in a multi-cloud environment, and has the advantages of high data integration, flexible governance, efficient transmission, strong privacy protection, and adaptability to a multi-cloud environment.

[0008] According to an embodiment of the present invention, a method for enterprise cloud asset global data integration management based on a data middle platform includes the following steps:

[0009] S1. Use an evolutionary data source identification algorithm to automatically identify and classify various data sources of an enterprise in a multi-cloud environment, generate an initial feature vector set, continuously optimize the data source identification and classification strategy by simulating the natural selection process, and finally generate a dynamic data source directory;

[0010] S2. Based on the generated dynamic data source directory, generate adaptive data access strategies according to the characteristics of different data sources, realize seamless access of structured, semi-structured and unstructured data to the data center, and synchronize and update data across cloud platforms to form a unified access data set;

[0011] S3. In the data center, use the access data set to build a unified data model for the enterprise, and use semantic mapping technology to map data with different structures and sources in the multi-cloud environment into a unified data model;

[0012] S4. Based on the constructed data model, we implement optimized management of cross-cloud data flows, dynamically adjust data transmission paths, form an optimal path set, select the optimal path based on real-time business needs and data characteristics, and perform stable data transmission;

[0013] S5. Use the optimal path set for data transmission, dynamically manage the received data through adaptive strategies, form a managed data set, and flexibly adjust the management process according to business priorities;

[0014] S6. Use the governed data sets to build a cross-cloud data virtualization layer. Use data virtualization technology to present data resources on different cloud platforms in a unified manner, generate virtualized data interfaces, provide unified access to cross-cloud data, and support transparent management and access of data.

[0015] S7. In the data center, virtualized data interfaces are used to adjust processing strategies in real time, optimize resource allocation, and generate real-time processing result sets in combination with dynamic changes in data flows during real-time data processing;

[0016] S8. Based on the real-time processing result set, a differential privacy enhancement algorithm is introduced to perform data analysis in the global data security framework. By adding noise in the data analysis process, personal privacy is protected and sensitive information leakage is avoided. At the same time, the analytical value of the data is maintained to achieve ultimate data security and compliance management.

[0017] Optionally, the S1 specifically includes:

[0018] S11. Use the evolutionary data source identification algorithm to perform preliminary scanning and identification of various enterprise data sources in a multi-cloud environment to generate an initial feature vector V i =(v i1 ,v i2 ,…,v in ), where v ij represents the jth characteristic attribute of the i-th data source, and the characteristic attributes include data type, data format, data source location, data size and update frequency;

[0019] S12, based on the initial feature vector V i , using the evolutionary data source identification algorithm to perform cluster analysis and generate the initial classification result set C = {C1, C2, ..., C m}, where C k Represents the k-th data source:

[0020]

[0021] Among them, S(V i ,V j ) represents the feature vector V i and V j The similarity between

[0022] S13, optimize the initial classification results by simulating the natural selection process, including selection, crossover and mutation operations. The selection process is based on the fitness function F(C k )Evaluate each type of data source:

[0023]

[0024] Among them, |C k |Indicates category C k The number of data sources in S(V i ,V j ) is the similarity value, and the category with high fitness is retained. The crossover operation generates a new data source category by combining feature vectors of different categories, and the mutation operation randomly adjusts the feature vector V′ i Some attribute values ​​of generate new feature vectors;

[0025] S14. Generate a dynamic data source directory D = {D1, D2, ..., D q}, where D i Represents the dynamic attribute set of the i-th type of data source, including the data source location, update cycle, and data type attributes, ultimately forming a dynamically adjustable data source directory.

[0026] Optionally, the S2 specifically includes:

[0027] S21, based on the generated dynamic data source directory D = {D1, D2, ..., D q}, where D i The dynamic attribute set of the i-th data source includes the data source location, update cycle and data type attributes. For different types of data source attributes, an initial access parameter set P = {P1, P2, ..., P q}, each access parameter includes data access protocol, data transmission rate and access frequency information related to the corresponding data source;

[0028] S22. Based on the access parameter set P, generate an adaptive data access policy set S = {S1, S2, ..., S q}, each access strategy dynamically adjusts the access frequency, data transmission rate and protocol selection according to the changing characteristics of the data source to optimize the access process;

[0029] S23. Introduce heterogeneous ensemble learning algorithms to integrate multiple heterogeneous models in a multi-cloud environment and dynamically select the optimal model combination to optimize data access strategies:

[0030]

[0031] Among them, S i (t) is the access strategy of the i-th data source at time t, M m (X i ) is the mth heterogeneous model for the input data X i The output, w m is the weight of the model, α is the learning rate, is the loss gradient, β is the gradient learning rate, R(S i ) is the regularization term, γ is the regularization weight;

[0032] S24. During the access strategy execution process, each data source is monitored in real time, and the access parameter set P is dynamically adjusted through the feedback control mechanism. i =P i +ΔP real-time update parameters, where ΔP represents the adjustment amount based on monitoring feedback;

[0033] S25. Use adaptive data access strategies to achieve seamless access to structured, semi-structured, and unstructured data, access various types of data to the data center through standardized interfaces, and synchronize and update data across cloud platforms based on the dynamic attributes of the data source to form a unified access data set C unified .

[0034] Optionally, the S5 specifically includes:

[0035] S51, based on the generated optimal path set R = {R1, R2, ..., R n}, select path R i For data transmission, each path R i Dynamic selection based on the characteristics of the data source and real-time business needs;

[0036] S52, receiving data transmitted through the optimal path set R, and performing preliminary analysis to generate an original data set in Represents the jth received original data;

[0037] S53, applying the adaptive strategy to the original data set O raw Perform dynamic governance, including data cleaning, deduplication, format conversion, and standardization. The governed data set is represented as

[0038] S54. Introduce hyperdimensional evolutionary algorithm to optimize governance strategies in high-dimensional space:

[0039]

[0040] Among them, θ * , * , σ * They represent the optimal solutions for cleaning parameters, deduplication strategies, format conversion rules, and standardization specifications, respectively. is the loss function between the governed dataset and the target dataset, α i is the weight of loss for different data sets, is the regularization term, β is the regularization weight;

[0041] S55. In the governance process, based on business priorities Dynamically adjust the governance process. The optimized governance process is updated by the formula G(t+1)=G(t)+ΔG, where ΔG represents the increment adjusted according to the business priority;

[0042] S56, the generated governance data set O clean Input into the storage and processing module of the data center as the basic data support for data analysis, governance and business decision-making.

[0043] Optionally, the S6 specifically includes:

[0044] S61. Using Governed Datasets Build a cross-cloud data virtualization layer to virtualize data resources on different cloud platforms and represent each data source as a virtualized data resource set V = {V1, V2, ..., V n}, where V i represents the i-th virtualized data resource;

[0045] S62, based on the virtualized data resource set V, unified management is achieved through data virtualization technology, and a virtualized data interface I = {I1, I2, ..., I k}, where I j represents the jth virtualized data interface, each interface I j Contains data access paths and access control policies across cloud platforms;

[0046] S63: Introduce a path optimization algorithm to determine the optimal access path P for each virtualized data interface. opt :

[0047]

[0048] Among them, P opt (I j ) represents the optimal access path of the jth interface, γ l is the weight associated with path node l, T l (P) is the access time through node l on path P, C(P) is the total access cost of path P, and δ is the weight coefficient of the path cost;

[0049] S64. The generated virtualized data interface I provides unified data access across cloud platforms and supports transparent management and access of data resources;

[0050] S65. Deploy the virtualized data interface I on the data middle platform so that data resources across cloud platforms can be accessed through a unified interface.

[0051] Optionally, the S7 specifically includes:

[0052] S71, in the data center, using the virtualized data interface I = {I1, I2, ..., I k}, receiving and processing real-time data streams from different cloud platforms F = {F1, F2, ..., F m}, where F i represents the i-th real-time data stream;

[0053] S72: After receiving the real-time data stream, the data stream is initially parsed and preprocessed through the virtualized data interface to generate a preprocessed data stream set. Each data stream After standardization;

[0054] S73, combined with the pre-processed data stream set F pre The dynamic change characteristics of the data flow are monitored in real time. i (t), data flow v i (t), data delay d i (t), and data volatility ω i (t), dynamically adjust the processing strategy S proc (t):

[0055]

[0056] Among them, S proc (t) represents the processing strategy at time t, α is the adjustment step size, γ i , i , μ i , i are the weight coefficients related to data flow rate, traffic, delay and volatility, respectively. represents the gradient calculation of the processing strategy;

[0057] S74, according to the dynamically adjusted processing strategy S proc (t),Optimize resource allocation in the data center;

[0058] S75. Generate real-time processing result set Q real-time = {Q1,Q2,…,Q n}, each result Q j It contains key data analysis, forecast information and business indicators that have been processed in real time, and is transmitted to the corresponding business modules through the virtualized data interface I, supporting enterprises' instant decision-making, intelligent analysis and flexible response in a multi-cloud environment.

[0059] Optionally, the S8 specifically includes:

[0060] S81, based on real-time processing result set Q real-time = {Q1,Q2,…,Q n}, perform differential privacy processing on key data analysis results, prediction information and business indicators;

[0061] S82. Introduce the differential privacy enhancement algorithm to generate a privacy-protected analysis data set Q by adding random noise N(∈) during the data analysis process private ={Q′1,Q′2,…,Q′n}, where Q j ′=Q j +N(∈) represents the jth analysis result after adding noise. The random noise N(∈) is controlled by the privacy budget parameter ∈, which adjusts the balance between privacy protection strength and data accuracy.

[0062] S83. The generation of noise N(∈) is based on the following formula, which adjusts the noise by balancing privacy protection and data accuracy in high-dimensional space:

[0063]

[0064] in, is the standard differential privacy noise generated based on Laplace distribution, Δf is the global sensitivity of the query function f, ∈ is the privacy budget, η is the adjustment coefficient for controlling the high-dimensional noise term, represents the second-order derivative matrix of the loss function L with respect to the data set Q, β is the regularization parameter, and R is a positive definite matrix in the high-dimensional space;

[0065] S84. For the data set Q processed by differential privacy private Conduct data analysis so that the analysis results can support effective business decisions and data applications while protecting privacy;

[0066] S85. After completing the data analysis, evaluate the effect of differential privacy processing and optimize the balance between privacy protection and data analysis accuracy by adjusting the privacy budget parameter ∈ and the noise generation mechanism.

[0067] The beneficial effects of the present invention are:

[0068] The present invention automatically identifies and classifies various types of data sources of an enterprise in a multi-cloud environment, and generates a dynamic data source directory. This automated identification and classification method overcomes the problems of scattered and isolated data sources in the prior art, and effectively improves the accuracy and consistency of data integration. Based on this, the dynamic data source directory combines adaptive data access strategies to achieve seamless access of structured, semi-structured, and unstructured data to the data center, and supports data synchronization across cloud platforms. Compared with traditional methods, the present invention greatly reduces the complexity of data access and synchronization, ensures real-time updating and consistency of data, and thus supports the continuity and real-time requirements of enterprise business.

[0069] The present invention constructs a unified data model through a data middle platform, and uses semantic mapping technology to uniformly map data of different structures and sources in a multi-cloud environment to this model. Combined with the dynamic adjustment of the optimal path set, the present invention significantly improves the efficiency and stability of data transmission, ensures the high efficiency of data transmission across cloud platforms, and reduces the risk of delays and data loss. In terms of data governance, the present invention introduces adaptive strategies and hyperdimensional evolutionary algorithms, which can dynamically govern received data and handle key data governance tasks including data cleaning, deduplication, format conversion and standardization. By flexibly adjusting the governance process and optimizing management according to business priorities, the governed data has high quality and high applicability, meeting the company's high standards for data quality.

[0070] The present invention realizes the unified presentation and management of data resources in a multi-cloud platform, and provides a virtualized data interface to support transparent access to data. This function greatly simplifies the data management process in a multi-cloud environment, enabling enterprises to more conveniently access and utilize data resources distributed on different cloud platforms, significantly improving management efficiency and data resource utilization. In terms of real-time data processing and resource optimization, the present invention uses a virtualized data interface to achieve efficient processing of real-time data streams, and generates a real-time processing result set by dynamically adjusting processing strategies and optimizing resource allocation. This real-time processing capability significantly improves the response speed and flexibility of enterprises to changes in the business environment, thereby supporting intelligent decision-making and dynamic adjustment of enterprises in complex business scenarios. By introducing a differential privacy enhancement algorithm, random noise is added during the data analysis process, personal privacy is protected, and the leakage of sensitive information is avoided. A balance is achieved between data analysis and privacy protection, ensuring that the value of data analysis is not affected, and meeting the high requirements of enterprises in data security and compliance management. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0072] Figure 1 This is a flow chart of the enterprise cloud asset global data integration management method based on the data middle platform proposed by the present invention;

[0073] Figure 2 A flow chart for generating an adaptive data access strategy using a heterogeneous ensemble learning algorithm in the present invention;

[0074] Figure 3 A schematic diagram of optimizing the data governance process using a hyperdimensional evolutionary algorithm in the present invention;

[0075] Figure 4This is a flow chart of the differential privacy enhancement algorithm in the present invention applied to data analysis and privacy protection. DETAILED DESCRIPTION

[0076] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0077] refer to Figure 1-4 The enterprise cloud asset global data integration management method based on the data middle platform includes the following steps:

[0078] S1. Use an evolutionary data source identification algorithm to automatically identify and classify various data sources of an enterprise in a multi-cloud environment, generate an initial feature vector set, continuously optimize the data source identification and classification strategy by simulating the natural selection process, and finally generate a dynamic data source directory;

[0079] S2. Based on the generated dynamic data source directory, generate adaptive data access strategies according to the characteristics of different data sources, realize seamless access of structured, semi-structured and unstructured data to the data center, and synchronize and update data across cloud platforms to form a unified access data set;

[0080] S3. In the data center, use the access data set to build a unified data model for the enterprise, and use semantic mapping technology to map data with different structures and sources in the multi-cloud environment into a unified data model;

[0081] S4. Based on the constructed data model, we implement optimized management of cross-cloud data flows, dynamically adjust data transmission paths, form an optimal path set, select the optimal path based on real-time business needs and data characteristics, and perform stable data transmission;

[0082] S5. Use the optimal path set for data transmission, dynamically manage the received data through adaptive strategies, form a managed data set, and flexibly adjust the management process according to business priorities;

[0083] S6. Use the governed data sets to build a cross-cloud data virtualization layer. Use data virtualization technology to present data resources on different cloud platforms in a unified manner, generate virtualized data interfaces, provide unified access to cross-cloud data, and support transparent management and access of data.

[0084] S7. In the data center, virtualized data interfaces are used to adjust processing strategies in real time, optimize resource allocation, and generate real-time processing result sets in combination with dynamic changes in data flows during real-time data processing;

[0085] S8. Based on the real-time processing result set, a differential privacy enhancement algorithm is introduced to perform data analysis in the global data security framework. By adding noise in the data analysis process, personal privacy is protected and sensitive information leakage is avoided. At the same time, the analytical value of the data is maintained to achieve ultimate data security and compliance management.

[0086] In this implementation, S1 specifically includes:

[0087] S11. Use the evolutionary data source identification algorithm to perform preliminary scanning and identification of various enterprise data sources in a multi-cloud environment to generate an initial feature vector V i =(v i1 ,v i2 ,…,v in ), where v ij represents the jth characteristic attribute of the i-th data source, and the characteristic attributes include data type, data format, data source location, data size and update frequency;

[0088] S12, based on the initial feature vector V i , using the evolutionary data source identification algorithm to perform cluster analysis and generate the initial classification result set C = {C1, C2, ..., C m}, where C k Represents the k-th data source:

[0089]

[0090] Among them, S(V i ,V j ) represents the feature vector V i and V j The similarity between

[0091] S13, optimize the initial classification results by simulating the natural selection process, including selection, crossover and mutation operations. The selection process is based on the fitness function F(C k )Evaluate each type of data source:

[0092]

[0093] Among them, |C k |Indicates category C k The number of data sources in S(V i ,V j ) is the similarity value, and the category with high fitness is retained. The crossover operation generates a new data source category by combining feature vectors of different categories, and the mutation operation randomly adjusts the feature vector V′ i Some attribute values ​​of generate new feature vectors;

[0094] S14. Generate a dynamic data source directory D = {D1, D2, ..., D q}, where D i Represents the dynamic attribute set of the i-th type of data source, including the data source location, update cycle, and data type attributes, ultimately forming a dynamically adjustable data source directory.

[0095] In this implementation, S2 specifically includes:

[0096] S21, based on the generated dynamic data source directory D = {D1, D2, ..., D q}, where D i The dynamic attribute set of the i-th data source includes the data source location, update cycle and data type attributes. For different types of data source attributes, an initial access parameter set P = {P1, P2, ..., P q}, each access parameter includes data access protocol, data transmission rate and access frequency information related to the corresponding data source;

[0097] S22. Based on the access parameter set P, generate an adaptive data access policy set S = {S1, S2, ..., S q}, each access strategy dynamically adjusts the access frequency, data transmission rate and protocol selection according to the changing characteristics of the data source to optimize the access process;

[0098] S23. Introduce heterogeneous ensemble learning algorithms to integrate multiple heterogeneous models in a multi-cloud environment and dynamically select the optimal model combination to optimize data access strategies:

[0099]

[0100] Among them, S i (t) is the access strategy of the i-th data source at time t, M m (X i ) is the mth heterogeneous model for the input data X i The output, w m is the weight of the model, α is the learning rate, is the loss gradient, β is the gradient learning rate, R(S i ) is the regularization term, γ is the regularization weight;

[0101] S24. During the access strategy execution process, each data source is monitored in real time, and the access parameter set P is dynamically adjusted through the feedback control mechanism. i =P i +ΔP real-time update parameters, where ΔP represents the adjustment amount based on monitoring feedback;

[0102] S25. Use adaptive data access strategies to achieve seamless access to structured, semi-structured, and unstructured data, access various types of data to the data center through standardized interfaces, and synchronize and update data across cloud platforms based on the dynamic attributes of the data source to form a unified access data set C unified .

[0103] In this implementation manner, S5 specifically includes:

[0104] S51, based on the generated optimal path set R = {R1, R2, ..., R n}, select path R i For data transmission, each path R i Dynamic selection based on the characteristics of the data source and real-time business needs;

[0105] S52, receiving data transmitted through the optimal path set R, and performing preliminary analysis to generate an original data set in Represents the jth received original data;

[0106] S53, applying the adaptive strategy to the original data set O raw Perform dynamic governance, including data cleaning, deduplication, format conversion, and standardization. The governed data set is represented as

[0107] S54. Introduce hyperdimensional evolutionary algorithm to optimize governance strategies in high-dimensional space:

[0108]

[0109] Among them, θ * , * , σ * They represent the optimal solutions for cleaning parameters, deduplication strategies, format conversion rules, and standardization specifications, respectively. is the loss function between the governed dataset and the target dataset, α i is the weight of loss for different data sets, is the regularization term, β is the regularization weight;

[0110] S55. In the governance process, based on business priorities Dynamically adjust the governance process. The optimized governance process is updated by the formula G(t+1)=G(t)+ΔG, where ΔG represents the increment adjusted according to the business priority;

[0111] S56, the generated governance data set O clean Input into the storage and processing module of the data center as the basic data support for data analysis, governance and business decision-making.

[0112] In this implementation manner, S6 specifically includes:

[0113] S61. Using Governed Datasets Build a cross-cloud data virtualization layer to virtualize data resources on different cloud platforms and represent each data source as a virtualized data resource set V = {V1, V2, ..., V n}, where V i represents the i-th virtualized data resource;

[0114] S62, based on the virtualized data resource set V, unified management is achieved through data virtualization technology, and a virtualized data interface I = {I1, I2, ..., I k}, where I j represents the jth virtualized data interface, each interface I j Contains data access paths and access control policies across cloud platforms;

[0115] S63: Introduce a path optimization algorithm to determine the optimal access path P for each virtualized data interface. opt :

[0116]

[0117] Among them, P opt (I j ) represents the optimal access path of the jth interface, γ l is the weight associated with path node l, T l (P) is the access time through node l on path P, C(P) is the total access cost of path P, and δ is the weight coefficient of the path cost;

[0118] S64. The generated virtualized data interface I provides unified data access across cloud platforms and supports transparent management and access of data resources;

[0119] S65. Deploy the virtualized data interface I on the data middle platform so that data resources across cloud platforms can be accessed through a unified interface.

[0120] In this implementation manner, the S7 specifically includes:

[0121] S71, in the data center, using the virtualized data interface I = {I1, I2, ..., I k}, receiving and processing real-time data streams from different cloud platforms F = {F1, F2, ..., F m}, where F i represents the i-th real-time data stream;

[0122] S72: After receiving the real-time data stream, the data stream is initially parsed and preprocessed through the virtualized data interface to generate a preprocessed data stream set. Each data stream After standardization;

[0123] S73, combined with the pre-processed data stream set F pre The dynamic change characteristics of the data flow are monitored in real time. i (t), data flow v i (t), data delay d i (t), and data volatility ω i (t), dynamically adjust the processing strategy S proc (t):

[0124]

[0125] Among them, S proc (t) represents the processing strategy at time t, α is the adjustment step size, γ i , i , μ i , i are the weight coefficients related to data flow rate, traffic, delay and volatility, respectively. represents the gradient calculation of the processing strategy;

[0126] S74, according to the dynamically adjusted processing strategy S proc (t),Optimize resource allocation in the data center;

[0127] S75. Generate real-time processing result set Q real-time = {Q1,Q2,…,Q n}, each result Q j It contains key data analysis, forecast information and business indicators that have been processed in real time, and is transmitted to the corresponding business modules through the virtualized data interface I, supporting enterprises' instant decision-making, intelligent analysis and flexible response in a multi-cloud environment.

[0128] In this implementation manner, S8 specifically includes:

[0129] S81, based on real-time processing result set Q real-time = {Q1,Q2,…,Q n}, perform differential privacy processing on key data analysis results, prediction information and business indicators;

[0130] S82. Introduce the differential privacy enhancement algorithm to generate a privacy-protected analysis data set Q by adding random noise N(∈) during the data analysis process private ={Q′1,Q′2,…,Q′n}, where Q j ′=Q j +N(∈) represents the jth analysis result after adding noise. The random noise N(∈) is controlled by the privacy budget parameter ∈, which adjusts the balance between privacy protection strength and data accuracy.

[0131] S83. The generation of noise N(∈) is based on the following formula, which adjusts the noise by balancing privacy protection and data accuracy in high-dimensional space:

[0132]

[0133] in, is the standard differential privacy noise generated based on Laplace distribution, Δf is the global sensitivity of the query function f, ∈ is the privacy budget, η is the adjustment coefficient for controlling the high-dimensional noise term, represents the second-order derivative matrix of the loss function L with respect to the data set Q, β is the regularization parameter, and R is a positive definite matrix in the high-dimensional space;

[0134] S84. For the data set Q processed by differential privacy private Conduct data analysis so that the analysis results can support effective business decisions and data applications while protecting privacy;

[0135] S85. After completing the data analysis, evaluate the effect of differential privacy processing and optimize the balance between privacy protection and data analysis accuracy by adjusting the privacy budget parameter ∈ and the noise generation mechanism.

[0136] Embodiment 1:

[0137] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a large multinational enterprise with branches in multiple regions around the world. The data of these branches are stored on different cloud platforms, including AWS, Azure and Google Cloud. The data structure of each branch is different. Some data are structured, such as customer information databases; some are semi-structured, such as log files; and some are unstructured, such as video and image files. With the expansion of business, the enterprise faces problems such as data dispersion, data silos, complex data management, and data security and privacy protection. It is unable to fully utilize data resources from various places for unified analysis and decision-making, which affects the efficient operation of the business and the accuracy of decision-making.

[0138] In order to solve the above problems, the enterprise decided to adopt the enterprise cloud asset global data integration management method based on the data middle platform proposed in the present invention. During the implementation process, the various data sources of each branch are first automatically identified and classified in a multi-cloud environment through an evolutionary data source identification algorithm. Specifically, the algorithm generates an initial feature vector by preliminarily scanning each data source, and continuously optimizes the data source identification and classification strategy through the natural selection process, and finally generates a dynamic data source directory. This process identified more than 200 data sources worldwide and successfully classified these data sources, covering structured, semi-structured and unstructured data types.

[0139] Based on the generated dynamic data source catalog, adaptive data access strategies are automatically generated for each type of data source. These strategies are customized according to the characteristics of each data source to ensure that different types of data can be seamlessly accessed to the data center. In order to further improve the efficiency and accuracy of the access process, a heterogeneous ensemble learning algorithm is also introduced to dynamically select the optimal model combination to optimize the data access strategy. In actual applications, the access time of all data sources is reduced by more than 30% compared with traditional methods, and the error rate in the data access process is reduced by nearly 50%. In the process of data transmission and governance, the optimal transmission path is dynamically selected for data transmission through the optimized management of cross-cloud data flows. Combined with the hyperdimensional evolutionary algorithm, the transmitted data is dynamically governed, including data cleaning, deduplication, format conversion and standardization, to generate high-quality governed data sets. Taking the data of North American and European branches as an example, the data quality of the governed data set has been improved by 20% and data redundancy has been reduced by 25%. In addition, the efficiency of data transmission has also been significantly improved, and the transmission time of cross-cloud data has been shortened by about 40%.

[0140] In terms of data virtualization management, a cross-cloud data virtualization layer is built using the governed data sets, and a unified virtualized data interface is generated. Through this interface, enterprises can achieve transparent management and unified access to data resources distributed on different cloud platforms. This function greatly simplifies the data management process of enterprises, especially when data needs to be retrieved across departments and regions, greatly improving the speed and accuracy of data access. For example, when analyzing global sales data, enterprises can retrieve sales data from all over the world in just a few minutes through a virtualized data interface, while it used to take hours to manually integrate this data.

[0141] The differential privacy enhancement algorithm was introduced in the data analysis process to ensure data privacy while still providing valuable analysis results. In multiple links of data analysis, by adding noise during the analysis process, personal privacy is effectively protected and the leakage of sensitive information is avoided. In practical applications, the algorithm reduces the risk of privacy leakage of customer information by 50%, while the accuracy of data analysis only drops by less than 5%, which shows that a good balance has been found between privacy protection and data accuracy.

[0142] In North American branches, the time for identifying and classifying various data sources was shortened from 48 hours to 12 hours through the above method, and the success rate of data access was increased from 85% to 97%. In European branches, data quality after data governance was improved by about 20%, data transmission efficiency was improved by 40%, and the monthly cloud storage cost savings reached about US$10,000. In addition, globally, while implementing differential privacy protection, the availability of data analysis was guaranteed, customer privacy leaks were reduced by 50%, and the accuracy of data analysis results was only reduced by less than 5%, with negligible impact on business decisions.

[0143] Table 1 Comparison of data source identification and classification effects

[0144]

[0145] Table 2 Comparison of data transmission and governance effects

[0146] area Improved data transmission efficiency Improved data quality Reduced data redundancy North America 40％ 20％ 25％ Europe 38％ 18％ 22％ Asia Pacific 42％ 21％ 27％ South America 39％ 19％ 24％ average 39.75％ 19.5％ 24.5％

[0147] Table 3 Comparison of data security and privacy protection effects

[0148]

[0149]

[0150] According to the data in the table, the evolutionary data source identification algorithm of the present invention has significantly improved the efficiency of data source identification. In the four regions of North America, Europe, Asia Pacific and South America, the traditional method takes an average of 48 hours to complete the identification of the data source, while the method of the present invention shortens this time to 12 hours, shortening the overall time by 75%. This shows that the present invention can complete the automatic identification and classification of data sources more quickly in a multi-cloud environment, greatly improving the efficiency of data management. In terms of data transmission and governance, the innovative algorithm of the present invention also shows obvious advantages. By applying the hyperdimensional evolutionary algorithm, the present invention improves the data transmission efficiency by 39.75%, the data quality by 19.5%, and the data redundancy by 24.5%. These results show that the present invention can not only optimize the speed of data transmission, but also improve the quality of data and reduce redundancy in the process of data governance, so as to better support the data management needs of enterprises. In terms of data security and privacy protection, the differential privacy enhancement algorithm of the present invention effectively reduces the incidence of customer privacy leakage incidents from an average of 20 to 10 per year, reducing the risk of leakage by 50%. At the same time, although privacy protection measures were taken, the accuracy of data analysis only dropped by 5%, which shows that the present invention has achieved a good balance between privacy protection and data accuracy, and can provide valuable analysis results while ensuring privacy. These data show that the present invention has significant improvements in data identification and classification, data transmission and governance, and data security and privacy protection, and can effectively solve the data management challenges of enterprises in a multi-cloud environment.

[0151] To sum up, in enterprise data management in a multi-cloud environment, the present invention solves multiple key problems in the prior art by automatically identifying and classifying data sources, optimizing data access and governance strategies, improving data transmission efficiency and security, and realizing data virtualization management and privacy protection, thereby significantly improving the enterprise's data management efficiency and business decision-making capabilities.

[0152] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. The enterprise cloud asset global data integration management method based on the data middle platform is characterized by: The steps include: S1. Use an evolutionary data source identification algorithm to automatically identify and classify various data sources of an enterprise in a multi-cloud environment, generate an initial feature vector set, continuously optimize the data source identification and classification strategy by simulating the natural selection process, and finally generate a dynamic data source directory; S2. Based on the generated dynamic data source directory, generate adaptive data access strategies according to the characteristics of different data sources, realize seamless access of structured, semi-structured and unstructured data to the data center, and synchronize and update data across cloud platforms to form a unified access data set; S3. In the data center, use the access data set to build a unified data model for the enterprise, and use semantic mapping technology to map data with different structures and sources in the multi-cloud environment into a unified data model; S4. Based on the constructed data model, we implement optimized management of cross-cloud data flows, dynamically adjust data transmission paths, form an optimal path set, select the optimal path based on real-time business needs and data characteristics, and perform stable data transmission; S5. Use the optimal path set for data transmission, dynamically manage the received data through adaptive strategies, form a managed data set, and flexibly adjust the management process according to business priorities; S6. Use the governed data sets to build a cross-cloud data virtualization layer. Use data virtualization technology to present data resources on different cloud platforms in a unified manner, generate virtualized data interfaces, provide unified access to cross-cloud data, and support transparent management and access of data. S7. In the data center, virtualized data interfaces are used to adjust processing strategies in real time, optimize resource allocation, and generate real-time processing result sets in combination with dynamic changes in data flows during real-time data processing; S8. Based on the real-time processing result set, a differential privacy enhancement algorithm is introduced to perform data analysis in the global data security framework. By adding noise in the data analysis process, personal privacy is protected and sensitive information leakage is avoided. At the same time, the analytical value of the data is maintained to achieve ultimate data security and compliance management.

2. The enterprise cloud asset global data integration management method based on the data middle platform according to claim 1 is characterized in that: The S1 specifically includes: S11. Use the evolutionary data source identification algorithm to perform preliminary scanning and identification of various enterprise data sources in a multi-cloud environment to generate an initial feature vector V i =(v i1 ,v i2 ,…,v in ), where v ij Represents the jth characteristic attribute of the i-th data source; S12, based on the initial feature vector V i , using the evolutionary data source identification algorithm to perform cluster analysis and generate the initial classification result set C = {C1, C2, ..., C m }, where C k Represents the k-th data source: Among them, S(V i ,V j ) represents the feature vector V i and V j The similarity between S13, optimize the initial classification results by simulating the natural selection process, including selection, crossover and mutation operations. The selection process is based on the fitness function F(C k )Evaluate each type of data source: Among them, |C k |Indicates category C k The number of data sources in S(V i ,V j ) is the similarity value; S14. Generate a dynamic data source directory D = {D1, D2, ..., D q }, where D i Represents the dynamic attribute set of the i-th type of data source, including the data source location, update cycle, and data type attributes, ultimately forming a dynamically adjustable data source directory.

3. The enterprise cloud asset global data integration management method based on the data middle platform according to claim 1 is characterized in that: The S2 specifically includes: S21, based on the generated dynamic data source directory D = {D1, D2, ..., D q }, where D i represents the dynamic attribute set of the i-th data source. For different types of data source attributes, an initial access parameter set P = {P1, P2, ..., P q }, each access parameter includes data access protocol, data transmission rate and access frequency information related to the corresponding data source; S22. Based on the access parameter set P, generate an adaptive data access policy set S = {S1, S2, ..., S q }, each access strategy dynamically adjusts the access frequency, data transmission rate and protocol selection according to the changing characteristics of the data source to optimize the access process; S23. Introduce heterogeneous ensemble learning algorithms to integrate multiple heterogeneous models in a multi-cloud environment and dynamically select the optimal model combination to optimize data access strategies: Among them, S i (t) is the access strategy of the i-th data source at time t, M m (X i ) is the mth heterogeneous model for the input data X i The output, w m is the weight of the model, α is the learning rate, is the loss gradient, β is the gradient learning rate, R(S i ) is the regularization term, γ is the regularization weight; S24. During the access strategy execution process, each data source is monitored in real time, and the access parameter set P is dynamically adjusted through the feedback control mechanism. i =P i +ΔP real-time update parameters, where ΔP represents the adjustment amount based on monitoring feedback; S25. Use adaptive data access strategies to achieve seamless access to structured, semi-structured, and unstructured data, access various types of data to the data center through standardized interfaces, and synchronize and update data across cloud platforms based on the dynamic attributes of the data source to form a unified access data set C unified .

4. The enterprise cloud asset global data integration management method based on the data middle platform according to claim 1 is characterized in that: The S5 specifically includes: S51, based on the generated optimal path set R = {R1, R2, ..., R n }, select path R i For data transmission, each path R i Dynamic selection based on the characteristics of the data source and real-time business needs; S52, receiving data transmitted through the optimal path set R, and performing preliminary analysis to generate an original data set in Represents the jth received original data; S53, applying the adaptive strategy to the original data set O raw Perform dynamic governance, including data cleaning, deduplication, format conversion, and standardization. The governed data set is represented as S54. Introduce hyperdimensional evolutionary algorithm to optimize governance strategies in high-dimensional space: Among them, θ * , * , σ * They represent the optimal solutions for cleaning parameters, deduplication strategies, format conversion rules, and standardization specifications, respectively. is the loss function between the governed dataset and the target dataset, α i is the weight of loss for different data sets, is the regularization term, β is the regularization weight; S55. In the governance process, based on business priorities Dynamically adjust the governance process. The optimized governance process is updated by the formula G(t+1)=G(t)+ΔG, where ΔG represents the increment adjusted according to the business priority; S56, the generated governance data set O clean Input into the storage and processing module of the data center as the basic data support for data analysis, governance and business decision-making.

5. The enterprise cloud asset global data integration management method based on data middle platform according to claim 1 is characterized in that: The S6 specifically includes: S61. Using Governed Datasets Build a cross-cloud data virtualization layer to virtualize data resources on different cloud platforms and represent each data source as a virtualized data resource set V = {V1, V2, ..., V n }, where V i represents the i-th virtualized data resource; S62, based on the virtualized data resource set V, unified management is achieved through data virtualization technology, and a virtualized data interface I = {I1, I2, ..., I k }, where I j represents the jth virtualized data interface, each interface I j Contains data access paths and access control policies across cloud platforms; S63: Introduce a path optimization algorithm to determine the optimal access path P for each virtualized data interface. opt : Among them, P opt (I j ) represents the optimal access path of the jth interface, γ l is the weight associated with path node l, T l (P) is the access time through node l on path P, C(P) is the total access cost of path P, and δ is the weight coefficient of the path cost; S64. The generated virtualized data interface I provides unified data access across cloud platforms and supports transparent management and access of data resources; S65. Deploy the virtualized data interface I on the data middle platform so that data resources across cloud platforms can be accessed through a unified interface.

6. The enterprise cloud asset global data integration management method based on data middle platform according to claim 1 is characterized in that: The S7 specifically includes: S71, in the data center, using the virtualized data interface I = {I1, I2, ..., I k }, receiving and processing real-time data streams from different cloud platforms F = {F1, F2, ..., F m }, where F i Represents the i-th real-time data stream; S72: After receiving the real-time data stream, the data stream is initially parsed and preprocessed through the virtualized data interface to generate a preprocessed data stream set. Each data stream After standardization; S73, combined with the pre-processed data stream set F pre The dynamic change characteristics of the data flow are monitored in real time. i (t), data flow v i (t), data delay d i (t), and data volatility ω i (t), dynamically adjust the processing strategy S proc (t): Among them, S proc (t) represents the processing strategy at time t, α is the adjustment step size, γ i , i , μ i , i are the weight coefficients related to data flow rate, traffic, delay and volatility, respectively. represents the gradient calculation of the processing strategy; S74, according to the dynamically adjusted processing strategy S proc (t),Optimize resource allocation in the data center; S75. Generate real-time processing result set Q real-time = {Q1,Q2,…,Q n }, each result Q j It contains key data analysis, forecast information and business indicators that have been processed in real time, and is transmitted to the corresponding business modules through the virtualized data interface I, supporting enterprises' instant decision-making, intelligent analysis and flexible response in a multi-cloud environment.

7. The enterprise cloud asset global data integration management method based on data middle platform according to claim 1 is characterized in that: The S8 specifically includes: S81, based on real-time processing result set Q real-time = {Q1,Q2,…,Q n }, perform differential privacy processing on key data analysis results, prediction information and business indicators; S82. Introduce the differential privacy enhancement algorithm to generate a privacy-protected analysis data set Q by adding random noise N(∈) during the data analysis process private ={Q′1,Q′2,…,Q′ n }, where Q j ′=Q j +N(∈) represents the jth analysis result after adding noise. The random noise N(∈) is controlled by the privacy budget parameter ∈, which adjusts the balance between privacy protection strength and data accuracy. S83. The generation of noise N(∈) is based on the following formula, which adjusts the noise by balancing privacy protection and data accuracy in high-dimensional space: in, is the standard differential privacy noise generated based on Laplace distribution, Δf is the global sensitivity of the query function f, ∈ is the privacy budget, η is the adjustment coefficient for controlling the high-dimensional noise term, represents the second-order derivative matrix of the loss function L with respect to the data set Q, β is the regularization parameter, and R is a positive definite matrix in the high-dimensional space; S84. For the data set Q processed by differential privacy private Conduct data analysis so that the analysis results can support effective business decisions and data applications while protecting privacy; S85. After completing the data analysis, evaluate the effect of differential privacy processing and optimize the balance between privacy protection and data analysis accuracy by adjusting the privacy budget parameter ∈ and the noise generation mechanism.

Images