Heterogeneous data integration and security maintenance method and system based on cloud storage and medium

Abstract

The invention provides a heterogeneous data integration and security maintenance method and system based on cloud storage and a medium. The method comprises the steps of obtaining heterogeneous data, performing standardization processing, labeling and classification, storing the standardized heterogeneous data in a cloud storage system, constructing a global index, performing dynamic partition management, associating the standardized heterogeneous data, providing a data sharing interface, and dynamically allocating access authority. The standardized heterogeneous data are encrypted and authenticated through identity authentication and access control, the incremental data are subjected to remote cloud backup and recovery, the standardized heterogeneous data are subjected to abnormal behavior detection, an operation log calculation hash value is stored and linked, and potential risk users are identified and isolated; therefore, efficient integration and security maintenance of heterogeneous data are realized through a unified data model, distributed storage management and a multilayer security mechanism, and the efficiency and security of data sharing and application are improved.

Description

Technical Field

[0001] The present application relates to the technical fields of cloud computing and big data processing, and more specifically, to a method, system, and medium for integrating and securely maintaining heterogeneous data based on cloud storage. Background Art

[0002] With the rapid development of information technology, data volumes are exploding, and data sources are becoming increasingly diverse, resulting in a vast amount of heterogeneous data. The construction of the national cultural big data system includes multi-source heterogeneous data, including cultural heritage specimen libraries, cultural gene libraries, and cultural material libraries. The integration and security maintenance of heterogeneous data face challenges such as inconsistent data formats, fragmented data storage, and difficulty ensuring data security, resulting in inefficient data sharing and integration.

[0003] In response to the above problems, effective technical solutions are urgently needed. Summary of the Invention

[0004] The purpose of this application is to provide a method, system and medium for heterogeneous data integration and security maintenance based on cloud storage, which can achieve efficient integration and security maintenance of heterogeneous data through a unified data model, distributed storage management and multi-layer security mechanism, and improve the efficiency and security of data sharing and application.

[0005] This application also provides a method for integrating and maintaining heterogeneous data based on cloud storage, which includes the following steps:

[0006] Acquire heterogeneous data and preprocess it, standardize, label, and classify the preprocessed heterogeneous data;

[0007] Store standardized heterogeneous data into cloud storage systems, build global indexes, and perform dynamic partition management;

[0008] Associating the standardized heterogeneous data, providing a data sharing interface and dynamically allocating data access rights based on data tags;

[0009] Encrypt the standardized heterogeneous data, authenticate the legality of access identities through identity authentication and access control, and perform off-site cloud backup and recovery of incremental data;

[0010] Abnormal behavior detection is performed on the standardized heterogeneous data and the recorded operation logs are uploaded to the chain to identify and isolate potential risk users.

[0011] Optionally, in the cloud storage-based heterogeneous data integration and security maintenance method described in the present application, the obtaining of heterogeneous data and preprocessing, and standardization, labeling, and classification of the preprocessed heterogeneous data include:

[0012] Obtain heterogeneous data through a preset cultural data gene library;

[0013] Performing redundancy removal, repair and cleaning pre-processing on the heterogeneous data;

[0014] Standardize the pre-processed heterogeneous data through a preset data conversion tool to obtain standardized heterogeneous data;

[0015] The standardized heterogeneous data are labeled and classified according to preset metadata tags.

[0016] Optionally, in the cloud storage-based heterogeneous data integration and security maintenance method described in this application, storing standardized heterogeneous data in the cloud storage system, building a global index, and performing dynamic partition management includes:

[0017] Storing the standardized heterogeneous data in a cloud storage system using a distributed storage architecture;

[0018] Dividing the standardized heterogeneous data into hot data and cold data according to access frequency;

[0019] Build a multi-dimensional global index through preset search and analysis engines;

[0020] The data growth trend of the standardized heterogeneous data is checked, and the storage nodes of the standardized heterogeneous data are automatically expanded to perform dynamic partition management.

[0021] Optionally, in the cloud storage-based heterogeneous data integration and security maintenance method described in this application, associating the standardized heterogeneous data, providing a data sharing interface, and dynamically allocating data access rights based on data tags include:

[0022] By associating standardized heterogeneous data through a pre-set data management system, a logically unified resource pool is formed;

[0023] Provide data sharing through preset protocol interface;

[0024] Dynamically assign data access permissions based on data tags through a preset access control model.

[0025] Optionally, in the cloud storage-based heterogeneous data integration and security maintenance method described in the present application, encrypting the standardized heterogeneous data, performing access identity authentication through identity authentication and access control, and performing off-site cloud backup and recovery of incremental data include:

[0026] Encrypting the standardized heterogeneous data by using a preset multi-layer encryption model;

[0027] Perform user identity authentication for data access users through preset multi-factors;

[0028] Legal verification of data access users through pre-set policy isolation access methods;

[0029] The data is detected incrementally during the period, and the incremental data is stored and backed up through the preset off-site multi-cloud, and point-in-time recovery is performed based on data classification and storage cycle.

[0030] Optionally, in the cloud storage-based heterogeneous data integration and security maintenance method described in this application, the abnormal behavior detection of the standardized heterogeneous data and the on-chain processing of the recorded operation logs, and the identification and isolation of potential risk users, include:

[0031] Performing real-time behavior monitoring on the standardized heterogeneous data through a preset system monitoring model;

[0032] Detect abnormal behavior through the preset stream processing engine;

[0033] Record data access operation logs and calculate hash values ​​for storage on-chain;

[0034] Potential access threats are processed through preset anomaly detection models, and risky users are isolated and alerted.

[0035] In a second aspect, the present application provides a cloud storage-based heterogeneous data integration and security maintenance system, the system comprising: a memory and a processor, the memory comprising a program for a cloud storage-based heterogeneous data integration and security maintenance method, the program for the cloud storage-based heterogeneous data integration and security maintenance method, when executed by the processor, implementing the following steps:

[0036] Acquire heterogeneous data and preprocess it, standardize, label, and classify the preprocessed heterogeneous data;

[0037] Store standardized heterogeneous data into cloud storage systems, build global indexes, and perform dynamic partition management;

[0038] Associating the standardized heterogeneous data, providing a data sharing interface and dynamically allocating data access rights based on data tags;

[0039] Encrypt the standardized heterogeneous data, authenticate the legality of access identities through identity authentication and access control, and perform off-site cloud backup and recovery of incremental data;

[0040] Abnormal behavior detection is performed on the standardized heterogeneous data and the recorded operation logs are uploaded to the chain to identify and isolate potential risk users.

[0041] Optionally, in the cloud storage-based heterogeneous data integration and security maintenance system described in the present application, the acquiring and preprocessing of heterogeneous data, and the standardization, labeling, and classification of the preprocessed heterogeneous data include:

[0042] Obtain heterogeneous data through a preset cultural data gene library;

[0043] Performing redundancy removal, repair and cleaning pre-processing on the heterogeneous data;

[0044] Standardize the pre-processed heterogeneous data through a preset data conversion tool to obtain standardized heterogeneous data;

[0045] The standardized heterogeneous data are labeled and classified according to preset metadata tags.

[0046] Optionally, in the cloud storage-based heterogeneous data integration and security maintenance system described in this application, storing standardized heterogeneous data in the cloud storage system, building a global index, and performing dynamic partition management includes:

[0047] Storing the standardized heterogeneous data in a cloud storage system using a distributed storage architecture;

[0048] Dividing the standardized heterogeneous data into hot data and cold data according to access frequency;

[0049] Build a multi-dimensional global index through preset search and analysis engines;

[0050] The data growth trend of the standardized heterogeneous data is checked, and the storage nodes of the standardized heterogeneous data are automatically expanded to perform dynamic partition management.

[0051] In a third aspect, the present application also provides a computer-readable storage medium, which stores a cloud storage-based heterogeneous data integration and security maintenance method program. When the cloud storage-based heterogeneous data integration and security maintenance method program is executed by a processor, the steps of the cloud storage-based heterogeneous data integration and security maintenance method as described in any one of the above items are implemented.

[0052] From the above, it can be seen that the cloud storage-based heterogeneous data integration and security maintenance method, system and medium provided in this application obtain heterogeneous data and perform standardized processing, labeling and classification, store standardized heterogeneous data in the cloud storage system, build a global index and perform dynamic partition management, associate standardized heterogeneous data, provide a data sharing interface and dynamically allocate access rights, encrypt standardized heterogeneous data and authenticate it through identity authentication and access control, perform off-site cloud backup and recovery of incremental data, detect abnormal behavior of standardized heterogeneous data and calculate hash values ​​of operation logs and store them on the chain, identify and isolate potential risk users; thereby, through a unified data model, distributed storage management, and multi-layer security mechanism, it supports complex heterogeneous data environments such as the national cultural big data system and enterprise multi-source data platforms, realizes efficient integration and security maintenance of heterogeneous data, and improves the efficiency and security of data sharing and application.

[0053] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or understood by practicing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 A flowchart of a cloud storage-based heterogeneous data integration and security maintenance method provided in an embodiment of the present application;

[0056] Figure 2 A flowchart of heterogeneous data collection and processing for the cloud storage-based heterogeneous data integration and security maintenance method provided in an embodiment of the present application;

[0057] Figure 3 A flowchart of heterogeneous data storage and management for a cloud storage-based heterogeneous data integration and security maintenance method provided in an embodiment of the present application;

[0058] Figure 4 A flowchart of heterogeneous data integration and sharing of a cloud storage-based heterogeneous data integration and security maintenance method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0060] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0061] Please refer to Figure 1 , Figure 1 This is a flow chart of a cloud storage-based heterogeneous data integration and security maintenance method in some embodiments of the present application. This cloud storage-based heterogeneous data integration and security maintenance method is used in terminal devices, such as computers and mobile terminals. This cloud storage-based heterogeneous data integration and security maintenance method includes the following steps:

[0062] S11. Obtain and preprocess heterogeneous data, and standardize, label, and classify the preprocessed heterogeneous data;

[0063] S12. Store standardized heterogeneous data into the cloud storage system, build a global index, and perform dynamic partition management;

[0064] S13, associating the standardized heterogeneous data, providing a data sharing interface and dynamically allocating data access rights based on data tags;

[0065] S14. Encrypt the standardized heterogeneous data, authenticate the legality of the access identity through identity authentication and access control, and perform off-site cloud backup and recovery of the incremental data;

[0066] S15. Detect abnormal behavior on the standardized heterogeneous data and upload the recorded operation logs to the chain to identify and isolate potential risk users.

[0067] This involves collecting, preprocessing, storing, and integrating heterogeneous data from various cultural institutions to form a unified cultural big data resource pool. Data security is ensured through encryption, identity authentication, and access control mechanisms. Data oversight and auditing mechanisms ensure timely detection and resolution of data security incidents, ensuring the secure and compliant use of data.

[0068] Please refer to Figure 2 , Figure 2 This is a flow chart of heterogeneous data collection and processing for a cloud storage-based heterogeneous data integration and security maintenance method in some embodiments of the present application. According to an embodiment of the present invention, the acquisition and preprocessing of heterogeneous data, and the standardization, labeling, and classification of the preprocessed heterogeneous data include:

[0069] S21. Obtain heterogeneous data through a preset cultural data gene library;

[0070] S22, performing redundancy removal, repairing, and cleaning preprocessing on the heterogeneous data;

[0071] S23, standardizing the pre-processed heterogeneous data using a preset data conversion tool to obtain standardized heterogeneous data;

[0072] S24: Label and classify the standardized heterogeneous data according to preset metadata tags.

[0073] Among them, various types of heterogeneous data are obtained from the preset cultural data gene library through API gateways, Kafka collection tools, etc. The cultural data gene library is the source database of various types of heterogeneous data. Then, according to the preset rule engine, various types of heterogeneous data are pre-processed by de-duplication, repair and cleaning to obtain the repaired heterogeneous data. Then, the heterogeneous data is standardized through the preset data conversion tool (ETL tool) to obtain a standardized format or standardized model, such as XML, JPEG. Then, according to the classification rules of the preset metadata tags, such as data source, type, encryption level, etc., the standardized heterogeneous data is labeled and classified to build a dynamic metadata database.

[0074] Please refer to Figure 3 , Figure 3 This is a flow chart of heterogeneous data storage and management of a cloud storage-based heterogeneous data integration and security maintenance method in some embodiments of the present application. According to an embodiment of the present invention, storing standardized heterogeneous data in a cloud storage system, building a global index, and performing dynamic partition management includes:

[0075] S31, storing the standardized heterogeneous data in a cloud storage system using a distributed storage architecture;

[0076] S32, dividing the standardized heterogeneous data into hot data and cold data according to access frequency;

[0077] S33. Build a multi-dimensional global index through a preset search and analysis engine;

[0078] S34: Check the data growth trend of the standardized heterogeneous data, and automatically expand the storage nodes of the standardized heterogeneous data to perform dynamic partition management.

[0079] Among them, a hybrid cloud distributed storage architecture is adopted to store hot data in high-performance SSDs and archive cold data in low-cost object storage. Multi-dimensional indexes are built for distributed stored data based on Elasticsearch. On the basis of dynamic partition management, service discovery and load balancing are achieved through preset open source tools such as Consul.

[0080] Please refer to Figure 4 , Figure 4 This is a flow chart of heterogeneous data integration and sharing in a cloud storage-based heterogeneous data integration and security maintenance method in some embodiments of the present application. According to an embodiment of the present invention, associating the standardized heterogeneous data, providing a data sharing interface, and dynamically allocating data access rights based on data tags include:

[0081] S41. Associating standardized heterogeneous data through a pre-set data management system to form a logically unified resource pool;

[0082] S42. Providing data sharing through a preset protocol interface;

[0083] S43. Dynamically assign data access rights based on data tags through a preset access control model.

[0084] Among them, heterogeneous data are associated through data management systems such as Apache Atlas' data virtualization technology, and different interfaces such as RESTful API and OData protocol interface are set to adapt to the access needs of different user scenarios. Then, through preset access control models such as attribute access model or role access model, fine-grained permission policies are dynamically generated according to data tags, and data access rights are dynamically allocated.

[0085] According to an embodiment of the present invention, encrypting the standardized heterogeneous data, performing legal authentication of access identity through identity authentication and access control, and performing off-site cloud backup and recovery of incremental data include:

[0086] Encrypting the standardized heterogeneous data by using a preset multi-layer encryption model;

[0087] Perform user identity authentication for data access users through preset multi-factors;

[0088] Legal verification of data access users through pre-set policy isolation access methods;

[0089] The data is detected incrementally during the period, and the incremental data is stored and backed up through the preset off-site multi-cloud, and point-in-time recovery is performed based on data classification and storage cycle.

[0090] Among them, the preset multi-layer encryption model includes the use of AES-256 and SM4 dual algorithms to encrypt static data, and the use of TLS / SSL encryption channels for transmitted data. Memory computing uses homomorphic encryption, and the preset multi-factor uses integrated authorization protocols, hardware tokens and biometrics to authenticate the identity of accessing users in this solution embodiment. The policy isolation access method uses micro-isolation technology Tetration and access control policies to achieve minimum permissions for data access and verification of sensitive data access operations.

[0091] According to an embodiment of the present invention, the abnormal behavior detection on the standardized heterogeneous data and the recorded operation logs are uploaded to the chain, and the potential risk users are identified and isolated, including:

[0092] Performing real-time behavior monitoring on the standardized heterogeneous data through a preset system monitoring model;

[0093] Detect abnormal behavior through the preset stream processing engine;

[0094] Record data access operation logs and calculate hash values ​​for storage on-chain;

[0095] Potential access threats are processed through preset anomaly detection models, and risky users are isolated and alerted.

[0096] Among them, the access records of visiting users are collected through the preset system monitoring model, and combined with the preset stream processing engine such as Flink to detect abnormal behaviors such as high-frequency illegal access, the operation logs of visiting users including ID, operation type, and timestamp are hashed and stored in the blockchain. In the implementation of this solution, the hash value of the log is collected in real time from at least three independent nodes, and the hash value is uploaded to the chain to ensure that it cannot be tampered with. Then, the potential security threats of visiting users are identified and risky users are marked through anomaly detection models such as LSTM and isolation forest algorithm. At the same time, the firewall is linked to isolate the risky IP to achieve the integration and security management of heterogeneous data.

[0097] In a second aspect, the present invention further discloses a cloud storage-based heterogeneous data integration and security maintenance system, comprising a memory and a processor, wherein the memory comprises a cloud storage-based heterogeneous data integration and security maintenance method program, and when the cloud storage-based heterogeneous data integration and security maintenance method program is executed by the processor, the following steps are implemented:

[0098] Acquire heterogeneous data and preprocess it, standardize, label, and classify the preprocessed heterogeneous data;

[0099] Store standardized heterogeneous data into cloud storage systems, build global indexes, and perform dynamic partition management;

[0100] Associating the standardized heterogeneous data, providing a data sharing interface and dynamically allocating data access rights based on data tags;

[0101] Encrypt the standardized heterogeneous data, authenticate the legality of access identities through identity authentication and access control, and perform off-site cloud backup and recovery of incremental data;

[0102] Abnormal behavior detection is performed on the standardized heterogeneous data and the recorded operation logs are uploaded to the chain to identify and isolate potential risk users.

[0103] This involves collecting, preprocessing, storing, and integrating heterogeneous data from various cultural institutions to form a unified cultural big data resource pool. Data security is ensured through encryption, identity authentication, and access control mechanisms. Data oversight and auditing mechanisms ensure timely detection and resolution of data security incidents, ensuring the secure and compliant use of data.

[0104] According to an embodiment of the present invention, the obtaining and preprocessing of heterogeneous data, and the standardization, labeling, and classification of the preprocessed heterogeneous data include:

[0105] Obtain heterogeneous data through a preset cultural data gene library;

[0106] Performing redundancy removal, repair and cleaning pre-processing on the heterogeneous data;

[0107] Standardize the pre-processed heterogeneous data through a preset data conversion tool to obtain standardized heterogeneous data;

[0108] The standardized heterogeneous data are labeled and classified according to preset metadata tags.

[0109] Among them, various types of heterogeneous data are obtained from the preset cultural data gene library through API gateways, Kafka collection tools, etc. The cultural data gene library is the source database of various types of heterogeneous data. Then, according to the preset rule engine, various types of heterogeneous data are pre-processed by de-duplication, repair and cleaning to obtain the repaired heterogeneous data. Then, the heterogeneous data is standardized through the preset data conversion tool (ETL tool) to obtain a standardized format or standardized model, such as XML, JPEG. Then, according to the classification rules of the preset metadata tags, such as data source, type, encryption level, etc., the standardized heterogeneous data is labeled and classified to build a dynamic metadata database.

[0110] According to an embodiment of the present invention, storing standardized heterogeneous data in a cloud storage system, building a global index, and performing dynamic partition management includes:

[0111] Storing the standardized heterogeneous data in a cloud storage system using a distributed storage architecture;

[0112] Dividing the standardized heterogeneous data into hot data and cold data according to access frequency;

[0113] Build a multi-dimensional global index through preset search and analysis engines;

[0114] The data growth trend of the standardized heterogeneous data is checked, and the storage nodes of the standardized heterogeneous data are automatically expanded to perform dynamic partition management.

[0115] Among them, a hybrid cloud distributed storage architecture is adopted to store hot data in high-performance SSDs and archive cold data in low-cost object storage. Multi-dimensional indexes are built for distributed stored data based on Elasticsearch. On the basis of dynamic partition management, service discovery and load balancing are achieved through preset open source tools such as Consul.

[0116] According to an embodiment of the present invention, associating the standardized heterogeneous data, providing a data sharing interface, and dynamically allocating data access rights based on data tags includes:

[0117] By associating standardized heterogeneous data through a pre-set data management system, a logically unified resource pool is formed;

[0118] Provide data sharing through preset protocol interface;

[0119] Dynamically assign data access permissions based on data tags through a preset access control model.

[0120] Among them, heterogeneous data are associated through data management systems such as Apache Atlas' data virtualization technology, and different interfaces such as RESTful API and OData protocol interface are set to adapt to the access needs of different user scenarios. Then, through preset access control models such as attribute access model or role access model, fine-grained permission policies are dynamically generated according to data tags, and data access rights are dynamically allocated.

[0121] According to an embodiment of the present invention, encrypting the standardized heterogeneous data, performing legal authentication of access identity through identity authentication and access control, and performing off-site cloud backup and recovery of incremental data include:

[0122] Encrypting the standardized heterogeneous data by using a preset multi-layer encryption model;

[0123] Perform user identity authentication for data access users through preset multi-factors;

[0124] Legal verification of data access users through pre-set policy isolation access methods;

[0125] The data is detected incrementally during the period, and the incremental data is stored and backed up through the preset off-site multi-cloud, and point-in-time recovery is performed based on data classification and storage cycle.

[0126] Among them, the preset multi-layer encryption model includes the use of AES-256 and SM4 dual algorithms to encrypt static data, and the use of TLS / SSL encryption channels for transmitted data. Memory computing uses homomorphic encryption, and the preset multi-factor uses integrated authorization protocols, hardware tokens and biometrics to authenticate the identity of accessing users in this solution embodiment. The policy isolation access method uses micro-isolation technology Tetration and access control policies to achieve minimum permissions for data access and verification of sensitive data access operations.

[0127] According to an embodiment of the present invention, the abnormal behavior detection on the standardized heterogeneous data and the recorded operation logs are uploaded to the chain, and the potential risk users are identified and isolated, including:

[0128] Performing real-time behavior monitoring on the standardized heterogeneous data through a preset system monitoring model;

[0129] Detect abnormal behavior through the preset stream processing engine;

[0130] Record data access operation logs and calculate hash values ​​for storage on-chain;

[0131] Potential access threats are processed through preset anomaly detection models, and risky users are isolated and alerted.

[0132] Among them, the access records of visiting users are collected through the preset system monitoring model, and combined with the preset stream processing engine such as Flink to detect abnormal behaviors such as high-frequency illegal access, the operation logs of visiting users including ID, operation type, and timestamp are hashed and stored in the blockchain. In the implementation of this solution, the hash value of the log is collected in real time from at least three independent nodes, and the hash value is uploaded to the chain to ensure that it cannot be tampered with. Then, the potential security threats of visiting users are identified and risky users are marked through anomaly detection models such as LSTM and isolation forest algorithm. At the same time, the firewall is linked to isolate the risky IP to achieve the integration and security management of heterogeneous data.

[0133] The third aspect of the present invention provides a readable storage medium, which stores a cloud storage-based heterogeneous data integration and security maintenance method program. When the cloud storage-based heterogeneous data integration and security maintenance method program is executed by a processor, the steps of the cloud storage-based heterogeneous data integration and security maintenance method as described in any one of the above items are implemented.

[0134] The cloud storage-based heterogeneous data integration and security maintenance method, system and medium disclosed in the present invention acquire heterogeneous data and perform standardization processing, labeling and classification, store the standardized heterogeneous data in the cloud storage system, build a global index and perform dynamic partition management, associate the standardized heterogeneous data, provide a data sharing interface and dynamically allocate access rights, encrypt the standardized heterogeneous data and authenticate it through identity authentication and access control, perform off-site cloud backup and recovery of incremental data, detect abnormal behavior of standardized heterogeneous data and calculate hash values ​​of operation logs and store them on the chain, identify and isolate potential risk users; thereby, through a unified data model, distributed storage management, and a multi-layer security mechanism, support complex heterogeneous data environments such as the national cultural big data system and the enterprise multi-source data platform, achieve efficient integration and security maintenance of heterogeneous data, and improve the efficiency and security of data sharing and application.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0136] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0137] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0138] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0139] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A method for integrating and maintaining heterogeneous data based on cloud storage, characterized in that: The following steps are involved: Acquire heterogeneous data and preprocess it, standardize, label, and classify the preprocessed heterogeneous data; Store standardized heterogeneous data into cloud storage systems, build global indexes, and perform dynamic partition management; Associating the standardized heterogeneous data, providing a data sharing interface and dynamically allocating data access rights based on data tags; Encrypt the standardized heterogeneous data, authenticate the legality of access identities through identity authentication and access control, and perform off-site cloud backup and recovery of incremental data; Abnormal behavior detection is performed on the standardized heterogeneous data and the recorded operation logs are uploaded to the chain to identify and isolate potential risk users.

2. The cloud storage-based heterogeneous data integration and security maintenance method according to claim 1, characterized in that: The obtaining and preprocessing of heterogeneous data, and the standardization, labeling, and classification of the preprocessed heterogeneous data include: Obtain heterogeneous data through a preset cultural data gene library; Performing redundancy removal, repair and cleaning pre-processing on the heterogeneous data; Standardize the pre-processed heterogeneous data through a preset data conversion tool to obtain standardized heterogeneous data; The standardized heterogeneous data are labeled and classified according to preset metadata tags.

3. The cloud storage-based heterogeneous data integration and security maintenance method according to claim 2, characterized in that: Storing standardized heterogeneous data in a cloud storage system, building a global index, and performing dynamic partition management includes: Storing the standardized heterogeneous data in a cloud storage system using a distributed storage architecture; Dividing the standardized heterogeneous data into hot data and cold data according to access frequency; Build a multi-dimensional global index through preset search and analysis engines; The data growth trend of the standardized heterogeneous data is checked, and the storage nodes of the standardized heterogeneous data are automatically expanded to perform dynamic partition management.

4. The cloud storage-based heterogeneous data integration and security maintenance method according to claim 3, characterized in that: The associating the standardized heterogeneous data, providing a data sharing interface and dynamically allocating data access rights according to data tags includes: By associating standardized heterogeneous data through a pre-set data management system, a logically unified resource pool is formed; Provide data sharing through preset protocol interface; Dynamically assign data access permissions based on data tags through a preset access control model.

5. The cloud storage-based heterogeneous data integration and security maintenance method according to claim 4, characterized in that: The standardized heterogeneous data is encrypted, and access identity is legally authenticated through identity authentication and access control, and incremental data is backed up and restored in an off-site cloud, including: Encrypting the standardized heterogeneous data by using a preset multi-layer encryption model; Perform user identity authentication for data access users through preset multi-factors; Legal verification of data access users through pre-set policy isolation access methods; The data is detected incrementally during the period, and the incremental data is stored and backed up through the preset off-site multi-cloud, and point-in-time recovery is performed based on data classification and storage cycle.

6. The cloud storage-based heterogeneous data integration and security maintenance method according to claim 5, characterized in that: The abnormal behavior detection of the standardized heterogeneous data and the on-chain processing of the recorded operation logs, and the identification and isolation of potential risk users, include: Performing real-time behavior monitoring on the standardized heterogeneous data through a preset system monitoring model; Detect abnormal behavior through the preset stream processing engine; Record data access operation logs and calculate hash values ​​for storage on-chain; Potential access threats are processed through preset anomaly detection models, and risky users are isolated and alerted.

7. Heterogeneous data integration and security maintenance system based on cloud storage, characterized by: The system includes: a memory and a processor, wherein the memory includes a program of a cloud storage-based heterogeneous data integration and security maintenance method, and when the program of the cloud storage-based heterogeneous data integration and security maintenance method is executed by the processor, the following steps are implemented: Acquire heterogeneous data and preprocess it, standardize, label, and classify the preprocessed heterogeneous data; Store standardized heterogeneous data into cloud storage systems, build global indexes, and perform dynamic partition management; Associating the standardized heterogeneous data, providing a data sharing interface and dynamically allocating data access rights based on data tags; Encrypt the standardized heterogeneous data, authenticate the legality of access identities through identity authentication and access control, and perform off-site cloud backup and recovery of incremental data; Abnormal behavior detection is performed on the standardized heterogeneous data and the recorded operation logs are uploaded to the chain to identify and isolate potential risk users.

8. The cloud storage-based heterogeneous data integration and security maintenance system according to claim 7, characterized in that: The obtaining and preprocessing of heterogeneous data, and the standardization, labeling, and classification of the preprocessed heterogeneous data include: Obtain heterogeneous data through a preset cultural data gene library; Performing redundancy removal, repair and cleaning pre-processing on the heterogeneous data; Standardize the pre-processed heterogeneous data through a preset data conversion tool to obtain standardized heterogeneous data; The standardized heterogeneous data are labeled and classified according to preset metadata tags.

9. The cloud storage-based heterogeneous data integration and security maintenance system according to claim 8, characterized in that: Storing standardized heterogeneous data in a cloud storage system, building a global index, and performing dynamic partition management includes: Storing the standardized heterogeneous data in a cloud storage system using a distributed storage architecture; Dividing the standardized heterogeneous data into hot data and cold data according to access frequency; Build a multi-dimensional global index through preset search and analysis engines; The data growth trend of the standardized heterogeneous data is checked, and the storage nodes of the standardized heterogeneous data are automatically expanded to perform dynamic partition management.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a cloud storage-based heterogeneous data integration and security maintenance method program. When the cloud storage-based heterogeneous data integration and security maintenance method program is executed by a processor, the steps of the cloud storage-based heterogeneous data integration and security maintenance method as described in any one of claims 1 to 6 are implemented.

Images