Large model data protection method, system and equipment based on data grid

Abstract

The invention provides a large model data protection method, system and equipment based on a data grid, and relates to the technical field of model data processing. The large model data protection method based on the data grid comprises the steps of S1, data classification and marking, S2, access control and authority management, S3, data encryption, S4, data desensitization, S5, data backup and recovery and S6, monitoring and auditing. Through data classification and marking, sensitive data can be accurately identified, and different protection strategies are formulated according to the sensitive degree of the sensitive data, so that undifferentiated protection on all the data is avoided, resources are saved, and meanwhile, the protection strength on the data which really needs to be protected is emphatically enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of model data processing, and in particular to a large model data protection method, system and device based on data grid. Background Art

[0002] Since the 21st century, the rise of the Internet and mobile Internet has brought about an explosive growth in data volume and diverse data forms. Traditional business systems are unable to meet the needs of big data processing. Big data processing platforms such as Hadoop and Spark have emerged, and new databases such as HBase and MongoDB have also emerged to meet the challenges of big data storage and processing.

[0003] The birth of large basic models (hereinafter referred to as large models, such as language models such as GPT-2-XL and OPT-1.3B) has brought about technological changes and opportunities for all walks of life. However, the cost of training large models is high, and most companies cannot afford the cost of training a complete large model.

[0004] Currently, the owners of large models provide general large models to enterprises that have demand for large models. These enterprises use the privacy data in their respective fields to fine-tune the large models, thereby reducing the cost of using large models for enterprises. However, since large models usually require huge computing power and data, and the training cost is high, the trained large models are generally proprietary and not open source. If the owner of the large model provides the large model to enterprises that have demand for large models for fine-tuning, it is easy to damage the model intellectual property rights of the owner of the large model. If the privacy data of the field is sent to the owner of the large model for fine-tuning by enterprises that have demand for large models, it is easy to cause privacy data leakage. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a large model data protection method, system and device based on a data grid, which solves the problem of privacy data leakage.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A large model data protection method based on data grid, comprising:

[0007] S1: Data classification and labeling

[0008] Sensitive data identification: comprehensively sort out the massive data involved in the big model, identify the sensitive information contained in it based on the nature, source, content and other factors of the data, formulate a labeling strategy, and formulate corresponding labeling schemes for the data according to the different levels of sensitivity;

[0009] S2: Access Control and Rights Management

[0010] Role-based access control (RBAC) defines access rights for different user roles in a data grid environment, and dynamically adjusts user access rights based on the data processing process and scenario;

[0011] S3: Data Encryption

[0012] On the storage nodes of the data grid, the data is encrypted using an encryption algorithm. When the data is transmitted between different nodes of the data grid, or between the data supplier and the large model training platform, a secure transmission protocol (such as HTTPS) and encryption technology are used to prevent the data from being stolen or tampered with during transmission.

[0013] S4: Data desensitization

[0014] For different types of sensitive data, select appropriate desensitization algorithms, integrate the data desensitization process into the data processing workflow, and automatically perform desensitization processing before the data enters the large model training phase;

[0015] S5: Data backup and recovery

[0016] Create data copies in different storage nodes or geographical areas of the data grid, develop a comprehensive disaster recovery plan, and clearly define the recovery process and division of responsibilities when data is damaged (such as due to natural disasters, cyber attacks, etc.);

[0017] S6: Monitoring and Auditing

[0018] Establish a comprehensive data security monitoring system to monitor data access, storage, and transmission activities in the data grid in real time, and keep detailed audit records of data-related operations, including the time of operation, user, operation type, and data information involved.

[0019] Preferably, in step S3, encryption is performed using AES (Advanced Encryption Standard) in a symmetric encryption algorithm.

[0020] Preferably, in step S4, a suitable desensitizing technology is selected according to the data type, sensitivity and application scenario, and the sensitive data is encrypted using an encryption algorithm, and then decrypted when the data is needed.

[0021] Preferably, a large-model data protection system based on a data grid comprises: a robotic learning unit that uses machine learning technology to analyze access patterns and identify potential security threats;

[0022] Real-time monitoring unit, deploy monitoring tools, monitor data access behavior in real time, and detect abnormal activities in time;

[0023] Emergency response unit: Establish emergency response mechanism to quickly respond to and handle security incidents and reduce the risk of data leakage;

[0024] The backup unit regularly backs up the large model data to prevent data loss or damage. When data problems occur, the data can be quickly restored to ensure the normal operation of the large model;

[0025] The data transmission unit ensures the security of data transmission between various nodes in the data grid and prevents data from being stolen or tampered with during transmission.

[0026] Preferably, a large-model data protection device based on a data grid comprises: a multi-core processor, a multi-layer storage structure, a security chip, an encryption hardware acceleration module and a high-speed network interface. The multi-core processor selects a processor that supports advanced instruction sets that can accelerate specific encryption and data processing operations. The high-speed network interface device is equipped with an interface that supports multiple network protocols.

[0027] Preferably, the multi-layer storage structure includes: a cache and a main memory, wherein the cache is a high-speed, small-capacity memory located near the processor and is used to temporarily store data and instructions that the processor may frequently access in the near future, and the main memory uses a large-capacity, high-bandwidth memory.

[0028] Preferably, the security chip is a hardware component specially designed to protect sensitive information.

[0029] Preferably, the encryption hardware acceleration module is a circuit designed for a specific encryption algorithm (such as AES, RSA), which can quickly perform encryption and decryption operations at the hardware level.

[0030] The present invention provides a large model data protection method, system and equipment based on data grid.

[0031] It has the following beneficial effects:

[0032] 1. The present invention can accurately identify sensitive data through data classification and labeling, and formulate different protection strategies according to its sensitivity, which avoids indiscriminate protection of all data, saves resources, and at the same time focuses on strengthening the protection of data that really needs to be protected.

[0033] 2. The present invention starts from the two key links of data storage and transmission through storage encryption and transmission encryption, and combines multiple encryption algorithms (such as symmetric and asymmetric encryption algorithms) to provide high-intensity protection for data. Whether in static storage state or dynamic transmission process, it can effectively prevent data from being stolen or tampered with, and ensure the confidentiality and integrity of data.

[0034] 3. The present invention uses a robot learning unit and machine learning technology to analyze access patterns, identify potential security threats, and classify different types of data such as text, images, and audio. For text data, key information can be extracted and classified through lexical analysis, semantic understanding and other technologies. For image and audio data, deep learning algorithms can be used to extract features and classify them. It can adaptively learn new attack patterns and changes in normal behaviors, thereby more accurately identifying potential security threats. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a large model data protection method based on a data grid according to the present invention;

[0036] Figure 2 It is a schematic diagram of a large-model data protection system based on data grid according to the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Example:

[0039] like Figure 1-2 As shown, an embodiment of the present invention provides a large model data protection method based on a data grid, comprising:

[0040] S1: Data classification and labeling

[0041] Sensitive data identification: comprehensively sort out the massive data involved in the big model, identify the sensitive information contained in it based on the nature, source, content and other factors of the data, formulate a labeling strategy, and formulate corresponding labeling schemes for the data according to the different levels of sensitivity;

[0042] A multi-level labeling system can be used, such as labeling data into categories such as "highly sensitive", "moderately sensitive", "normally sensitive", etc., and attaching detailed instructions to each label so that the sensitive attributes of the data can be quickly and accurately identified during data processing;

[0043] S2: Access Control and Rights Management

[0044] Role-based access control (RBAC) defines access rights for different user roles in a data grid environment, and dynamically adjusts user access rights based on the data processing process and scenario;

[0045] During the model training phase, data engineers may need access to more data, but after the model is deployed, their access to training data can be appropriately reduced;

[0046] S3: Data Encryption

[0047] On the storage nodes of the data grid, the data is encrypted using an encryption algorithm. When the data is transmitted between different nodes of the data grid, or between the data supplier and the large model training platform, a secure transmission protocol (such as HTTPS) and encryption technology are used to prevent the data from being stolen or tampered with during transmission.

[0048] For highly sensitive data, a high-strength encryption method is used to ensure that the data cannot be easily decrypted even if the storage device is stolen;

[0049] S4: Data desensitization

[0050] For different types of sensitive data, select appropriate desensitization algorithms, integrate the data desensitization process into the data processing workflow, and automatically perform desensitization processing before the data enters the large model training phase;

[0051] Establish a quality assessment mechanism for desensitized data to ensure that the desensitized data can still meet the needs of large model training and that model performance will not deteriorate due to excessive desensitization;

[0052] S5: Data backup and recovery

[0053] Create data copies in different storage nodes or geographical areas of the data grid, develop a comprehensive disaster recovery plan, and clearly define the recovery process and division of responsibilities when data is damaged (such as due to natural disasters, cyber attacks, etc.);

[0054] Regularly conduct drills on the disaster recovery plan to ensure that data can be quickly and effectively restored in an emergency, reduce the impact of data loss on large models, and ensure data security during the recovery process to prevent new security risks from being introduced during the recovery operation;

[0055] S6: Monitoring and Auditing

[0056] Establish a comprehensive data security monitoring system to monitor data access, storage, and transmission activities in the data grid in real time, and conduct detailed audit records of data-related operations, including the time, user, type of operation, and data information involved;

[0057] By analyzing monitoring data, potential security threats can be discovered in a timely manner, such as abnormal data access patterns and sudden changes in data traffic.

[0058] In step S3, encryption is performed using AES (Advanced Encryption Standard) among symmetric encryption algorithms.

[0059] In step S4, a suitable desensitizing technology is selected according to the data type, sensitivity and application scenario, and the sensitive data is encrypted using an encryption algorithm, and then decrypted when the data is needed.

[0060] A large-model data protection system based on a data grid includes: a robot learning unit that uses machine learning technology to analyze access patterns and identify potential security threats;

[0061] Using machine learning algorithms and natural language processing technology, different types of data such as text, images, and audio can be automatically classified. For text data, key information can be extracted and classified through lexical analysis, semantic understanding and other technologies; for image and audio data, deep learning algorithms can be used to extract features and classify them;

[0062] Real-time monitoring unit, deploy monitoring tools, monitor data access behavior in real time, and detect abnormal activities in time;

[0063] Emergency response unit: Establish emergency response mechanism to quickly respond to and handle security incidents and reduce the risk of data leakage;

[0064] The backup unit regularly backs up the large model data to prevent data loss or damage. When data problems occur, the data can be quickly restored to ensure the normal operation of the large model;

[0065] Using the distributed storage characteristics of the data grid, data is backed up to multiple nodes to improve the reliability and availability of backup data;

[0066] Data transmission unit, which ensures the security of data transmission between various nodes of the data grid and prevents data from being stolen or tampered with during transmission;

[0067] Use secure transmission protocols, such as HTTPS, SSL / TLS, etc., to encrypt data for transmission. The data is encrypted before transmission and decrypted after the recipient receives the data.

[0068] A large-model data protection device based on a data grid, the large-model data protection device comprising: a multi-core processor, a multi-layer storage structure, a security chip, an encryption hardware acceleration module and a high-speed network interface, the multi-core processor selects a processor that supports advanced instruction sets, these instruction sets can accelerate specific encryption and data processing operations, and the high-speed network interface device is equipped with an interface that supports multiple network protocols;

[0069] Equipped with powerful multi-core processors to quickly handle high-intensity computing tasks such as data encryption, decryption, and access control verification, which is essential for real-time protection of large model data security and ensures that there will be no delays or performance bottlenecks during data transmission and use. For example, high-performance server-level processors such as Intel Xeon series or AMD EPYC series are used.

[0070] The multi-layer storage structure includes: cache and main memory. The cache is a high-speed, small-capacity memory located near the processor, which is used to temporarily store data and instructions that the processor may frequently access in the near future. The main memory uses a large-capacity, high-bandwidth memory.

[0071] Contains high-speed storage devices with large capacity, such as solid-state drives (SSDs) or high-speed hard disk arrays.

[0072] Security chips are hardware components designed specifically to protect sensitive information;

[0073] The security chip has hardware-level security protection, which can prevent the key from being illegally obtained or tampered with, providing a reliable foundation for data encryption.

[0074] The encryption hardware acceleration module is a circuit designed for specific encryption algorithms (such as AES, RSA) that can quickly perform encryption and decryption operations at the hardware level.

[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A large model data protection method based on data grid, characterized in that: include: S1: Data classification and labeling Sensitive data identification: comprehensively sort out the massive data involved in the big model, identify the sensitive information contained in it based on the nature, source, content and other factors of the data, formulate a labeling strategy, and formulate corresponding labeling schemes for the data according to the different levels of sensitivity; S2: Access Control and Rights Management Role-based access control (RBAC) defines access rights for different user roles in a data grid environment, and dynamically adjusts user access rights based on the data processing process and scenario; S3: Data Encryption On the storage nodes of the data grid, the data is encrypted using an encryption algorithm. When the data is transmitted between different nodes of the data grid, or between the data supplier and the large model training platform, a secure transmission protocol (such as HTTPS) and encryption technology are used to prevent the data from being stolen or tampered with during transmission; S4: Data desensitization For different types of sensitive data, select appropriate desensitization algorithms, integrate the data desensitization process into the data processing workflow, and automatically perform desensitization processing before the data enters the large model training phase; S5: Data backup and recovery Create data copies in different storage nodes or geographical areas of the data grid, develop a comprehensive disaster recovery plan, and clearly define the recovery process and division of responsibilities when data is damaged (such as due to natural disasters, cyber attacks, etc.); S6: Monitoring and Auditing Establish a comprehensive data security monitoring system to monitor data access, storage, and transmission activities in the data grid in real time, and keep detailed audit records of data-related operations, including the time of operation, user, operation type, and data information involved.

2. The large model data protection method based on data grid according to claim 1 is characterized by: In step S3, encryption is performed using AES (Advanced Encryption Standard) among symmetric encryption algorithms.

3. The large model data protection method based on data grid according to claim 1 is characterized by: In step S4, a suitable desensitizing technology is selected according to the data type, sensitivity and application scenario, and the sensitive data is encrypted using an encryption algorithm, and then decrypted when the data is needed.

4. A large model data protection system based on data grid, characterized by: include: A robotic learning unit that uses machine learning techniques to analyze access patterns and identify potential security threats; Real-time monitoring unit, deploy monitoring tools, monitor data access behavior in real time, and detect abnormal activities in time; Emergency response unit: Establish emergency response mechanism to quickly respond to and handle security incidents and reduce the risk of data leakage; The backup unit backs up the large model data regularly to prevent data loss or damage. When data problems occur, the data can be quickly restored to ensure the normal operation of the large model; The data transmission unit ensures the security of data transmission between various nodes in the data grid and prevents data from being stolen or tampered with during transmission.

5. A large-model data protection device based on a data grid, characterized in that: The large-model data protection device includes: a multi-core processor, a multi-layer storage structure, a security chip, an encryption hardware acceleration module and a high-speed network interface. The multi-core processor selects a processor that supports advanced instruction sets that can accelerate specific encryption and data processing operations. The high-speed network interface device is equipped with an interface that supports multiple network protocols.

6. The large-model data protection device based on data grid according to claim 5, characterized in that: The multi-layer storage structure includes: a cache and a main memory. The cache is a high-speed, small-capacity memory located near the processor and is used to temporarily store data and instructions that the processor may frequently access in the near future. The main memory uses a large-capacity, high-bandwidth memory.

7. The large-model data protection device based on data grid according to claim 5, characterized in that: The security chip is a hardware component specially designed to protect sensitive information.

8. The large-model data protection device based on data grid according to claim 5, characterized in that: The encryption hardware acceleration module is a circuit designed for a specific encryption algorithm (such as AES, RSA), which can quickly perform encryption and decryption operations at the hardware level.

Images