A cluster big data training process security checking and monitoring method

Abstract

This invention belongs to the field of data training and is a method for security inspection and monitoring of cluster big data training processes, including: S1: performing information reduction processing on the original training data, simultaneously extracting data risk features and binding them to the reduced data; S2: targeting and identifying dangerous data based on the risk feature labels, performing encryption processing on the dangerous data, encapsulating it, matching the transmission protocol, and completing the transmission of the encapsulated package; S3: parsing and decrypting the encrypted package, completing risk assessment and generating a processing strategy through a privately deployed risk processing decision-making big model; S4: distributing the processing strategy to the GPU training cluster and starting real-time security monitoring, collecting monitoring data and feeding it back to the previous module to generate optimization parameters; S5: iterating repeatedly until the convergence condition is met.

Description

Technical Field

[0001] This invention belongs to the field of data training, and in particular, it is a method for security inspection and monitoring of the training process of clustered big data. Background Technology

[0002] Data-driven technologies such as AI, NLP, and big data have become research hotspots in academia and industry. Their applications span multiple fields including biomedicine, education, and healthcare, while also placing higher demands on data and models, such as data security, model convergence, and data redundancy. While existing technologies are quite mature in data processing, vulnerabilities still exist in data security, including: (1) It is difficult to avoid malicious data attacks. Existing technologies lack security checks during data transmission and model training. Data models are vulnerable to malicious attacks and security vulnerabilities, which can cause deviations or crashes in training results.

[0003] (2) There is a lack of training process monitoring solutions. Existing technologies mainly target static data and known risks for prevention and control, but lack real-time monitoring and evaluation of the training process, and cannot promptly detect and respond to potential risks and problems.

[0004] (3) Data confidentiality is difficult to guarantee. Existing technologies often require data sharing in distributed computing and collaborative training. The lack of effective data encryption, isolation and access control can easily lead to data leakage and privacy violations.

[0005] (4) Insufficient technology adaptability. Existing technologies mainly process data through fixed rules and algorithms, lacking the ability to adapt to different business scenarios and models, and are difficult to meet personalized security needs.

[0006] Therefore, there is an urgent need to propose a security inspection and monitoring method for the training process of clustered big data in order to solve the above problems. Summary of the Invention

[0007] This invention proposes a method for security inspection and monitoring of the training process of clustered big data.

[0008] A method for security inspection and monitoring of cluster big data training process includes the following steps: S1: Perform information reduction processing on the raw training data, and simultaneously extract data risk features and bind them to the reduced data; S2: Based on the risk feature labels, target and identify dangerous data, perform encryption processing on the dangerous data, encapsulate it, match the transmission protocol, and complete the transmission of the encapsulated packet. S3: Parse and decrypt the encrypted package, complete the risk assessment and generate the processing strategy through the privately deployed risk handling decision-making big model; S3 includes: S31: Decrypt and restore cached data; S32: Based on risk decision-making strategies, match risk characteristics and perform risk assessments, and calculate the comprehensive risk value by considering the probability of occurrence of various risks, the degree of impact, and the response time. S33: Generate a handling strategy based on the risk assessment results; S34: Establish security rules for risk handling; S35: Perform safety checks and output safety metrics, and calculate the comprehensive safety metric value by the total number of safety check items, the weight of each item, the actual score and the full score; S36: Generate a security report; S4: The processing strategy is distributed to the training cluster and real-time security monitoring is started. Monitoring data is collected and fed back to the previous module to generate optimization parameters. In S4, the real-time monitoring adopts a distributed monitoring node architecture based on federated learning. Each node independently collects monitoring data and performs local preprocessing. Only the feature vector is uploaded to the central node for aggregation analysis. S5: Repeatedly execute the S2-S4 process using the optimized parameters and test data until the iteration meets the convergence condition.

[0009] Preferably, the data reduction process in S1 is used to reduce the amount and complexity of the data to be trained. The data reduction process uses knowledge transfer learning technology and feature selection method to reduce the dimensionality of the data and transform the data into a structure that is more suitable for machine learning processing. Data reduction processing also includes cleaning and standardizing the raw training data, and filtering, cleaning and deleting data containing irrelevant and duplicate information; The feature selection method uses the information gain algorithm to screen key features. It obtains the information gain of the feature on the dataset by calculating the information entropy of the dataset and the sample proportion of the feature value subset.

[0010] Preferably, the risk characteristics in S1 include sensitive information, malicious attack characteristics, abnormal characteristics, model poisoning characteristics, and privacy information, wherein: The sensitive information includes personal identity and financial data. Sensitive fields and data are extracted through regular expression matching and rule matching. The malicious attack characteristics include DoS attack characteristics, DDoS attack characteristics, and brute-force attack characteristics, which are detected and extracted through a feature database and behavioral feature analysis. The abnormal features include unknown data format features, unknown image features, and unknown text features, which are detected and extracted using anomaly detection algorithms; The poisoning features of the model include data label disorder, malicious tampering, and data deviation, which are detected and extracted using anomaly detection algorithms; The privacy information includes data leakage characteristics and time leakage characteristics, and features are extracted from encrypted data using an encryption feature extraction algorithm.

[0011] Preferably, step S2 includes the following sub-steps: S21. Data security encapsulation strategy decision: Match the encapsulation strategy to the dangerous data, and select the encryption algorithm and encapsulation scheme; S22. Encapsulating and transmitting encapsulated data, wherein the encapsulated data includes a checksum of risk characteristics, and the encapsulated data is compressed, segmented and encrypted according to the data size and encryption level before transmission; S23. Distributed data caching and distribution: Based on the number of packages, determine the storage nodes and start the cache queue. Allocate data cache nodes in sequence according to the queue, and then distribute the data from the cache nodes to the target nodes. S24. Packet integrity verification and storage: Integrity verification is performed on the data cache node. The integrity verification is calculated based on the checksum in the packaged data and compared with the checksum. If the verification result does not meet the expectations, the package is an abnormal package. S25. Risk feature comparison and storage: Decode the identification content of dangerous data, compare the decoded abnormal package with the risk feature library, and if dangerous data exists, formulate a data storage strategy based on the actual situation and business needs. S26. Secure decryption and storage of the package: After verifying the integrity of the packaged data and extracting risk features, the abnormal package is securely decrypted and the decrypted content is stored.

[0012] Preferably, in S21, the dangerous data matching and encapsulation strategy includes matching encryption algorithms and encapsulation levels according to data content, and matching encapsulation schemes according to data size. The encapsulated data consists of a compression module, a segmentation module, and an encryption module; The compression module performs compression on different data types within the package; The encryption module is used to perform encryption processing on sensitive and private data; The segmentation module is used to encapsulate data segments and risk labels.

[0013] Preferably, in step S22, the encapsulated data is encapsulated and compressed: the data is compressed according to the size, type and actual needs of the encapsulated data, the data is converted into binary stream and bytecode form, and several compression algorithms are selected to compress the data according to the content and size of the encapsulated data. In step S22, the encapsulated data is segmented and encrypted: S221: Segment the data, obtain the data segments and risk labels, and generate risk label checksums; S222: Perform encryption processing on the data segment by calling the encryption algorithm and key from the local encryption module and completing the encryption process; S223: Encapsulate the data segment, risk label, and risk label checksum into a complete package.

[0014] Preferably, step S23 includes selecting a cache node and cache capacity based on the size of the encapsulated data and the data transmission bandwidth; The storage node determines the data storage structure and storage method according to storage requirements. The storage node includes servers, data warehouse type databases, and uses distributed file systems, data warehouses, and distributed object storage. In step S25, if the data contains anomalies, a corresponding data processing strategy is formulated based on the type and severity of the anomalies, wherein: For sensitive and privacy-sensitive data, an asymmetric encryption scheme is used to encrypt the data, and a layered encryption method is adopted to encrypt the data. For other types of data, symmetric encryption is used for encryption, and hierarchical encryption is employed to further encrypt the data. For highly confidential data, a signature encryption algorithm is used to encrypt the data and verify its integrity and authenticity.

[0015] As a preferred option, the risk processing decision-making big model deployed privately in S3 is a risk decision-making big model based on BERT-wwm-ext, which introduces an attention mechanism to focus on key risk features and integrates a federated learning framework, so that the model can be updated without leaving the local data of each node. The model is pre-trained based on a preset number of malicious attack samples, privacy breach cases, and model poisoning data, and then fine-tuned to adapt to specific business scenarios.

[0016] Preferably, the processing strategy generated in S3 based on the risk assessment results includes data labeling and removal, data updating and retraining, data verification and model optimization. The risk handling security rules include data sources and data characteristics, security check items, security check frequency, and testing methods; Security metrics specifically include performing vulnerability scanning on data and models, privacy risk analysis, data signature consistency verification, and cybersecurity assessment. The security report includes security vulnerability analysis, security risk assessment, and data security control recommendations.

[0017] Preferably, the real-time monitoring in S4 includes detecting security vulnerabilities in the data, verifying the consistency of the data, verifying the integrity of the data, and identifying data from illegal sources; Perform false alarm rate analysis and anomaly detection on real-time running models and their prediction results; monitor and analyze the training process of machine learning models; and identify and prevent adversarial attacks against the models. Based on real-time monitoring results, the parameters of the training process are optimized and fed back to the data preprocessing module; based on the monitored anomaly detection results and false alarm rate, the model training method is optimized and the preprocessing process is iteratively updated; based on the monitored data consistency verification results, the data cleaning and standardization process is optimized; based on the monitored data integrity verification results and illegal source identification results, data update and deletion operations are performed for the monitored data integrity and security risks, and the real-time monitoring false alarm rate analysis results are fed back to the risk decision-making process to iteratively optimize the construction of security rules.

[0018] The present invention has the following beneficial effects: This invention reduces the data size and speeds up processing by preprocessing large clusters of data without losing necessary information. This invention achieves targeted detection of potential risks by extracting risk characteristics, which helps to identify and address security issues in advance. This invention protects sensitive data and improves the security of data transmission by encapsulating and encrypting dangerous data. This invention provides a rapid response capability for risk events during cluster training through real-time monitoring and strategy generation; This invention continuously improves the effectiveness of risk inspection and monitoring through a feedback optimization mechanism, providing continuous security for the enterprise application cluster training process. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the steps of a method for security inspection and monitoring of a cluster big data training process according to the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly described below in conjunction with the examples.

[0021] like Figure 1 As shown, this invention proposes a method for security inspection and monitoring of cluster big data training processes, specifically including the following steps: S1: Data preprocessing and risk feature extraction: Perform information reduction processing on the original training data, and simultaneously extract data risk features and bind them to the reduced data. S2: Encryption and secure transmission of hazardous data: Hazardous data is detected and identified based on the risk feature tags, encrypted and encapsulated after the hazardous data is encrypted, and the transmission protocol is matched to complete the transmission of the encapsulated packet. S3: Private Large Model Risk Decision-Making and Strategy Generation, parsing and decrypting the encrypted package, completing risk assessment and generating processing strategies through the privately deployed risk processing decision-making large model; S4: Real-time operation monitoring and optimization parameter feedback, the processing strategy is sent to the GPU training cluster and real-time security monitoring is started, monitoring data is collected and fed back to the previous module to generate optimization parameters; S5: Repeatedly test the iteration and model convergence. Using the optimization parameters and test data, repeatedly execute the S2-S4 process until the iteration meets the convergence condition.

[0022] In a preferred embodiment, the data reduction process in S1 is used to reduce the amount and complexity of the training data. The data reduction method employs knowledge transfer learning and feature selection to reduce the dimensionality of the data and transform it into a structure more suitable for machine learning. The data reduction process also includes cleaning and standardizing the original training data, and filtering, cleaning, and deleting data containing irrelevant or repetitive information. The feature selection method uses the information gain algorithm to select key features, and the core formula is: Representation of features For dataset Information gain Represents the dataset Information entropy Representation of features The set of all possible values, Represents the dataset Chinese characteristics Values a subset of Representing a subset The number of samples, Represents the dataset The total number of samples.

[0023] In a preferred embodiment, the risk characteristics in S1 include sensitive information, malicious attack characteristics, abnormal characteristics, model poisoning characteristics, and privacy information.

[0024] In a preferred embodiment, the sensitive information is sensitive information containing personal identity or financial data, and sensitive fields and data are extracted through regular expression matching or rule matching; the malicious attack features include DoS attack features, DDoS attack features, or brute-force attack features, which are detected and extracted through feature databases and behavioral feature analysis; the abnormal features include unknown data format features, unknown image features, or unknown text features, which are detected and extracted through anomaly detection algorithms; the model poisoning features include data label disorder, malicious tampering, or data deviation, which are detected and extracted through anomaly detection algorithms; the privacy information includes data leakage features and time leakage features, and features are extracted from encrypted data through encryption feature extraction algorithms.

[0025] In a preferred embodiment, after processing the original data to be trained in step S2, the process further includes storing the reduced data, the model to be trained, and the feature set extracted from the risk features in a distributed file database respectively; after processing the original data to be trained and generating the task process, the process further includes calling the task process to perform data encapsulation and preliminary matching with risk targets. After completing the preliminary matching through a preset algorithm, the process further executes corresponding encapsulation strategies for the features of different risk targets.

[0026] In a preferred embodiment, S2 includes the following process: S21. Data security encapsulation strategy decision: Match the encapsulation strategy to the dangerous data, and select the encryption algorithm and encapsulation scheme; S22. Encapsulating and transmitting encapsulated data, wherein the encapsulated data includes a checksum of risk characteristics, and the encapsulated data is compressed, segmented and encrypted according to the data size and encryption level before transmission; S23. Distributed data caching and distribution: Based on the number of packages, determine the storage nodes and start the cache queue. Allocate data cache nodes in sequence according to the queue, and then distribute the data from the cache nodes to the target nodes. S24. Packet integrity verification and storage: Integrity verification is performed on the data cache node. The integrity verification is calculated based on the checksum in the packaged data and compared with the checksum. If the verification result does not meet the expectations, the package is an abnormal package. S25. Risk feature comparison and storage: Decode the identification content of dangerous data, compare the decoded abnormal package with the risk feature library, and if dangerous data exists, formulate a data storage strategy based on the actual situation and business needs. S26. Secure decryption and storage of the package: After verifying the integrity of the packaged data and extracting risk features, the abnormal package is securely decrypted and the decrypted content is stored.

[0027] In a preferred embodiment, step S21 involves matching the dangerous data with a packaging strategy, including matching encryption algorithms and packaging levels based on data content and matching packaging schemes based on data size; the packaged data consists of a compression module, a segmentation module, and an encryption module. The compression module compresses data of different data types in the package to different sizes; the encryption module performs encryption processing on sensitive and private data; and the segmentation module encapsulates data segments and risk labels.

[0028] In a preferred embodiment, in step S22, the encapsulated data is encapsulated and compressed: the data is compressed according to the size, type and actual needs of the encapsulated data, and the data is converted into a binary stream or bytecode form; One or more compression algorithms are selected based on the encapsulated data content and data size to compress the data, thereby reducing data capacity and optimizing memory storage; In step S22, the encapsulated data is segmented and encrypted. The data is segmented to obtain data segments and risk tags, and a risk tag checksum is generated. The data segment is encrypted by calling the encryption algorithm and key from the local encryption module and completing the encryption process. The encrypted data segment cannot be directly read or understood. The data segment, risk label, and risk label checksum are encapsulated into a complete package.

[0029] In a preferred embodiment, step S23 includes selecting appropriate cache nodes and cache capacity based on the size of the encapsulated data and the data transmission bandwidth; the storage node determines the data storage structure and storage method according to storage requirements; the storage node includes servers, databases of the data warehouse type, and uses distributed file systems, data warehouses, or distributed object storage; the server uses memory, SSD, or HDD type storage media. In step S25, if the data contains anomalies, a corresponding data processing strategy is formulated based on the type and severity of the anomalies. For sensitive and privacy-related data, an asymmetric encryption scheme is used to encrypt the data, and a layered encryption method is adopted to encrypt the data. For other types of data, a symmetric encryption method is used to encrypt the data, and a hierarchical encryption method is adopted to encrypt the data. For highly confidential data, a signature encryption algorithm is used to encrypt the data and verify its integrity and authenticity. In step S26, data decryption includes executing a decryption algorithm on the data stream, extracting data tags based on risk tags in the encapsulated data, and calculating a risk feature checksum based on the risk tags.

[0030] In a preferred embodiment, S3 includes the following process: S31. Data decryption and content restoration: Decrypt the cached data in S2 and restore the data, while simultaneously performing data preprocessing. S32. Risk Feature Matching and Decision-Making: Based on the reduction data in S1, risk feature matching is performed, and risk assessment is conducted. The core formula for risk assessment is: This represents the overall risk value, ranging from 0 to 100. A higher value indicates a higher risk. Indicates the number of risk types. Indicates the first The probability of this type of risk occurring ranges from 0 to 1. Indicates the first The severity of the impact of this type of risk is rated on a scale of 1 to 10, with higher scores indicating a more severe impact. Indicates the first Response and processing time for similar risks; S33. Risk analysis and strategy generation: Generate processing strategies based on risk assessment results. The processing strategies include data labeling and removal, data updating and retraining, data verification and model optimization. S34. Risk handling security rule establishment: Based on the content of S32 and S33, risk handling security rules are constructed. The security rules include data sources and data characteristics, security inspection items, security inspection frequency and testing methods. S35. Data security checks and security metrics: Based on security rules, perform security checks and output quantitative security metrics. The core formula is: This represents a comprehensive security metric, ranging from 0 to 1. Values ​​closer to 1 indicate higher security. This indicates the total number of security checks. Indicates the first The weight of each inspection item , Indicates the first The actual score of each inspection item. Indicates the first The full score for each inspection item; Security metrics specifically include performing vulnerability scanning on data and models, privacy risk analysis, data signature consistency verification, and cybersecurity assessment. S36. Security Report Generation: The results of security metric analysis are used to generate a security report, which includes security vulnerability analysis, security risk assessment, and data security control recommendations.

[0031] The risk handling decision-making big model for the private deployment is an improved risk decision-making big model based on BERT-wwm-ext. It introduces an attention mechanism to focus on key risk features and integrates a federated learning framework. The model can be updated without leaving the local data of each node, thus avoiding the risk of privacy leakage caused by data sharing. The model was pre-trained based on 100,000+ malicious attack samples, 50,000+ privacy breach cases, and 30,000+ data points of model poisoning. Through fine-tuning to adapt to specific business scenarios, the model accuracy reached 98.2%, and the false positive rate was less than 1.5%.

[0032] In a preferred embodiment, the real-time monitoring in S4 includes detecting security vulnerabilities in the data, verifying the consistency and integrity of the data, and identifying data from illegal sources; performing false positive rate analysis and anomaly detection on the real-time running model and its prediction results; monitoring and analyzing the training process of the machine learning model; and identifying and preventing adversarial attacks against the model. Real-time monitoring provides decision-making support for the monitoring module and the anomaly handling module based on security analysis results, risk assessment results, and security reports. The real-time monitoring adopts a distributed monitoring node architecture based on federated learning. Each node independently collects monitoring data and performs local preprocessing, uploading only feature vectors to the central node for aggregation analysis, ensuring data privacy is not compromised.

[0033] In a preferred embodiment, in step S4, the parameters of the training process are optimized based on the real-time monitoring results and fed back to the data preprocessing module; the model training method is optimized and the preprocessing process is iteratively updated based on the monitored anomaly detection results and false alarm rate; the data cleaning and standardization process is optimized based on the monitored data consistency verification results; and data update and deletion operations are performed for the monitored data integrity and security risks based on the monitored data integrity verification results and illegal source identification results. At the same time, the false alarm rate analysis results of the real-time monitoring are fed back to the risk decision-making process to iteratively optimize the construction of security rules.

[0034] Example 1 This invention discloses a method for security inspection and monitoring during the training process of clustered big data. The application scenario is the cluster training of a medical imaging AI diagnostic model in a tertiary hospital. The training data consists of 100,000 chest CT images (DICOM format) and corresponding electronic medical record text data. The goal is to train an early lung cancer screening model. Patient privacy must be protected, such as protecting their ID numbers, medical record numbers, and medical history from disclosure, preventing malicious injection of erroneous image data that could lead to diagnostic biases in the model. Simultaneously, the security of data transmission and model operation during training must be ensured. The method specifically includes the following steps: S1: Data preprocessing and risk feature extraction. Information reduction processing was performed on the original training data of 100,000 chest CT images and electronic medical record texts. This involved knowledge transfer learning based on the transferred features of a trained ImageNet model and feature selection based on the information gain formula, including: in For a dataset of 100,000 images, For image features, such as lesion size, density, and location, 20 key features were selected by calculating information gain, reducing the data volume from 100GB to 30GB. Simultaneously, risk features were extracted and bound to the reduced data. These risk features included: sensitive information, such as patient ID numbers and medical record numbers, extracted using regular expression matching; malicious attack features, including image transmission interruption features caused by DDoS attacks, matched using a feature library; abnormal features, including non-DICOM format junk files, identified using anomaly detection algorithms; model poisoning features, including tampered lesion annotation data, such as labeling benign lesions as malignant, extracted using anomaly detection algorithms; and privacy information, including genetic disease information in medical history texts, extracted using encryption feature extraction algorithms.

[0035] S2: Encryption and secure transmission of dangerous data. Based on the risk feature tags, dangerous data is targeted and detected, including 3,000 medical record texts with patient ID numbers and 500 cases of tampered labeled images. The dangerous data is encrypted, including sensitive privacy data using RSA-2048 asymmetric encryption and other dangerous data using AES-256 symmetric encryption. Then, the data is encrypted and encapsulated. The encapsulated package is divided into 1,000 segments, each 300MB, and the HTTPS transmission protocol is matched to complete the transmission of the encapsulated package. The specific process includes: S21. Data security encapsulation strategy decision: match encryption algorithms and encapsulation levels according to data content, including sensitive privacy data / ordinary dangerous data, and match segmented encapsulation schemes according to data size of 30GB. S22. Encapsulation and transmission of encapsulated data: The encapsulated data contains SHA-256 checksums of risk characteristics. Before transmission, the data is compressed, segmented, and encrypted. The compression uses the GZIP algorithm, and each segment is 300MB, converting the data into a binary stream. After encrypting the data segments, the data segments, risk labels, and risk label checksums are encapsulated into a complete encapsulated packet. For example, if the checksum of a certain encapsulated packet is a8f57c3d2e1b0987654321fedcba0987654321fedcba, it is encapsulated into a complete encapsulated packet. S23. Distributed data caching and distribution: For 1000 packaged packets, select 3 SSD storage servers as cache nodes, each with a cache capacity of 10GB, start a cache queue, allocate data cache nodes according to the queue, and then distribute data from the cache nodes to 8 GPU training nodes. S24. Packet integrity verification and storage: Perform integrity verification on data cache nodes. Calculate and compare the SHA-256 checksum in the packaged data. If the calculated checksum of a package matches the preset value, it is stored normally. If it does not match, it is marked as an abnormal package. S25. Risk feature comparison and storage: Decode the identification content of the abnormal package, and after confirming the existence of dangerous data by comparing with the risk feature database, formulate a storage strategy. Sensitive data is encrypted and stored on an offline server, while ordinary dangerous data is stored in isolation. S26. Secure decryption and storage of the package: After integrity verification and risk feature extraction, decrypt the abnormal package, use the RSA private key to decrypt sensitive data and the AES key to decrypt ordinary data, and store the decrypted content.

[0036] S3: Privatization Large-Scale Model Risk Decision-Making and Strategy Generation. The encrypted package is parsed and decrypted. Risk assessment is performed using a privatization risk handling decision-making large-scale model based on BERT-wwm-ext, where the model integrates a federated learning framework. The risk assessment uses the following formula: in This includes risks such as malicious data, privacy breaches, and abnormal transmission. , , Privacy risk , , Transmission anomaly risk , , Calculated Based on the risk assessment results, the following processing strategies were generated: 500 cases of tampered labeled images were removed, updated with the correct labeled data, and retrained; 3,000 medical record texts containing sensitive information were anonymized, including replacing ID card numbers and medical record numbers with virtual identifiers.

[0037] The S3 process also includes: S31, Data Decryption and Content Restoration: Decrypt the cached data in S2 and perform data restoration, while simultaneously performing data preprocessing, including image format standardization and text segmentation. S32, Risk Feature Matching and Decision Making: Perform risk feature matching and assessment on the reduction data in S1; S33. Risk analysis and strategy generation: Generate the above data removal, desensitization, and update strategies. S34. Risk handling security rule establishment: Construct security rules, limit the data source to the hospital PACS system, and include security check items such as data integrity and privacy anonymization. The check frequency is once per hour, and the testing method adopts sampling verification. S35. Data security inspection and security metrics: Perform inspections according to security rules, output quantitative security metrics, using the following formula: in This includes vulnerability scanning, privacy risks, consistency verification, network security, and weighting. , , , Actual score , , , Full marks Calculated The initial value is used, and it will be gradually improved after optimization; S36, security report generation, generates a report based on the security measurement results, including vulnerability analysis, risk assessment, and control recommendations, including 1 low-risk transmission vulnerability, the main risk is privacy leakage, and it is recommended to strengthen the desensitization of sensitive fields.

[0038] S4: Real-time operation monitoring and optimization parameter feedback: The processing strategy is distributed to the 8 NVIDIA A100 GPU training clusters and real-time security monitoring is initiated. A distributed monitoring node architecture based on federated learning is adopted. Each node independently collects monitoring data, verifies the SHA-256 value of image files every hour, identifies data uploaded from non-hospital PACS systems, and only uploads feature vectors to the central node for aggregation and analysis. Real-time monitoring includes detecting data security vulnerabilities, verifying data consistency and integrity, identifying data from illegal sources, performing false alarm rate analysis on model prediction results (initial false alarm rate 3%), and anomaly detection to prevent adversarial attacks. Based on the real-time monitoring results, training parameters are optimized: the data cleaning threshold is adjusted to image filtering with ambiguity > 0.3, the privacy desensitization rule is updated to automatically replace all personally identifiable information fields, the security check frequency is increased to once every 30 minutes, and the optimized parameters are fed back to the data preprocessing module and risk decision module.

[0039] S5: Repeatedly test iterations and model convergence. Using the optimized parameters and 20,000 test cases, repeatedly execute processes S2-S4. After each iteration, the model accuracy improves by 2%-3%, and the false positive rate decreases by 0.5%-1%. After 10 iterations, the model accuracy reaches 95%, the false positive rate is less than 1%, and the safety metric is [value missing]. The convergence condition is met, and training is complete.

[0040] Example 2 This embodiment is a further optimization based on the above embodiment, including the following: S1 includes: the sensitive information being sensitive information containing personal identity or financial data, with sensitive fields and data extracted through regular expression matching or rule matching; the malicious attack features including DoS attack features, DDoS attack features, or brute-force attack features, which are detected and extracted through feature databases and behavioral feature analysis; the abnormal features including unknown data format features, unknown image features, or unknown text features, which are detected and extracted through anomaly detection algorithms; the model poisoning features including data label disorder, malicious tampering, or data deviation, which are detected and extracted through anomaly detection algorithms; and the privacy information including data leakage features and time leakage features, with features extracted from encrypted data through encryption feature extraction algorithms.

[0041] S2 includes the following process: S21. Data security encapsulation strategy decision: Match the encapsulation strategy to the dangerous data, and select the encryption algorithm and encapsulation scheme; S22. Encapsulating and transmitting encapsulated data, wherein the encapsulated data includes a checksum of risk characteristics, and the encapsulated data is compressed, segmented and encrypted according to the data size and encryption level before transmission; S23. Distributed data caching and distribution: Based on the number of packages, determine the storage nodes and start the cache queue. Allocate data cache nodes in sequence according to the queue, and then distribute the data from the cache nodes to the target nodes. S24. Packet integrity verification and storage: Integrity verification is performed on the data cache node. The integrity verification is calculated based on the checksum in the packaged data and compared with the checksum. If the verification result does not meet the expectations, the package is an abnormal package. S25. Risk feature comparison and storage: Decode the identification content of dangerous data, compare the decoded abnormal package with the risk feature library, and if dangerous data exists, formulate a data storage strategy according to the actual situation and business needs. S26. Secure decryption and storage of the package: After verifying the integrity of the packaged data and extracting risk features, the abnormal package is securely decrypted and the decrypted content is stored.

[0042] Example 3 This embodiment is a further optimization based on the above embodiment, including the following: In step S21, the dangerous data matching and encapsulation strategy includes matching encryption algorithms and encapsulation levels based on data content, and matching encapsulation schemes based on data size; the encapsulated data consists of a compression module, a segmentation module, and an encryption module; the compression module compresses data of different data types in the encapsulated package to different sizes, the encryption module performs encryption processing on sensitive and private data, and the segmentation module encapsulates data segments and risk labels.

[0043] Example 4 This embodiment is a further optimization based on the above embodiment, including the following: In step S22, the encapsulated data is encapsulated and compressed: the data is compressed according to its size, type and actual needs, and converted into a binary stream or bytecode format; one or more compression algorithms are selected to compress the data according to its content and size, thereby reducing data capacity and optimizing memory storage. In step S22, the encapsulated data is segmented and encrypted. The data is segmented to obtain data segments and risk tags, and a risk tag checksum is generated. The data segments are encrypted by calling an encryption algorithm and key from the local encryption module and completing the encryption process. The encrypted data segments cannot be directly read or understood. The data segments, risk tags, and risk tag checksums are encapsulated into a complete package. S23 includes selecting appropriate cache nodes and cache capacity based on the size of the encapsulated data and the data transmission bandwidth; the storage nodes determine the data storage structure and storage method according to storage requirements; the storage nodes include servers, data warehouse type databases, and use distributed file systems, data warehouses, or distributed object storage; the servers use memory, SSD, or HDD type storage media. In step S25, if the data contains anomalies, a corresponding data processing strategy is formulated based on the type and severity of the anomalies. For sensitive and privacy-related data, an asymmetric encryption scheme is used to encrypt the data, and a layered encryption method is adopted to encrypt the data. For other types of data, a symmetric encryption method is used to encrypt the data, and a hierarchical encryption method is adopted to encrypt the data. For highly confidential data, a signature encryption algorithm is used to encrypt the data and verify its integrity and authenticity. In step S26, data decryption includes executing a decryption algorithm on the data stream, extracting data tags based on risk tags in the encapsulated data, and calculating a risk feature checksum based on the risk tags.

[0044] S3 includes the following process: S31. Data decryption and content restoration: Decrypt the cached data in S2 and restore the data, while simultaneously performing data preprocessing. S32. Risk Feature Matching and Decision-Making: Based on the reduction data in S1, risk feature matching is performed, and risk assessment is conducted. The core formula for risk assessment is: This represents the overall risk value, ranging from 0 to 100. A higher value indicates a higher risk. Indicates the number of risk types. Indicates the first The probability of this type of risk occurring ranges from 0 to 1. Indicates the first The severity of the impact of this type of risk is rated on a scale of 1 to 10, with higher scores indicating a more severe impact. Indicates the first Response and processing time for similar risks; S33. Risk analysis and strategy generation: Generate processing strategies based on risk assessment results. The processing strategies include data labeling and removal, data updating and retraining, data verification and model optimization. S34. Risk handling security rule establishment: Based on the content of S32 and S33, risk handling security rules are constructed. The security rules include data sources and data characteristics, security inspection items, security inspection frequency and testing methods. S35. Data security checks and security metrics: Based on security rules, perform security checks and output quantitative security metrics. The core formula is... This represents a comprehensive security metric, ranging from 0 to 1. Values ​​closer to 1 indicate higher security. This indicates the total number of security checks. Indicates the first The weight of each inspection item , Indicates the first The actual score of each inspection item. Indicates the first The full score for each inspection item; security metrics specifically include performing vulnerability scanning, privacy risk analysis, data signature consistency verification, and cybersecurity assessment on data and models; S36. Security Report Generation: The results of security metric analysis are used to generate a security report, which includes security vulnerability analysis, security risk assessment, and data security control recommendations.

[0045] The risk handling decision-making big model for the private deployment is an improved risk decision-making big model based on BERT-wwm-ext. It introduces an attention mechanism to focus on key risk features and integrates a federated learning framework. The model can be updated without leaving the local data of each node, thus avoiding the risk of privacy leakage caused by data sharing. The model was pre-trained based on 100,000+ malicious attack samples, 50,000+ privacy breach cases, and 30,000+ data points of model poisoning. Through fine-tuning to adapt to specific business scenarios, the model accuracy reached 98.2%, and the false positive rate was less than 1.5%.

[0046] The real-time monitoring in S4 includes detecting security vulnerabilities in the data, verifying data consistency and integrity, and identifying data from illegal sources; performing false positive rate analysis and anomaly detection on the real-time running model and its prediction results; monitoring and analyzing the training process of the machine learning model; and identifying and preventing adversarial attacks against the model. Real-time monitoring provides decision-making support for the monitoring and anomaly handling modules based on security analysis results, risk assessment results, and security reports. The real-time monitoring adopts a distributed monitoring node architecture based on federated learning. Each node independently collects monitoring data and performs local preprocessing, uploading only feature vectors to the central node for aggregation analysis, ensuring data privacy and preventing data leakage.

[0047] In step S4, the parameters of the training process are optimized based on the real-time monitoring results and fed back to the data preprocessing module; the model training method is optimized and the preprocessing process is iteratively updated based on the monitored anomaly detection results and false alarm rate; the data cleaning and standardization process is optimized based on the monitored data consistency verification results; and data update and deletion operations are performed for the monitored data integrity and security risks based on the monitored data integrity verification results and illegal source identification results. At the same time, the false alarm rate analysis results of the real-time monitoring are fed back to the risk decision-making process to iteratively optimize the construction of security rules.

[0048] In summary, this invention, based on large model technology and detection and decision technology, completes a cluster big data training process monitoring system, constructs a complete monitoring system and a multi-module collaborative working mechanism, and ensures that the cluster operation is in a safe and controllable state.

[0049] It should be specifically noted that this invention is a method for security inspection and monitoring of cluster big data training process. There are many methods and approaches to implement this technical solution. The above description is only a preferred embodiment of this invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention. These improvements and modifications should also be considered within the scope of protection of this invention.

[0050] All components not explicitly described in the embodiments can be implemented using existing technologies. The above descriptions are merely preferred embodiments of the present invention and are not intended to limit the invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention fall within the protection scope of the present invention.

Claims

1. A method for security inspection and monitoring of clustered big data training process, characterized in that, Includes the following steps: S1: Perform information reduction processing on the raw training data, and simultaneously extract data risk features and bind them to the reduced data; S2: Based on the risk feature labels, target and identify dangerous data, perform encryption processing on the dangerous data, encapsulate it, match the transmission protocol, and complete the transmission of the encapsulated packet. S3: Parse and decrypt the encrypted package, complete the risk assessment and generate the processing strategy through the privately deployed risk handling decision-making big model; S3 includes: S31: Decrypt and restore cached data; S32: Based on risk decision-making strategies, match risk characteristics and perform risk assessments, and calculate the comprehensive risk value by considering the probability of occurrence of various risks, the degree of impact, and the response time. S33: Generate a handling strategy based on the risk assessment results; S34: Establish security rules for risk handling; S35: Perform safety checks and output safety metrics, and calculate the comprehensive safety metric value by the total number of safety check items, the weight of each item, the actual score and the full score; S36: Generate a security report; S4: The processing strategy is distributed to the training cluster and real-time security monitoring is started. Monitoring data is collected and fed back to the previous module to generate optimization parameters. In S4, the real-time monitoring adopts a distributed monitoring node architecture based on federated learning. Each node independently collects monitoring data and performs local preprocessing. Only the feature vector is uploaded to the central node for aggregation analysis. S5: Repeatedly execute the S2-S4 process using the optimized parameters and test data until the iteration meets the convergence condition.

2. The method for security inspection and monitoring of cluster big data training process according to claim 1, characterized in that: The data reduction process in S1 is used to reduce the amount and complexity of the data to be trained. The data reduction process uses knowledge transfer learning technology and feature selection method to reduce the dimensionality of the data and transform the data into a structure that is more suitable for machine learning processing. Data reduction processing also includes cleaning and standardizing the raw training data, and filtering, cleaning and deleting data containing irrelevant and duplicate information; The feature selection method uses the information gain algorithm to screen key features. It obtains the information gain of the feature on the dataset by calculating the information entropy of the dataset and the sample proportion of the feature value subset.

3. The method for security inspection and monitoring of cluster big data training process according to claim 1, characterized in that: The risk characteristics in S1 include sensitive information, malicious attack characteristics, abnormal characteristics, model poisoning characteristics, and privacy information, among which: The sensitive information includes personal identity and financial data. Sensitive fields and data are extracted through regular expression matching and rule matching. The malicious attack characteristics include DoS attack characteristics, DDoS attack characteristics, and brute-force attack characteristics, which are detected and extracted through a feature database and behavioral feature analysis. The abnormal features include unknown data format features, unknown image features, and unknown text features, which are detected and extracted using anomaly detection algorithms; The poisoning features of the model include data label disorder, malicious tampering, and data deviation, which are detected and extracted using anomaly detection algorithms; The privacy information includes data leakage characteristics and time leakage characteristics, and features are extracted from encrypted data using an encryption feature extraction algorithm.

4. The method for security inspection and monitoring of cluster big data training process according to claim 1, characterized in that: S2 includes the following sub-steps: S21. Data security encapsulation strategy decision: Match the encapsulation strategy to the dangerous data, and select the encryption algorithm and encapsulation scheme; S22. Encapsulating and transmitting encapsulated data, wherein the encapsulated data includes a checksum of risk characteristics, and the encapsulated data is compressed, segmented and encrypted according to the data size and encryption level before transmission; S23. Distributed data caching and distribution: Based on the number of packages, determine the storage nodes and start the cache queue. Allocate data cache nodes in sequence according to the queue, and then distribute the data from the cache nodes to the target nodes. S24. Packet integrity verification and storage: Integrity verification is performed on the data cache node. The integrity verification is calculated based on the checksum in the packaged data and compared with the checksum. If the verification result does not meet the expectations, the package is an abnormal package. S25. Risk feature comparison and storage: Decode the identification content of dangerous data, compare the decoded abnormal package with the risk feature library, and if dangerous data exists, formulate a data storage strategy based on the actual situation and business needs. S26. Secure decryption and storage of the package: After verifying the integrity of the packaged data and extracting risk features, the abnormal package is securely decrypted and the decrypted content is stored.

5. The method for security inspection and monitoring of cluster big data training process according to claim 4, characterized in that: In S21, the dangerous data matching and encapsulation strategy includes matching encryption algorithms and encapsulation levels according to data content, and matching encapsulation schemes according to data size. The encapsulated data consists of a compression module, a segmentation module, and an encryption module; The compression module performs compression on different data types within the package; The encryption module is used to perform encryption processing on sensitive and private data; The segmentation module is used to encapsulate data segments and risk labels.

6. The method for security inspection and monitoring of cluster big data training process according to claim 4, characterized in that: In step S22, the encapsulated data is encapsulated and compressed: the data is compressed according to the size, type and actual needs of the encapsulated data, the data is converted into binary stream and bytecode form, and several compression algorithms are selected to compress the data according to the content and size of the encapsulated data. In step S22, the encapsulated data is segmented and encrypted: S221: Segment the data, obtain the data segments and risk labels, and generate risk label checksums; S222: Perform encryption processing on the data segment by calling the encryption algorithm and key from the local encryption module and completing the encryption process; S223: Encapsulate the data segment, risk label, and risk label checksum into a complete package.

7. The method for security inspection and monitoring of cluster big data training process according to claim 4, characterized in that: S23 includes selecting cache nodes and cache capacity based on the size of the encapsulated data and the data transmission bandwidth; The storage node determines the data storage structure and storage method according to storage requirements. The storage node includes servers, data warehouse type databases, and uses distributed file systems, data warehouses, and distributed object storage. In step S25, if the data contains anomalies, a corresponding data processing strategy is formulated based on the type and severity of the anomalies, wherein: For sensitive and privacy-sensitive data, an asymmetric encryption scheme is used to encrypt the data, and a layered encryption method is adopted to encrypt the data. For other types of data, symmetric encryption is used for encryption, and hierarchical encryption is employed to further encrypt the data. For highly confidential data, a signature encryption algorithm is used to encrypt the data and verify its integrity and authenticity.

8. The method for security inspection and monitoring of cluster big data training process according to claim 1, characterized in that: The risk management decision-making big model deployed in S3 is an improved risk decision-making big model based on BERT-wwm-ext. It introduces an attention mechanism to focus on key risk features and integrates a federated learning framework so that the model can be updated without leaving the local data of each node. The model is pre-trained based on a preset number of malicious attack samples, privacy breach cases, and model poisoning data, and then fine-tuned to adapt to specific business scenarios.

9. The method for security inspection and monitoring of cluster big data training process according to claim 1, characterized in that: The processing strategies generated in S3 based on the risk assessment results include data labeling and removal, data updating and retraining, data verification and model optimization. The risk handling security rules include data sources and data characteristics, security check items, security check frequency, and testing methods; Security metrics specifically include performing vulnerability scanning on data and models, privacy risk analysis, data signature consistency verification, and cybersecurity assessment. The security report includes security vulnerability analysis, security risk assessment, and data security control recommendations.

10. The method for security inspection and monitoring of cluster big data training process according to claim 1, characterized in that: The real-time monitoring in S4 includes detecting security vulnerabilities in the data, verifying the consistency and integrity of the data, and identifying data from illegal sources. Perform false alarm rate analysis and anomaly detection on real-time running models and their prediction results; monitor and analyze the training process of machine learning models; and identify and prevent adversarial attacks against the models. Based on real-time monitoring results, the parameters of the training process are optimized and fed back to the data preprocessing module; Based on the monitored anomaly detection results and false alarm rate, optimize the model training method and iteratively update the preprocessing process; Based on the data consistency verification results from the monitoring, optimize the data cleaning and standardization process; Based on the data integrity verification results and illegal source identification results, data update and deletion operations are performed for the monitored data integrity and security risks. At the same time, the false alarm rate analysis results are fed back to the risk decision-making process to iteratively optimize the construction of security rules.

Images