Big data analysis-based cloud computing platform data storage method and system

Abstract

This invention relates to the field of data storage technology, specifically disclosing a data storage method and system for a cloud computing platform based on big data analytics. The method includes: classifying the data to be stored; transmitting the classified secure data and risky data to the main storage area and backup storage area respectively; constructing data nodes and evaluation time periods; measuring data coverage points and data coverage frequency within the evaluation time period; after determining the evaluation time period, determining the specific time of data coverage occurrence as reference data by comprehensively evaluating the data, data coverage points, and data coverage frequency within that time period; and based on the reference data, assessing the data coverage risk and generating a data coverage risk assessment result. This invention, by isolating and managing data of different security levels, avoids high-risk data from impacting and contaminating the core main storage area, thereby improving the storage security and reliability of core data on the cloud platform.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, specifically relating to a data storage method and system for a cloud computing platform based on big data analysis. Background Technology

[0002] A cloud computing platform is a network platform that provides computing, networking, and storage services based on hardware and software resources. Cloud computing platforms can be categorized by function into storage-based cloud computing platforms that primarily focus on data storage, computing-based cloud computing platforms that primarily focus on data processing, and comprehensive cloud computing platforms that balance computing, data storage, and processing. Specifically, cloud computing platforms provide computing power and storage support for enterprise digital business through data storage, processing, and analysis. Furthermore, by reliably accessing and managing data in the cloud, cloud computing platforms support enterprise data analysis, artificial intelligence applications, and other services.

[0003] Existing cloud computing platform data storage mechanisms mostly adopt strategies such as encrypting first and then storing, or directly storing in a designated database. This indiscriminate storage method leads to the mixed storage of high-priority and time-sensitive critical data with low-value data, increasing the internal management pressure of storage resources. It is easy to cause congestion during peak data read and write periods, resulting in response delays and reduced operating speed. Furthermore, congestion problems exacerbate the risk of data loss or corruption. Core business data may be damaged during storage, which can easily cause losses to the company's production and operation.

[0004] Furthermore, existing cloud computing platform data storage technologies perform only superficial classification and categorization before data is stored, failing to combine big data analytics to conduct forward-looking analysis of data correlation, importance, and access frequency. This can easily lead to unreasonable data storage layout, resulting in low efficiency in subsequent queries and retrievals, and making it difficult to meet the requirements for real-time and accurate data access.

[0005] In view of this, this application provides a data storage method and system for a cloud computing platform based on big data analysis. Summary of the Invention

[0006] The purpose of this invention is to provide a data storage method and system for a cloud computing platform based on big data analysis, in order to solve the problem in the prior art that the data coverage points are not effectively monitored during data recovery, resulting in insufficient compatibility between the original data and the risky data.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] The data storage method for cloud computing platforms based on big data analytics includes the following steps:

[0009] Retrieve the data to be stored;

[0010] Based on the risk level data contained in the data to be stored, the data to be stored is classified into safe data or risky data.

[0011] Perform storage operations on the tiered data to be stored;

[0012] Build a data analysis interface and obtain the latest data;

[0013] The latest data and the matching baseline data are synchronously transmitted to the data verification interface;

[0014] A comparison operation is performed through the data verification interface to generate a verification result;

[0015] Based on the analysis of historical call records in the main storage area, a data overwrite risk assessment result is generated regarding the data to be stored;

[0016] Based on the data coverage risk assessment results, determine whether the data to be stored is replacement data;

[0017] If it is determined that the data to be stored is replaceable data, then the data to be stored is loaded into the replacement operation queue.

[0018] Preferably, the step of classifying the data to be stored into safe data or risk data based on the risk level data contained therein includes:

[0019] Obtain detailed information about the data to be stored, including detection results for sensitive words; execute a risk level judgment function to determine the risk level based on the sensitivity in the detection results, and classify the data to be stored into safe data or risky data based on the judgment result.

[0020] Preferably, the storage operation on the hierarchical data to be stored includes:

[0021] Secure data is transferred to the primary storage area, while risky data is transferred to the backup storage area.

[0022] Preferably, the step of generating a data overwrite risk assessment result for the data to be stored based on the analysis of historical call records in the main storage area includes:

[0023] Obtain historical call records from the main storage area; construct data nodes based on the historical call records, and construct an evaluation time period based on the data nodes; measure the data coverage points and data coverage frequency within the evaluation time period to determine the specific time when data coverage occurred, and define the specific time as reference data; assess the data coverage risk based on the reference data to generate a data coverage risk assessment result.

[0024] Preferably, the step of assessing data coverage risk based on reference data to generate a data coverage risk assessment result includes:

[0025] The assessment is performed in descending order of data coverage frequency; based on the assessment, a risk score is obtained, and the risk score is used as a reference for the data coverage risk assessment results.

[0026] The cloud computing platform data storage system based on big data analytics, used to execute the aforementioned cloud computing platform data storage method based on big data analytics, includes the following modules:

[0027] The data storage classification module is used to acquire data to be stored; classify the data to be stored according to the risk level data contained in the data to be stored, dividing the data to be stored into safe data or risk data; and perform storage operations on the classified data to be stored.

[0028] The data coverage assessment module is used to generate a data coverage risk assessment result for the data to be stored based on the analysis of historical call records in the main storage area.

[0029] The risk data processing module is used to determine whether the data to be stored is replaceable data based on the data coverage risk assessment results; if the data to be stored is determined to be replaceable data, the data to be stored is loaded into the replacement operation queue.

[0030] Preferably, it also includes a data verification module, used to construct a data analysis interface and obtain the latest data; synchronously transmit the latest data and matching benchmark data to the data verification interface; and perform comparison operations through the data verification interface to form a verification result.

[0031] Preferably, the step of classifying the data to be stored into safe data or risk data based on the risk level data contained therein includes:

[0032] Obtain detailed information about the data to be stored, including detection results for sensitive words; execute a risk level judgment function to determine the risk level based on the sensitivity in the detection results, and classify the data to be stored into safe data or risky data based on the judgment result.

[0033] Preferably, the storage operation on the hierarchical data to be stored includes:

[0034] Secure data is transferred to the primary storage area, while risky data is transferred to the backup storage area.

[0035] Preferably, the step of generating a data overwrite risk assessment result for the data to be stored based on the analysis of historical call records in the main storage area includes:

[0036] Obtain historical call records from the main storage area; construct data nodes based on the historical call records, and construct an evaluation time period based on the data nodes; evaluate the data coverage points and data coverage frequency within the evaluation time period to determine the specific time when data coverage occurred, and define the specific time as reference data; based on the reference data, assess the data coverage risk to generate a data coverage risk assessment result.

[0037] Beneficial effects: This invention obtains risk level data contained in the data to be stored, classifies the data to be stored, and transmits the classified safe data and risk data to the main storage area and the backup storage area respectively. It isolates and manages data of different security levels, avoids high-risk data from impacting and contaminating the core main storage area, and improves the storage security and reliability of the core data of the cloud platform.

[0038] This invention acquires historical call records from the main storage area, constructs data nodes and evaluation time periods, measures data coverage points and data coverage frequency within the evaluation time period, transforms discrete historical call behaviors into structured data nodes, and defines a time window for risk assessment. It locks data coverage risks within the evaluation time period for analysis, providing an analytical basis for determining the specific time when data coverage occurs.

[0039] After determining the assessment period, this invention comprehensively evaluates the data, data coverage points, and data coverage frequency within that period to determine the specific time when data coverage occurs as reference data, and establishes a multi-dimensional data coverage risk identification model. This model can identify the specific moment when data coverage risk occurs, thereby improving the accuracy and precision of risk assessment.

[0040] Based on reference data, this invention assesses data overlay risk and generates a data overlay risk assessment result. It also determines whether the data to be stored is replaceable data and loads the data to be stored that is determined to be replaceable into the replacement operation queue, thereby avoiding potential data overlay risks, ensuring the stability and consistency of cloud platform data, and reducing data call delays caused by abnormal data overlay. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention;

[0042] Figure 2 This is a system module diagram of the present invention. Detailed Implementation

[0043] The technical solution of this application will be further described in detail below with reference to specific embodiments. The following embodiments are used to illustrate the present invention, but should not be used to limit the scope of protection of the present invention. The conditions in the embodiments can be further adjusted according to specific conditions. Simple improvements to the method of the present invention under the premise of the concept of the present invention are all within the scope of protection claimed by the present invention.

[0044] Example 1: Please refer to Figure 1 This embodiment provides a data storage method for a cloud computing platform based on big data analytics, including the following steps:

[0045] S1. Obtain the data to be stored;

[0046] S2. Classify the data to be stored according to the risk level data contained therein;

[0047] The graded processing includes: obtaining detailed information of the data to be stored; calling the sensitive word detection component to scan the data content, identify and locate the sensitive words contained in the data to be stored; based on the detection results of the sensitive words obtained by scanning, starting the preset risk level judgment function, which quantitatively evaluates the sensitivity of each sensitive word in the detection results, and weights and sums the evaluation scores to obtain the total evaluation score.

[0048] If the total score of the assessment is higher than the preset high-risk threshold, it indicates that the data contains highly sensitive information or violates compliance policies, and the data to be stored is classified as risky data; if the total score of the assessment is not higher than the preset high-risk threshold, it indicates that the data meets the storage security requirements, and the data to be stored is classified as safe data.

[0049] The detailed information of the data to be stored includes the data's own content, risk level data, and metadata such as the data's creation time, source, and format; the risk level data is metadata or tags attached to the data to be stored to identify the degree of potential security risks of the data to be stored; the sensitive word detection component has a built-in dynamically updated sensitive word library related to specific business areas; the preset high-risk threshold can be set to 100.

[0050] The functional relationship of the risk level judgment function includes:

[0051]

[0052] In the formula, The total risk score represents the cumulative risk quantification caused by the inclusion of sensitive words in the data to be stored.

[0053] This indicates the number of sensitive word categories, meaning the number of categories from the set of sensitive words. The total number of different sensitive words detected;

[0054] This represents a set of sensitive words. Indicates the first Sensitive words, This represents the sensitivity weight, which means that for the th... The risk quantification coefficient for each sensitive keyword is preset, and its value is determined by business rules. Indicates the frequency of occurrence, meaning the number of occurrences. The total number of times that sensitive words appear in the data content.

[0055] S3. After the classification is completed, the secure data is transferred to the main storage area, and the risky data is transferred to the backup storage area to achieve isolated storage.

[0056] The data verification process also includes: obtaining the latest data through a data analysis interface and aggregating it into a temporary dataset; and determining the benchmark data that matches the latest data from a benchmark database that stores various types of benchmark data, based on the metadata or content characteristics of the latest data.

[0057] The latest data and the matched baseline data are synchronously transmitted to the data verification interface. The data verification interface performs field-by-field or hash-based comparison operations to form a verification result. If the verification result shows that the two are consistent or within the allowable deviation range, the latest data is collected and organized to construct a standard data set for replacement operations. Among them, when performing hash-based comparison operations, the allowable deviation range is that the number of inconsistent fields is no more than 3.

[0058] S4. Retrieve historical call records from the main storage area;

[0059] To anticipate data overwrite risks, a unified log aggregation service is used to obtain historical call records from the primary storage area. The service will then parse and format the different formatted retrieval request logs from the primary and backup storage areas according to a unified data model, forming a consistent historical call record. The historical call record is a structured log entry that accesses and operates on data in the storage area, containing fields such as timestamp, source, and operation type.

[0060] S5. Based on the acquired historical call records, each historical call record is abstracted into a data node that marks the time of its occurrence on a time series, thus constructing a serialized data foundation for risk assessment.

[0061] S6. Based on the distribution of data nodes in the time series, construct at least one evaluation time period;

[0062] The process of constructing the evaluation time period includes: setting a sliding time window on the time axis, counting the number of nodes within the time window, and taking the time window in which the number of nodes exceeds the frequency threshold as the evaluation time period. Specifically, the sliding time window can be set to 1 minute, and the frequency threshold can be set to 10 times.

[0063] Alternatively, a clustering method based on node density can be used to locate the clustered areas of data access requests in time, and use the clustered areas as the evaluation time period.

[0064] S7. Within the defined assessment period, measure the assessment data; the assessment data consists of a series of statistical indicators extracted to quantify risk, including total number of visits, number of unique IPs, etc.

[0065] S8. Within the defined evaluation period, define the starting time of the evaluation period as the data coverage point and output the specific time location information of the data coverage point.

[0066] S9. Further measure the total number of times each data coverage point is accessed during the evaluation period, as the data coverage count.

[0067] S10. Comprehensively evaluate the data, the location of the data coverage points and the number of data coverages, determine the specific time when the data coverage risk occurs, and define the specific time when the data coverage occurs as reference data.

[0068] S11. Based on the reference data, assess the data coverage risk and generate the data coverage risk assessment results;

[0069] Based on multiple data coverage points, the evaluation operation is performed one by one in descending order of their data coverage frequency. The evaluation operation combines other evaluation data measured, and the data coverage frequency is directly used as the risk score to calculate the risk score corresponding to each data coverage point. The higher the data coverage frequency, the more concentrated the access, and the higher the risk.

[0070] S12. Based on the data coverage risk assessment results, i.e. the risk scores calculated for each data coverage point, determine whether the original data to be stored is replaceable data.

[0071] If the risk score associated with any coverage point exceeds the preset replacement threshold, the original data to be stored related to the coverage risk will be identified as replacement data; otherwise, the original data to be stored related to the coverage risk will be identified as not being replacement data. The replacement threshold can be set to 5.

[0072] S13. If it is determined that the data to be stored is replaceable data, the relevant information is loaded into the replacement operation queue and the replacement process is scheduled to be executed; the relevant information is the information corresponding to the risk data previously stored in the backup storage area.

[0073] When performing a replacement operation, to ensure data consistency and compatibility, the security data selected from the standard dataset for replacement is verified first.

[0074] Verification of security data includes: extracting the verification value corresponding to the security data to be replaced using deterministic algorithms such as hash operation, and comparing it with the verification value of the data record in the standard data set. If the two are consistent, it indicates that the security data is qualified to replace the risky data, and the replacement operation can continue to be executed.

[0075] If there is inconsistency or replacement fails, the original risk data will be retained to ensure that the system does not crash due to differences in data format or content.

[0076] By verifying security data, errors caused by human judgment are minimized while ensuring operational efficiency, and risky data is identified and securely replaced.

[0077] Example 2: Please refer to Figure 2 This embodiment provides a cloud computing platform data storage system based on big data analysis, which includes the following modules.

[0078] The data storage classification module is used to acquire the data to be stored, parse the risk level data contained in the data to be stored, classify the data to be stored according to the risk level data, and perform storage operations on the classified data to be stored.

[0079] The tiered processing includes: obtaining detailed information about the data to be stored; calling a preset risk level judgment function, which determines the risk level of the data based on the sensitivity index in the detection results of sensitive words; content with higher sensitivity will be judged as risky data, while content with lower sensitivity will be judged as safe data;

[0080] The detailed information includes data source, creation time, access permissions, and detection results of sensitive words; the risk level judgment function can be a logical judgment based on a rule engine or a classifier trained by a machine learning model.

[0081] After completing the hierarchical processing, secure data is transferred to the main storage area, while risky data is transferred to a separate backup storage area, thus achieving data isolation and storage.

[0082] The data coverage assessment module is used to analyze the risk of data being accidentally overwritten. Based on the analysis of historical call records in the main storage area, it generates a data coverage risk assessment result for the data to be stored.

[0083] The system retrieves historical call records from the main storage area, constructs data nodes based on these records, and then constructs one or more evaluation time periods based on these data nodes. Within each evaluation time period, the system measures the data coverage points and the number of data coverage events to accurately determine the specific time when the data coverage risk, which serves as the reference data, occurs.

[0084] Based on reference data, the data coverage risk is assessed. The assessment includes: performing assessment operations on relevant historical data coverage events in descending order of the number of times data was covered, and calculating the corresponding risk scores; the risk score is a quantitative indicator used to measure the likelihood that the data to be stored will be covered after being stored in the main storage area; the assessed risk scores and related analysis information are integrated to form the data coverage risk assessment result.

[0085] The risk data processing module is used to process the generated assessment results and make final decisions on potentially high-risk data.

[0086] After receiving the data overwrite risk assessment results, the system determines whether a piece of data to be stored that was previously stored in the main storage area is replaceable data. If the determination result is that it is replaceable data, the data to be stored is identified as replaceable data, and then the data to be stored is loaded into a specific replacement operation queue.

[0087] The data verification module is used to build a data analysis interface, obtain the latest data, and perform comparison operations through the data verification interface to generate verification results.

[0088] The latest data is retrieved from the main storage area or the standby storage area through the data analysis interface. Based on the characteristics of the latest data, such as ID or timestamp, it is matched and benchmark data for comparison is obtained from the benchmark database. The latest data and the matched benchmark data are synchronously transmitted to the data verification interface. Through the data verification interface, a comparison operation is performed to perform hash verification or bit-by-bit comparison on the data content to form a verification result. This verification result can be used to verify whether the data has been damaged or tampered with during storage, ensuring data integrity.

[0089] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A cloud computing platform data storage method based on big data analysis, characterized in that, Includes the following steps: Retrieve the data to be stored; Based on the risk level data contained in the data to be stored, the data to be stored is classified into safe data or risky data. Perform storage operations on the tiered data to be stored; Build a data analysis interface and obtain the latest data; The latest data and the matching baseline data are synchronously transmitted to the data verification interface; A comparison operation is performed through the data verification interface to generate a verification result; Based on the analysis of historical call records in the main storage area, a data overwrite risk assessment result is generated regarding the data to be stored; Based on the data coverage risk assessment results, determine whether the data to be stored is replacement data; If it is determined that the data to be stored is replaceable data, then the data to be stored is loaded into the replacement operation queue; The process of generating a data coverage risk assessment result for the data to be stored based on the analysis of historical call records in the main storage area includes: obtaining historical call records in the main storage area; constructing data nodes based on the historical call records and constructing an assessment time period based on the data nodes; measuring the data coverage points and the number of data coverages within the assessment time period to determine the specific time when data coverage occurs, and defining the specific time as reference data; and assessing the data coverage risk based on the reference data to generate a data coverage risk assessment result.

2. The cloud computing platform data storage method based on big data analysis according to claim 1, characterized in that, The step of classifying the data to be stored into safe data or risky data based on the risk level data contained therein includes: Obtain detailed information about the data to be stored, including detection results for sensitive words; execute a risk level judgment function to determine the risk level based on the sensitivity in the detection results, and classify the data to be stored into safe data or risky data based on the judgment result.

3. The cloud computing platform data storage method based on big data analysis according to claim 1, characterized in that, The storage operation performed on the hierarchical data to be stored includes: Secure data is transferred to the primary storage area, while risky data is transferred to the backup storage area.

4. The cloud computing platform data storage method based on big data analysis according to claim 1, characterized in that, The process of assessing data coverage risk based on reference data to generate a data coverage risk assessment result includes: The assessment is performed in descending order of data coverage frequency; based on the assessment, a risk score is obtained, and the risk score is used as a reference for the data coverage risk assessment results.

5. A cloud computing platform data storage system based on big data analytics, used to execute the cloud computing platform data storage method based on big data analytics as described in any one of claims 1-4, characterized in that, Includes the following modules: The data storage classification module is used to acquire data to be stored; classify the data to be stored according to the risk level data contained in the data to be stored, dividing the data to be stored into safe data or risk data; and perform storage operations on the classified data to be stored. The data coverage assessment module is used to generate a data coverage risk assessment result for the data to be stored based on the analysis of historical call records in the main storage area. The risk data processing module is used to determine whether the data to be stored is replaceable data based on the data coverage risk assessment results. If it is determined that the data to be stored is replaceable data, then the data to be stored is loaded into the replacement operation queue.

6. The cloud computing platform data storage system based on big data analytics according to claim 5, characterized in that, It also includes a data verification module, which is used to build a data analysis interface and obtain the latest data; synchronize the latest data with the matching benchmark data to the data verification interface; and perform comparison operations through the data verification interface to form a verification result.

7. The cloud computing platform data storage system based on big data analytics according to claim 5, characterized in that, The step of classifying the data to be stored into safe data or risky data based on the risk level data contained therein includes: Obtain detailed information about the data to be stored, including detection results for sensitive words; execute a risk level judgment function to determine the risk level based on the sensitivity in the detection results, and classify the data to be stored into safe data or risky data based on the judgment result.

8. The cloud computing platform data storage system based on big data analytics according to claim 5, characterized in that, The storage operation performed on the hierarchical data to be stored includes: Secure data is transferred to the primary storage area, while risky data is transferred to the backup storage area.

9. The cloud computing platform data storage system based on big data analytics according to claim 5, characterized in that, The data overwrite risk assessment results generated based on the analysis of historical call records in the main storage area include: Obtain historical call records from the main storage area; construct data nodes based on the historical call records, and construct an evaluation time period based on the data nodes; evaluate the data coverage points and data coverage frequency within the evaluation time period to determine the specific time when data coverage occurred, and define the specific time as reference data; based on the reference data, assess the data coverage risk to generate a data coverage risk assessment result.

Images