Data storage management system based on cloud server

Abstract

The invention discloses a data storage management system based on a cloud server, particularly relates to the technical field of data storage management, and solves the problem that it is difficult to extract unstructured and semi-structured data features and combine the unstructured and semi-structured data features with structured data features to construct a data type classification model for real-time classification. The technical problems that a value evaluation index value cannot be obtained by integrating the data use frequency, the data key ratio and the data sensitivity degree, and hierarchical storage of various data types cannot be carried out according to the value evaluation index value are solved. Comprising the following modules: a data acquisition module used for deploying a plurality of data acquisition units to acquire data; the data classification module is used for constructing a classification model; the data processing and analysis module is used for analyzing and calculating the data use frequency, the data key ratio and the data sensitivity degree, and obtaining a value evaluation index value simultaneously; and the data storage and backup module is used for storing and distributing the data according to the value evaluation index value.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage management, and in particular to a data storage management system based on a cloud server. Background Art

[0002] With the advent of the big data era, the types of data generated and processed by enterprises are becoming increasingly complex and diverse. Traditional data storage and management solutions are relatively mature for structured data, but they face numerous challenges managing unstructured data, such as text files, images, and audio, as well as semi-structured data in XML and JSON formats. These data are often stored in disparate formats, making efficient storage, retrieval, and analysis difficult. This leads to low data utilization and increased enterprise operating costs. While existing cloud-based data storage management systems have achieved some success in data management, they still face several challenges. They struggle to extract the diverse features of unstructured and semi-structured data and combine them with structured data features to construct data type classification models for real-time classification. They also lack value evaluation metrics based on data usage frequency, key ratios, and sensitivity, and they lack a tiered storage system for various data types based on these metrics. This results in low data storage efficiency, limited processing capabilities for different data types, and a lack of intelligent storage management strategies. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a data storage management system based on a cloud server, which is used to solve the following technical problems:

[0004] Although the existing cloud server-based data storage management system has achieved data storage management to a certain extent, there are still some problems. It is difficult to extract the various features contained in unstructured data and semi-structured data and combine them with structured data features to build a data type classification model to classify data types in real time. There is a lack of comprehensive data usage frequency, data key ratio and data sensitivity to obtain value evaluation index values. There is also a lack of hierarchical storage of various data types according to value evaluation index values, resulting in low data storage efficiency, limited processing capabilities for different types of data, and a lack of intelligent storage management strategies.

[0005] To solve the above problems, the present invention provides a data storage management system based on a cloud server, comprising the following modules:

[0006] Data collection module: deploys structured data collection units, unstructured data collection units, and semi-structured data collection units to collect data;

[0007] Data classification module: builds a classification model to classify the data types in the three acquisition units;

[0008] Data processing and analysis module: After analyzing and calculating the data usage frequency, data key ratio and data sensitivity of the three data types, the values are added together to obtain the value evaluation index values of the corresponding data types;

[0009] Data storage and backup module: Sends processed data to the cloud storage resource layer for storage, allocates data according to value evaluation index values, and regularly controls and backs up data according to preset access control and backup policies.

[0010] Preferably, the deploying of the structured data collection unit, the unstructured data collection unit and the semi-structured data collection unit to collect data comprises the following steps:

[0011] The data collection module is constructed by three data type collection units. The structured data collection unit connects to the company's existing business database and captures the structured data generated by business operations in real time through the database interface; the unstructured data collection unit deploys a file crawler program to traverse and collect documents, pictures, and audio resources in the company's internal file servers and cloud storage; the semi-structured data collection unit uses the corresponding parser to extract key data elements from configuration information and log files stored in a specified format, and incorporates the semi-structured data into a unified collection system. The specified format refers to XML and JSON formats.

[0012] Preferably, the classification model is constructed to classify the data in the three acquisition units, comprising the following steps:

[0013] Collect samples of structured, unstructured and semi-structured data from the data sources of three units, and formulate corresponding cleaning rules and operations for different types of data; unstructured data includes text, images and audio. The pre-trained Word2Vec model is used for text data. After taking the vector of each word in the text, the average vector is calculated to obtain the numerical representation of the text; for image data, the input image is adjusted to a fixed size and normalized. The pre-trained convolutional neural network is used as the backbone network. Feature maps are extracted from different layers of the backbone network, deep features are upsampled, and the upsampled deep features are added to the shallow features to obtain the fused feature map, which is output in multiple sizes. The multi-scale features output by the convolutional neural network are fused and normalized to generate a numerical representation of the image; the audio file is loaded using the audio processing library, and the time domain, frequency domain, and time-frequency domain features of the audio are extracted. The extracted features are normalized, and after splicing the processed multiple features, the different features are weighted summed. Principal component analysis is used to reduce the dimension and fuse the features, and the fused features are used as the numerical representation of the audio; semi-structured data includes data in JOSN and XML formats. The key values of the JOSN data are extracted, the nested fields are flattened, and the XML data content is extracted using the XPath tool and converted into a table format;

[0014] Use semi-automatic annotation tools to annotate the cleaned data, develop annotation guidelines, determine the annotation content of various types of data, extract features from fields in structured data, text values, image values, and audio values in unstructured data, and key fields after parsing in semi-structured data, use a neural network framework to build a data type classification model, input the extracted features into the model, train the model with the data collected for the training set, and deploy the trained model to the server for real-time classification.

[0015] Preferably, the data usage frequency includes the following steps:

[0016] Calculate the value evaluation indicators of structured data, unstructured data, and semi-structured data separately, where the value evaluation indicators include data usage frequency, data key ratio, and data sensitivity;

[0017] During the working days of the past month, count the number of times and duration of each type of data set being called, viewed, processed, or used in other ways. Add these to get the total number of times and duration of use, respectively. Divide the total number of times the data is used by the total number of days and the duration of use by the total duration to get the frequency of data use.

[0018] Data usage frequency calculation formula:

[0019]

[0020] Among them, U represents the frequency of data usage, N b Indicates the number of times it is called, N s Indicates the number of times it has been viewed, N w Indicates the number of times it is processed, N o Indicates the number of times other methods are used, D work Indicates the total number of working days in the past month, T b Indicates the time when it is called, T s Indicates the time of viewing, T w Indicates the processing time, T o indicates the time spent using other methods, and T indicates the total time spent on working days in the past month.

[0021] Preferably, the data key ratio comprises the following steps:

[0022] Collect relevant business goals from the company's strategic planning documents, annual reports, and business department plans, classify and organize the collected goals, and categorize the collected goals according to the time dimension, including short-term goals, medium-term goals, and long-term goals. Break down each business goal into multiple measurable sub-goals and identify the key success factors for each sub-goal. Build a hierarchical model with the company's overall goals at the top level, sub-goals for each business area in the middle level, and the key success factors for each sub-goal at the bottom level. Label the goals at different levels and the corresponding time attributes. Use the labeled training data to train the hierarchical model and deploy the trained model to the server.

[0023] Determine the dependencies between goals at all levels, analyze the dependencies, and use the key path method to find the goal sequence that plays a decisive role in the overall goal as the key goal path;

[0024] Map the collected data to the goals and key success factors in the goal hierarchy to determine whether the corresponding key success factors are on the key goal path. Count the number of nodes on the key goal path and the total number of nodes on the key goal path, and divide them to get the data criticality ratio.

[0025] Data key ratio calculation formula:

[0026]

[0027] Among them, C represents the data key ratio, C G represents the number of nodes on the critical target path, C T Indicates the total number of nodes on the critical target path.

[0028] Preferably, the data sensitivity comprises the following steps:

[0029] Data information types are divided into personal identity information, financial information, health information, and commercial secrets. Sensitivity base values for different data information types are defined. According to specified rules, the proportion of the data volume of different data information types in the total data volume is assigned a data volume factor. Risk factors are assigned according to the severity of leakage and theft of different data information types, and ease of use factors are assigned according to the ease of access and use of different data information types. The sensitivity base values of different data information types are normalized and then combined with the data volume factor, risk factor, and ease of use factor to obtain the data sensitivity of the corresponding data information type.

[0030] Data sensitivity calculation formula:

[0031]

[0032] Among them, M represents the sensitivity of the data, B represents the sensitivity base value, and B min Indicates the minimum value of sensitivity basis, B max represents the maximum value of the sensitivity basis, L represents the data volume factor, F represents the risk factor, Y represents the ease of use factor, and α, β and γ represent the corresponding weight coefficients.

[0033] Preferably, the value evaluation index value includes the following steps:

[0034] The value evaluation index value is obtained by adding the data usage frequency, data key ratio and data sensitivity. The value evaluation index value calculation formula is:

[0035] V=U+C+M

[0036] Among them, V represents the value evaluation index value, U represents the frequency of data use, C represents the data criticality ratio, and M represents the data sensitivity;

[0037] Collect the value evaluation index values of different data types and sort them in ascending order using the sorting function of spreadsheet software. According to the distribution of the sorted data, determine the three-level division criteria, with the 33% and 67% quantiles as the dividing points;

[0038] According to the determined classification labels, the value evaluation index values of different data types are graded:

[0039] The value evaluation index values between the 0%-33% quantiles are regarded as the primary data;

[0040] The value evaluation index values between the 34%-67% quantiles are used as secondary data;

[0041] The evaluation index values between 68% and 100% quantiles are regarded as the third-level data.

[0042] Preferably, the processing data is sent to the cloud storage resource layer for storage, the data is allocated according to the value evaluation index value, and the data is regularly controlled and backed up according to the preset access control backup strategy, including the following steps:

[0043] For structured data, high-performance database services are used for primary data, standard database services are used for secondary data, and low-cost database services are used for tertiary data. For unstructured data, primary data is stored in the high-performance tier of object storage, secondary data is stored in the standard tier of object storage, and tertiary data is stored in the low-cost tier of object storage. For semi-structured data, high-performance NoSQL databases are used for primary data, standard NoSQL databases are used for secondary data, and low-cost NoSQL databases are used for tertiary data.

[0044] At the same time, strict access control and high-frequency backup are set for the first-level data corresponding to all data types, medium-intensity access control and regular backup are set for the second-level data, and basic access control and low-frequency backup are set for the third-level data.

[0045] Beneficial effects of the present invention:

[0046] The present invention extracts and analyzes features of text, image, and audio data in unstructured data to obtain numerical representations of text, image, and audio, and extracts and analyzes features of JOSN and XML format data in semi-structured data to obtain key numerical representations. These features are then combined with features in structured data to construct a data type classification model based on a neural network to classify data in real time. This achieves precise feature extraction for different data types, learns the associations and distinctions between different features, and thus classifies data types more accurately.

[0047] The present invention obtains data usage frequency by taking the average of the average number of daily uses and the data usage time. A hierarchical model is constructed, with the top layer representing the enterprise's overall goal, the middle layer representing the sub-goals of each business area, and the bottom layer representing the key success factors of each sub-goal. The model is deployed on a server, and a target sequence that plays a decisive role in the overall goal is identified as a key target path by combining target dependencies and a key path method. The data key ratio is obtained based on the proportion of the target in the key directory path. The data sensitivity is obtained by dividing data information types and formulating rules to assign sensitivity base values, normalizing them, and combining multiple factors. The data usage frequency, data key ratio, and data sensitivity are added together to obtain a value evaluation index value. Different data types are graded according to the value evaluation index value and stored in corresponding databases, thus realizing multi-dimensional calculation of the value evaluation index value. The data usage frequency reflects the frequency of data application in daily business; the data key ratio reflects the importance of data in the key target path; and the data sensitivity focuses on the security and sensitivity of the data. This multi-dimensional comprehensive evaluation can more comprehensively and objectively quantify the value of data, avoiding the one-sidedness that may be caused by single-dimensional evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the module flow of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0050] See also Figure 1 As shown, the present invention is a cloud server-based data storage management system, including the following modules:

[0051] Data collection module: deploys structured data collection units, unstructured data collection units, and semi-structured data collection units to collect data;

[0052] Data classification module: builds a classification model to classify the data types in the three acquisition units;

[0053] Data processing and analysis module: After analyzing and calculating the data usage frequency, data key ratio and data sensitivity of the three data types, the values are added together to obtain the value evaluation index values of the corresponding data types;

[0054] Data storage and backup module: Sends processed data to the cloud storage resource layer for storage, allocates data according to value evaluation index values, and regularly controls and backs up data according to preset access control and backup policies.

[0055] Specifically, deploy structured data collection units, unstructured data collection units, and semi-structured data collection units to collect data; extract features from fields in structured data, text values, image values, and audio values in unstructured data, and parsed key fields in semi-structured data, and input the extracted features into the model to construct a data type classification model; obtain the data usage frequency by dividing the total number of data usage times by the total number of days and the data usage time by the total time, determine whether the corresponding key success factor is on the key target path, and count the number of nodes on the key target path and the total number of nodes on the key target path, divide them to obtain the data key ratio, normalize the sensitivity base values of different data information types, and combine them with the data volume factor, risk factor, and ease of use factor to obtain the data sensitivity of the corresponding data information type, and add the data usage frequency, data key ratio, and data sensitivity to obtain the value evaluation index value; classify different data types according to the value evaluation index value and store them, including primary data, secondary data, and tertiary data.

[0056] In one embodiment of the present invention, the deploying of the structured data collection unit, the unstructured data collection unit, and the semi-structured data collection unit to collect data includes the following steps:

[0057] The data collection module is constructed by three data type collection units. The structured data collection unit connects to the company's existing business database and captures the structured data generated by business operations in real time through the database interface; the unstructured data collection unit deploys a file crawler program to traverse and collect documents, pictures, and audio resources in the company's internal file servers and cloud storage; the semi-structured data collection unit uses the corresponding parser to extract key data elements from configuration information and log files stored in a specified format, and incorporates the semi-structured data into a unified collection system. The specified format refers to XML and JSON formats.

[0058] Specifically, determine the connection information of the company's existing business database, use the corresponding database connection driver, establish a connection with the business database in the data collection program, determine the database tables, fields and data change events involved in the structured data generated by the business operations that need to be captured based on business needs, and transfer the captured structured data to the temporary storage area of the data collection module; determine the documents, pictures, and audio in the company's internal file servers and cloud storage, select a suitable file crawler framework, write a crawler program based on the resource storage path and directory structure, start the file crawler program, traverse the file resources in the company's internal file servers and cloud storage from the specified starting path, determine the file type based on its extension, and extract the content of the file. The content or basic information is organized according to the set data format; according to the enterprise's technical architecture and business system deployment, the server location, directory path and file naming rules for storing configuration information and log files are determined; for XML format configuration information and log files, a suitable XML parser is selected; for JSON format files, the corresponding JSON parsing library is used; according to the structural characteristics of the configuration files and log files, parsing rules and scripts are written, the parsing program is started, and the located XML and JSON format files are parsed according to the parsing rules, key data elements are extracted from the parsing results, and the extracted key data elements are organized according to the pre-designed data structure and format so as to be included in a unified collection system.

[0059] In one embodiment of the present invention, the step of constructing a classification model to classify the data in the three acquisition units includes the following steps:

[0060] Collect samples of structured, unstructured and semi-structured data from the data sources of three units, and formulate corresponding cleaning rules and operations for different types of data; unstructured data includes text, images and audio. The pre-trained Word2Vec model is used for text data. After taking the vector of each word in the text, the average vector is calculated to obtain the numerical representation of the text; for image data, the input image is adjusted to a fixed size and normalized. The pre-trained convolutional neural network is used as the backbone network. Feature maps are extracted from different layers of the backbone network, deep features are upsampled, and the upsampled deep features are added to the shallow features to obtain the fused feature map, which is output in multiple sizes. The multi-scale features output by the convolutional neural network are fused and normalized to generate a numerical representation of the image; the audio file is loaded using the audio processing library, and the time domain, frequency domain, and time-frequency domain features of the audio are extracted. The extracted features are normalized, and after splicing the processed multiple features, the different features are weighted summed. Principal component analysis is used to reduce the dimension and fuse the features, and the fused features are used as the numerical representation of the audio; semi-structured data includes data in JOSN and XML formats. The key values of the JOSN data are extracted, the nested fields are flattened, and the XML data content is extracted using the XPath tool and converted into a table format;

[0061] Use semi-automatic annotation tools to annotate the cleaned data, develop annotation guidelines, determine the annotation content of various types of data, extract features from fields in structured data, text values, image values, and audio values in unstructured data, and key fields after parsing in semi-structured data, use a neural network framework to build a data type classification model, input the extracted features into the model, train the model with the data collected for the training set, and deploy the trained model to the server for real-time classification.

[0062] Specifically, the collected text data is preprocessed and word segmentation is performed, the pre-trained Word2Vec model is loaded, the average value of all word vectors is calculated, and the numerical representation vector of the text is obtained as the feature vector of the text data for subsequent processing; the collected image data is uniformly adjusted to a fixed size, such as 224×224 pixels, and normalized, the pixel value range is mapped to the interval [0,1] or [-1,1], the pre-trained convolutional neural network is selected as the backbone network, the resized and normalized image is input into the backbone network, feature maps are extracted from different layers of the backbone network, the deep features are upsampled, and then the upsampled deep features are added to the corresponding shallow features to obtain the fused feature map, and feature maps of multiple scales are output, the feature maps of all scales are spliced together, the feature maps of different scales are weighted and summed, the fused features are normalized to make their numerical ranges consistent, and the numerical representation vector of the image is generated; use the audio processing library to add Upload audio files, extract the time domain features, frequency domain features and time-frequency domain features of the audio, normalize the extracted features, eliminate dimensional differences, splice the processed features to form a complete audio feature vector, set weights according to the importance of different features, perform weighted summation on the spliced feature vectors, and then use principal component analysis to further reduce the dimension and fuse the features to obtain the final audio numerical representation vector; clean the extracted JSON data key-value pairs, and flatten the nested fields. For example, for JSON configuration information containing multiple layers of nesting, expand the nested structure into a single-layer key-value pair form to facilitate subsequent processing and analysis. After using the XPath tool to extract the XML data content, convert it into a tabular form for storage according to established rules, such as converting XML format log data into a two-dimensional table form, where each row represents a log record and each column represents a different data field; formulate a detailed annotation guide to clarify the annotation content and rules for various types of data. For example, for structured data, label its data type and the specific meaning of each field; for text values, image values, and audio values in unstructured data, label their corresponding numerical feature vectors; for the parsed key fields in semi-structured data, label their semantics and functions in the original data, select a suitable neural network framework, build a data type classification model, use the extracted structured data features, unstructured data features, and semi-structured data features as the input of the model, and the model outputs the classification results of the data. After the classification model training is completed, select a suitable cloud computing server for model deployment.

[0063] In one embodiment of the present invention, the data usage frequency includes the following steps:

[0064] Calculate the value evaluation indicators of structured data, unstructured data, and semi-structured data separately, where the value evaluation indicators include data usage frequency, data key ratio, and data sensitivity;

[0065] During the working days of the past month, count the number of times and duration of each type of data set being called, viewed, processed, or used in other ways. Add these to get the total number of times and duration of use, respectively. Divide the total number of times the data is used by the total number of days and the duration of use by the total duration to get the frequency of data use.

[0066] Data usage frequency calculation formula:

[0067]

[0068] Among them, U represents the frequency of data usage, N b Indicates the number of times it is called, N s Indicates the number of times it has been viewed, N w Indicates the number of times it is processed, N o Indicates the number of times other methods are used, D work Indicates the total number of working days in the past month, T b Indicates the time when it is called, T s Indicates the time of viewing, T w Indicates the processing time, T o indicates the time spent using other methods, and T indicates the total time spent on working days in the past month.

[0069] Specifically, for structured, unstructured, and semi-structured datasets involved, within the domestic working day timeframe of the past month, detailed records should be made of the specific time points and duration at which each dataset was called, viewed, processed, and used in other ways, using system logs, the application's built-in logging function, or manual recording. The recorded information should be accurate and complete, including the date, time, dataset type, method of use, and duration of use. This can be recorded in a table format. Based on the collected recorded data, statistics can be compiled for different types of datasets. For each dataset type, the total number and total duration of its calls, views, processing, and other uses on weekdays in the past month are calculated. The total number of weekdays in the past month and the total duration of weekday use for all datasets in the past month are determined. The total duration of use for each dataset type obtained previously is added together to obtain the total data usage time. For each dataset type, the total number of uses in the past month is divided by the total number of weekdays to obtain the average daily usage of that dataset type. The total duration of use for each dataset type in the past month is divided by the total duration of use for all datasets to obtain the percentage of the total usage time for that dataset type. The frequency of data usage is determined by combining the two calculation results above, namely the average daily usage number and the percentage of data usage time.

[0070] In one embodiment of the present invention, the data key ratio comprises the following steps:

[0071] Collect relevant business goals from the company's strategic planning documents, annual reports, and business department plans, classify and organize the collected goals, and categorize the collected goals according to the time dimension, including short-term goals, medium-term goals, and long-term goals. Break down each business goal into multiple measurable sub-goals and identify the key success factors for each sub-goal. Build a hierarchical model with the company's overall goals at the top level, sub-goals for each business area in the middle level, and the key success factors for each sub-goal at the bottom level. Label the goals at different levels and the corresponding time attributes. Use the labeled training data to train the hierarchical model and deploy the trained model to the server.

[0072] Determine the dependencies between goals at all levels, analyze the dependencies, and use the key path method to find the goal sequence that plays a decisive role in the overall goal as the key goal path;

[0073] Map the collected data to the goals and key success factors in the goal hierarchy to determine whether the corresponding key success factors are on the key goal path. Count the number of nodes on the key goal path and the total number of nodes on the key goal path, and divide them to get the data criticality ratio.

[0074] Data key ratio calculation formula:

[0075]

[0076] Among them, C represents the data key ratio, C G represents the number of nodes on the critical target path, C T Indicates the total number of nodes on the critical target path.

[0077] Specifically, goals should be categorized by timeframe: short-term (less than one year), medium-term (one to three years), and long-term (more than three years). For example, a goal like "increase sales by 10% this quarter" is a short-term goal; "launch three new products in the next two years" is a medium-term goal; and "become a top three company in the industry in market share within five years" is a long-term goal. Keep a detailed record of each goal, including information such as the goal content, the business area it belongs to, and the timeframe, to form a preliminary list of goals. For each business goal, break it down into multiple measurable sub-goals. For example, the goal of "increasing market share" can be broken down into sub-goals such as "increasing the number of new customers," "improving customer loyalty," and "expanding new sales channels." For each sub-goal, analyze the key success factors required to achieve it. For example, for the sub-goal of "increasing the number of new customers," key success factors may include "advertising effectiveness," "sales team lead conversion rate," "product competitiveness," etc. A hierarchical model is constructed, with the top layer representing the company's overall goal, the middle layer representing the sub-goals of each business area, and the bottom layer representing the key success factors for each sub-goal. The model clarifies the attribution between the various layers, labels the goals at different layers with their corresponding time attributes, collects relevant data for training sets, including historical business data and market trend data, trains the hierarchical model with the labeled data, and deploys the trained model to the server for use in actual business environments. Analyze the dependencies between goals at different levels, that is, whether the achievement of one goal depends on the completion of other goals. For example, "launching new products" may depend on goals such as "research and development success" and "production process improvement". Combined with the key path method, the target sequence that plays a decisive role in the overall goal is found from many target sequences as the key target path. The collected data is mapped with each target and key success factor in the target hierarchy. By comparing the key success factors with the targets in the key target path, it is determined whether their position is on the key target path. The number of nodes on the key target path and the total number of nodes on the key target path are counted and divided to obtain the data criticality ratio.

[0078] In one embodiment of the present invention, determining the sensitivity of the data includes the following steps:

[0079] Data information types are divided into personal identity information, financial information, health information, and commercial secrets. Sensitivity base values for different data information types are defined. According to specified rules, the proportion of the data volume of different data information types in the total data volume is assigned a data volume factor. Risk factors are assigned according to the severity of leakage and theft of different data information types, and ease of use factors are assigned according to the ease of access and use of different data information types. The sensitivity base values of different data information types are normalized and then combined with the data volume factor, risk factor, and ease of use factor to obtain the data sensitivity of the corresponding data information type.

[0080] Data sensitivity calculation formula:

[0081]

[0082] Among them, M represents the sensitivity of the data, B represents the sensitivity base value, and B min Indicates the minimum value of sensitivity basis, B max represents the maximum value of the sensitivity basis, L represents the data volume factor, F represents the risk factor, Y represents the ease of use factor, and α, β and γ represent the corresponding weight coefficients.

[0083] Specifically, data is clearly divided into four types: personal identity information, financial information, health information, and trade secrets. Based on the general sensitivity characteristics of the data, a sensitivity base value is defined for each data information type. For example, the minimum sensitivity is set to 1 and the maximum sensitivity is set to 10. Assuming that the sensitivity base value of personal identity information is 7, financial information is 9, health information is 8, and trade secrets is 10, this sensitivity base value reflects the sensitivity of this type of data when it is not affected by other factors. The proportion of the data volume of different data information types in the total data volume is determined. This proportion can be obtained through data analysis and statistics. According to certain proportion rules, these proportion values are converted into data volume factors. For example, the proportion value can be directly used as the data volume factor to evaluate the severity of leakage and theft of different data information types to assign a risk factor. For example, the risk factor can be assigned between 0.1 and 1. Assuming that the risk factor of personal identity information is 0.7, financial information is 0.9, health information is 0.8, and trade secrets is 1, the ease of access and use of different data information types is analyzed to assign an ease of use factor. Within a given numerical range, the ease of use factor is assigned. For example, an ease of use factor is assigned to each data information type, ranging from 1 to 10. The larger the value, the easier it is to access and use. Assuming that the ease of use factor of personal identity information is 6, financial information is 4, health information is 3, and commercial secrets is 2, the sensitivity base values of different data information types are normalized so that they can be calculated jointly with other factors. The normalized sensitivity base value is combined with the data volume factor, risk factor, and ease of use factor to obtain the data sensitivity of the corresponding data information type; among them, the value of α is 0.3, the value of β is 0.4, and the value of γ is 0.3.

[0084] In one embodiment of the present invention, the value evaluation index value includes the following steps:

[0085] The value evaluation index value is obtained by adding the data usage frequency, data key ratio and data sensitivity. The value evaluation index value calculation formula is:

[0086] V=U+C+M

[0087] Among them, V represents the value evaluation index value, U represents the frequency of data use, C represents the data criticality ratio, and M represents the data sensitivity;

[0088] Collect the value evaluation index values of different data types and sort them in ascending order using the sorting function of spreadsheet software. According to the distribution of the sorted data, determine the three-level division criteria, with the 33% and 67% quantiles as the dividing points;

[0089] According to the determined classification labels, the value evaluation index values of different data types are graded:

[0090] The value evaluation index values between the 0%-33% quantiles are regarded as the primary data;

[0091] The value evaluation index values between the 34%-67% quantiles are used as secondary data;

[0092] The evaluation index values between 68% and 100% quantiles are regarded as the third-level data.

[0093] Specifically, enter the value evaluation index values of different data types into the spreadsheet software, select the column where the value evaluation index value is located, click the "Sort" button in the "Data" tab, select the "Ascending" sorting method in the pop-up dialog box, and then confirm. Calculate the 33% and 67% quantiles of the sorted data, and use the calculated 33% and 67% quantiles as the dividing points for the three levels. Level one data represents high value, high sensitivity, and high frequency of use; level two data represents medium value, medium sensitivity, and medium frequency of use; level three data represents low value, low sensitivity, and low frequency of use. According to the determined value evaluation index value and level division standard, the value evaluation index value of each data type is graded.

[0094] In one embodiment of the present invention, the processing data is sent to the cloud storage resource layer for storage, the data is allocated according to the value evaluation index value, and the data is regularly controlled and backed up according to the preset access control backup policy, including the following steps:

[0095] For structured data, high-performance database services are used for primary data, standard database services are used for secondary data, and low-cost database services are used for tertiary data. For unstructured data, primary data is stored in the high-performance tier of object storage, secondary data is stored in the standard tier of object storage, and tertiary data is stored in the low-cost tier of object storage. For semi-structured data, high-performance NoSQL databases are used for primary data, standard NoSQL databases are used for secondary data, and low-cost NoSQL databases are used for tertiary data.

[0096] At the same time, strict access control and high-frequency backup are set for the first-level data corresponding to all data types, medium-intensity access control and regular backup are set for the second-level data, and basic access control and low-frequency backup are set for the third-level data.

[0097] Specifically, select a suitable high-performance relational database, install and configure the selected database on a high-performance server, and perform initial setup, including creating a database instance, allocating storage space, and configuring memory and CPU resources. Migrate existing primary structured data from the original storage location to the newly deployed high-performance database, create database user groups with different permissions, assign specific access rights to each user group, and configure a high-frequency backup strategy, such as daily full backups and incremental backups every few hours. Select a standard relational database that meets business needs, install and configure the database on a standard server, perform necessary initial setup, such as setting the character set and creating database users, migrate secondary structured data to the standard database, create a user group with moderate access rights, such as restricting access to specific IP address segments and encrypting certain sensitive fields, and configure a regular backup strategy, such as weekly full backups. Install and configure the database on lower-cost server resources, perform basic initial setup, migrate tertiary structured data to a low-cost database, set up basic access controls, such as a simple username and password verification mechanism, and configure a low-frequency backup strategy, such as monthly full backups.

[0098] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A cloud server-based data storage management system, characterized in that: Includes the following modules: Data collection module: deploys structured data collection units, unstructured data collection units, and semi-structured data collection units to collect data; Data classification module: builds a classification model to classify the data types in the three acquisition units; Data processing and analysis module: After analyzing and calculating the data usage frequency, data key ratio and data sensitivity of the three data types, the values are added together to obtain the value evaluation index values of the corresponding data types; Data storage and backup module: Sends processed data to the cloud storage resource layer for storage, allocates data according to value evaluation index values, and regularly controls and backs up data according to preset access control and backup policies.

2. A cloud server-based data storage management system according to claim 1, characterized in that: The deployment of the structured data collection unit, the unstructured data collection unit, and the semi-structured data collection unit to collect data includes the following steps: The data collection module is constructed by three data type collection units. The structured data collection unit connects to the company's existing business database and captures the structured data generated by business operations in real time through the database interface; the unstructured data collection unit deploys a file crawler program to traverse and collect documents, pictures, and audio resources in the company's internal file servers and cloud storage; the semi-structured data collection unit uses the corresponding parser to extract key data elements from configuration information and log files stored in a specified format, and incorporates the semi-structured data into a unified collection system. The specified format refers to XML and JSON formats.

3. A cloud server-based data storage management system according to claim 1, characterized in that: The construction of the classification model to classify the data in the three acquisition units includes the following steps: Collect samples of structured, unstructured and semi-structured data from the data sources of three units, and formulate corresponding cleaning rules and operations for different types of data; unstructured data includes text, images and audio. The pre-trained Word2Vec model is used for text data. After taking the vector of each word in the text, the average vector is calculated to obtain the numerical representation of the text; for image data, the input image is adjusted to a fixed size and normalized. The pre-trained convolutional neural network is used as the backbone network. Feature maps are extracted from different layers of the backbone network, deep features are upsampled, and the upsampled deep features are added to the shallow features to obtain the fused feature map, which is output in multiple sizes. The multi-scale features output by the convolutional neural network are fused and normalized to generate a numerical representation of the image; the audio file is loaded using the audio processing library, and the time domain, frequency domain, and time-frequency domain features of the audio are extracted. The extracted features are normalized, and after splicing the processed multiple features, the different features are weighted summed. Principal component analysis is used to reduce the dimension and fuse the features, and the fused features are used as the numerical representation of the audio; semi-structured data includes data in JOSN and XML formats. The key values of the JOSN data are extracted, the nested fields are flattened, and the XML data content is extracted using the XPath tool and converted into a table format; Use semi-automatic annotation tools to annotate the cleaned data, develop annotation guidelines, determine the annotation content of various types of data, extract features from fields in structured data, text values, image values, and audio values in unstructured data, and key fields after parsing in semi-structured data, use a neural network framework to build a data type classification model, input the extracted features into the model, train the model with the data collected for the training set, and deploy the trained model to the server for real-time classification.

4. A cloud server-based data storage management system according to claim 1, characterized in that: The data usage frequency includes the following steps: Calculate the value evaluation indicators of structured data, unstructured data, and semi-structured data separately, where the value evaluation indicators include data usage frequency, data key ratio, and data sensitivity; During the working days of the past month, count the number of times and duration of each type of data set being called, viewed, processed, or used in other ways. Add these to get the total number of times and duration of use, respectively. Divide the total number of times the data is used by the total number of days and the duration of use by the total duration to get the frequency of data use. Data usage frequency calculation formula: Among them, U represents the frequency of data usage, N b Indicates the number of times it is called, N s Indicates the number of times it has been viewed, N w Indicates the number of times it is processed, N o Indicates the number of times other methods are used, D work Indicates the total number of working days in the past month, T b Indicates the time when it is called, T s Indicates the time of viewing, T w Indicates the processing time, T o indicates the time spent using other methods, and T indicates the total time spent on working days in the past month.

5. A cloud server-based data storage management system according to claim 1, characterized in that: The data key ratio comprises the following steps: Collect relevant business goals from the company's strategic planning documents, annual reports, and business department plans, classify and organize the collected goals, and categorize the collected goals according to the time dimension, including short-term goals, medium-term goals, and long-term goals. Break down each business goal into multiple measurable sub-goals and identify the key success factors for each sub-goal. Build a hierarchical model with the company's overall goals at the top level, sub-goals for each business area in the middle level, and the key success factors for each sub-goal at the bottom level. Label the goals at different levels and the corresponding time attributes. Use the labeled training data to train the hierarchical model and deploy the trained model to the server. Determine the dependencies between goals at all levels, analyze the dependencies, and use the key path method to find the goal sequence that plays a decisive role in the overall goal as the key goal path; Map the collected data to the goals and key success factors in the goal hierarchy to determine whether the corresponding key success factors are on the key goal path. Count the number of nodes on the key goal path and the total number of nodes on the key goal path, and divide them to get the data criticality ratio. Data key ratio calculation formula: Among them, C represents the data key ratio, C G represents the number of nodes on the critical target path, C T Indicates the total number of nodes on the critical target path.

6. A cloud server-based data storage management system according to claim 1, characterized in that: The data sensitivity level includes the following steps: Data information types are divided into personal identity information, financial information, health information, and commercial secrets. Sensitivity base values for different data information types are defined. According to specified rules, the proportion of the data volume of different data information types in the total data volume is assigned a data volume factor. Risk factors are assigned according to the severity of leakage and theft of different data information types, and ease of use factors are assigned according to the ease of access and use of different data information types. The sensitivity base values of different data information types are normalized and then combined with the data volume factor, risk factor, and ease of use factor to obtain the data sensitivity of the corresponding data information type. Data sensitivity calculation formula: Among them, M represents the sensitivity of the data, B represents the sensitivity base value, and B min Indicates the minimum value of sensitivity basis, B max represents the maximum value of the sensitivity basis, L represents the data volume factor, F represents the risk factor, Y represents the ease of use factor, and α, β and γ represent the corresponding weight coefficients.

7. A cloud server-based data storage management system according to claim 1, characterized in that: The value evaluation index value includes the following steps: The value evaluation index value is obtained by adding the data usage frequency, data key ratio and data sensitivity. The value evaluation index value calculation formula is: V=U+C+M Among them, V represents the value evaluation index value, U represents the frequency of data use, C represents the data criticality ratio, and M represents the data sensitivity; Collect the value evaluation index values of different data types and sort them in ascending order using the sorting function of spreadsheet software. According to the distribution of the sorted data, determine the three-level division criteria, with the 33% and 67% quantiles as the dividing points; According to the determined classification labels, the value evaluation index values of different data types are graded: The value evaluation index values between the 0%-33% quantiles are regarded as the primary data; The value evaluation index values between the 34%-67% quantiles are used as secondary data; The evaluation index values between 68% and 100% quantiles are regarded as the third-level data.

8. The cloud server-based data storage management system according to claim 1, characterized in that: The processed data is sent to the cloud storage resource layer for storage, the data is allocated according to the value evaluation index value, and the data is regularly controlled and backed up according to the preset access control backup strategy, including the following steps: For structured data, high-performance database services are used for primary data, standard database services are used for secondary data, and low-cost database services are used for tertiary data. For unstructured data, primary data is stored in the high-performance tier of object storage, secondary data is stored in the standard tier of object storage, and tertiary data is stored in the low-cost tier of object storage. For semi-structured data, high-performance NoSQL databases are used for primary data, standard NoSQL databases are used for secondary data, and low-cost NoSQL databases are used for tertiary data. At the same time, strict access control and high-frequency backup are set for the first-level data corresponding to all data types, medium-intensity access control and regular backup are set for the second-level data, and basic access control and low-frequency backup are set for the third-level data.

Images