Excel data sharing management method and system, electronic equipment and storage medium

Abstract

The invention discloses an Excel data sharing management method and system, electronic equipment and a storage medium. The Excel data sharing management method comprises the following steps: constructing a data lake; creating an index of the data lake; adding the data with the added index to a data lake; and sharing the data of the data lake. According to the method, data is graded on a data lake platform, consequences occurring when the data are damaged are estimated, the degree of damage caused by data damage is analyzed, and the data are divided and graded according to the importance of the data, for example, the data are divided from high to low according to the degree of damage after damage, and the data are classified according to the sensitivity and importance of the data. Data classification levels are defined as different levels: first-level data, second-level data and third-level data, and the first-level data are highly sensitive and important data, such as personal privacy information, enterprise core commercial secret and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of data sharing, and in particular to an Excel data sharing management method, system, electronic equipment and storage medium. Background Art

[0002] Part of the value of a data lake is to bring together different types of data, and another part of the value is that data analysis can be performed without the need for predefined models. Today's big data architecture is scalable and can provide users with more and more real-time analysis. While business intelligence (BI) and data warehouses have not been eliminated, big data analysis and big data lakes are developing into more types of real-time intelligent services that can support real-time decision making.

[0003] Data persistence and security are priorities for data lakes. Many options can deliver a reasonable cost, but not all can meet the long-term storage needs of data lakes. The challenge is that much of the data in a data lake will never be deleted. The value of this data is that it is used for analysis and comparison with data year after year, which will offset its capacity costs, while information security is often overlooked. The data lake architecture by definition puts all eggs in one basket. If the security of the repository is compromised, then unknown parties may have access to all data, but storing all data in different repositories in different levels and performing different levels of encrypted access can greatly improve security. Summary of the invention

[0004] In view of the shortcomings existing in the above problems, the present invention provides an Excel data sharing management method, system, electronic device and storage medium.

[0005] To achieve the above object, the present invention provides an Excel data sharing management method, comprising:

[0006] Build a data lake;

[0007] Creating an index for the data lake;

[0008] Adding the data with the index added to the data lake;

[0009] Sharing the data in the data lake;

[0010] Wherein, constructing the data lake includes:

[0011] Collect various types of data;

[0012] Processing the data, including data cleaning, data conversion, and data aggregation;

[0013] Analyzing the processed data to obtain analysis results;

[0014] The analysis results are entered into an Excel file and shared via the Excel file.

[0015] Preferably, the sharing of the Excel file includes:

[0016] Select "Tools" in the menu bar of the Excel file, and select "Shared Workbook" in the pop-up function list;

[0017] Click the "Share Workbook" to open the Share Workbook dialog box;

[0018] In the Shared Workbook dialog box, check "Allow multiple users to edit simultaneously and allow workbook merging".

[0019] Preferably, the data includes structured data, semi-structured data and unstructured data.

[0020] Preferably, the data cleaning includes data quality assessment, missing value processing, outlier value processing, and duplicate value processing;

[0021] The data conversion includes data format conversion and data normalization;

[0022] The data aggregation includes data screening and data merging.

[0023] Preferably, analyzing the processed data includes data mining, data visualization, and data reporting.

[0024] Preferably, creating the index of the data lake includes establishing the index on one or more columns of a table, and the index establishment method includes index name, table name, column name, and order.

[0025] Preferably, sharing of the Excel file includes grading the data according to its importance, and the grading levels include primary data, secondary data, and tertiary data, wherein the primary data is highly sensitive, the secondary data is generally sensitive, and the tertiary data is publicly available data.

[0026] This application also provides an Excel data sharing management system, including:

[0027] Building blocks for constructing a data lake;

[0028] A creation module, used to create an index of the data lake;

[0029] An adding module, used to add the data of adding the index to the data lake;

[0030] A sharing module, used for sharing the data in the data lake;

[0031] Wherein, constructing the data lake includes:

[0032] Collect various types of data;

[0033] Processing the data, including data cleaning, data conversion, and data aggregation;

[0034] Analyzing the processed data to obtain analysis results;

[0035] The analysis results are entered into an Excel file and shared via the Excel file.

[0036] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention classifies data on a data lake platform, estimates the consequences of data damage, analyzes the degree of harm caused by data damage, and classifies and grades data according to the importance of the data. For example, data is classified from high to low according to the degree of harm after damage. According to the sensitivity and importance of the data, the data classification levels are defined as primary data, secondary data, and tertiary data. Primary data is highly sensitive and important data, such as personal privacy information, core business secrets of enterprises, etc. Secondary data is generally sensitive and important data, such as daily office documents, routine business data, etc. Third-level data is publicly accessible data, such as public information, industry statistics, etc. Different levels of encryption systems are added according to different levels of data to improve data security, thereby avoiding the danger that the data lake architecture, by definition, puts all eggs in one basket. If the security of the repository is compromised, then unknown parties may have access to all data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the Excel data sharing management method of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] This method and system propose to obtain detailed VIP drift events and abnormal events of failure to connect to VIP based on eBPF technology, and after obtaining the abnormal failure event of connecting to VIP, construct a network detection task according to the destination IP of VIP, further obtain the network link information of the failed connection, and finally pre-process the task and network link data according to the constraints of large model reasoning and input them into the large model for intelligent analysis to give the analysis result of failure. This method can accurately capture the VIP drift behavior without modifying the keepalived component source code. Due to the use of eBPF kernel technology, it can avoid missing any drift events, and can also obtain better real-time performance, lower latency and system overhead. This method can also achieve VIP access failure, that is, when a connection abnormality occurs, further diagnostic information acquisition is carried out, and the large model is used as the base to provide an intelligent analysis process to reduce the expert experience of operation and maintenance troubleshooting; at the same time, this method is universal, and any high-availability software based on the VRRP protocol can realize drift perception based on this method. Finally, this method can further shorten the troubleshooting time for operation and maintenance in high-availability scenarios.

[0042] Reference Figure 1 The present invention provides an Excel data sharing management method, comprising:

[0043] Build a data lake;

[0044] Create an index for the data lake;

[0045] Add the indexed data to the data lake;

[0046] Share the data in the data lake;

[0047] Building a data lake includes:

[0048] Collect various types of data;

[0049] Process the data, including data cleaning, data conversion, and data aggregation;

[0050] Analyze the processed data to obtain analysis results;

[0051] Enter the analysis results into an Excel file and share them via the Excel file.

[0052] In this embodiment, sharing the Excel file includes:

[0053] Select "Tools" in the menu bar of the Excel file, and select "Shared Workbook" in the pop-up function list;

[0054] Click "Share Workbook" to open the Share Workbook dialog box;

[0055] In the Shared Workbook dialog box, check "Allow multiple users to edit simultaneously and allow workbook merging";

[0056] Click the "Advanced" option in the Share Workbook dialog box. Here we can make more settings. Generally, there is no need to set them by default, but we can also make some more user-friendly settings. Please set them as needed;

[0057] After clicking OK on the above settings, the following warning box will pop up. Ignore this prompt and click the "OK" button directly;

[0058] After setting up a shared Excel file, "Shared" will be displayed after the title.

[0059] Furthermore, you can use Excel to perform data statistics. The statistical steps include using the COUNT function to count the number of cells containing numbers in a cell range. For example, entering =COUNT(A1:A10) can count the number of numbers in cells A1 to A10, using the SUM function to calculate the sum of all numbers in the cell range, for example =SUM(A1:A10), and using the AVERAGE function to calculate the average of all numbers in the cell range, for example =AVERAGE(A1:A10), to complete the data statistics.

[0060] In this embodiment, the steps of building a data lake include:

[0061] Data collection: First, you need to collect various types of data, including structured data, semi-structured data, and unstructured data. These data can come from different data sources, such as databases, log files, sensors, social media, etc.

[0062] Data storage: The collected data needs to be stored in a centralized data storage system, such as a distributed file system, relational database, NoSQL database, etc.

[0063] Data processing: Processing the stored data, including data cleaning, data conversion, data aggregation, etc. These processes can use big data processing frameworks such as Apache Hadoop, Apache Spark, Apache Flink, etc.

[0064] Specifically, before data cleaning is performed in the data lake, data collection is required. Data collection refers to obtaining data from different data sources and storing it in the data lake. Data sources may include structured data (such as relational databases), semi-structured data (such as XML files), and unstructured data (such as text files, images, and videos). The data collection process needs to consider factors such as the reliability of the data source, the amount of data, and the type of data. Then, data cleaning is performed in the data lake. The purpose of data cleaning is to detect and correct errors, omissions, duplications, and inconsistencies in the data. Data cleaning may include the following steps:

[0065] Data quality assessment: First, the data needs to be assessed for quality to identify problems in the data. The quality assessment can include assessments of data integrity, consistency, accuracy, and legality.

[0066] Missing value processing: For data with missing values, you can choose to delete the records containing missing values, or use interpolation methods to fill in the missing values. Common interpolation methods include mean interpolation, regression interpolation, and multiple interpolation.

[0067] Outlier processing: Outliers are observations that are significantly different from other observations in the data set. Statistical methods (such as the 3σ principle) or methods based on domain knowledge can be used to detect and process outliers.

[0068] Duplicate value processing: For data with duplicate values, you can choose to delete duplicate values ​​or merge duplicate values. When merging duplicate values, you need to consider the consistency and accuracy of the data. Data conversion: In the data cleaning stage, you can also convert the data to meet the needs of subsequent analysis. Data conversion can include data format conversion, data normalization, data aggregation and data standardization. Next, data integration is performed. Data integration is to integrate data from different data sources into a unified data model.

[0069] Data aggregation can include steps such as data screening, data matching, and data merging. In the process of data aggregation, factors such as data consistency, completeness, and accuracy need to be considered. Finally, data conversion is performed in the data lake. Data conversion is to convert raw data into a format that can be analyzed and mined. Data conversion can include operations such as data reduction, data transformation, and data discretization. The purpose of data conversion is to reduce the complexity and dimension of data to improve the efficiency and accuracy of subsequent analysis.

[0070] In short, the cleaning and preprocessing methods of data in the data lake are important steps to ensure data quality and accuracy. These methods include multiple stages such as data collection, data cleaning, data aggregation and data conversion. Through reasonable data cleaning and preprocessing methods, the data quality in the data lake can be improved, providing a reliable foundation for subsequent data analysis and mining.

[0071] Data analysis: Analyze the processed data, including data mining, data visualization, data reporting, etc. These analyses can use data analysis tools such as Tableau, PowerBI, Jupyter Notebook, etc.;

[0072] Specifically, data analysis is the process of exploring and understanding the data in the data lake. The goal of data exploration is to discover patterns, trends and association rules in the data through methods such as data visualization, summary and statistical analysis. Data visualization is to present data in a graphical way to help users understand the data more intuitively. Summary statistical analysis is to summarize and descriptive statistics of the data, such as calculating the mean, standard deviation and frequency distribution of the data. Through data exploration, users can have an in-depth understanding of the characteristics of the data in the data lake and provide a basis for subsequent data modeling and analysis; data modeling is the process of building mathematical or statistical models using data in the data lake. The goal of data modeling is to reveal the hidden laws and associations behind the data through modeling and analysis of the data. Commonly used data modeling methods include machine learning, data mining and statistical analysis. Machine learning is a method of achieving automatic learning and prediction by training models. Commonly used algorithms include decision trees, support vector machines and neural networks. Data mining is a method of automatically discovering patterns and association rules from large-scale data. Commonly used algorithms include association rule mining, clustering analysis and classification algorithms. Statistical analysis is a method of analyzing and inferring data through statistical methods. Commonly used techniques include hypothesis testing, regression analysis, and time series analysis. Data visualization is the process of presenting data analysis results to users in a visual way. Data visualization can help users understand data analysis results more intuitively and discover patterns and trends in data. Data visualization methods include charts, maps, and dashboards. Charts can be used to represent the distribution, relationship, and trend of data, such as bar charts, line charts, and scatter plots. Maps can be used to represent the distribution and correlation of geographic data, such as heat maps and geographic scatter plots. Dashboards are a visual interface that comprehensively displays multiple indicators and data, and can monitor and analyze business conditions in real time. Data analysis and mining technologies in data lakes include data preparation, data exploration, data modeling, and data visualization. These technologies can help enterprises deeply understand the data in the data lake, discover patterns and trends in the data, and provide support for decision-making. Through data analysis and mining technologies in data lakes, enterprises can achieve comprehensive mining and value extraction of data of various types and sizes, thereby gaining an advantage in competition.

[0073] Data sharing: Share the analysis results with other teams or departments so that they can better understand the data and make decisions;

[0074] Specifically, data sharing directly uses Excel files, enters all the data to be shared, and then puts it in a network sharing directory for everyone to download and view.

[0075] In this embodiment, when the target data is acquired in the data lake, the index creation obtains the data dynamic change information of the target data, and the data dynamic change information is used to map the transaction actions of the target data in the process of entering the data lake. According to the data dynamic change information of the target data, the data features of the target data are extracted, and the data lake index is created according to the extracted data features of the target data;

[0076] Specifically, the steps to obtain target data are as follows: Extracting metadata is the main challenge faced in the data intake stage. In order to adapt to the uncertainty of heterogeneous data sources, it is crucial to adopt a flexible and extensible metadata structure. GEMMS is a flexible and extensible data lake metadata management system that can extract metadata from heterogeneous data sources and store the metadata in an extensible meta-model. First, metadata attributes are stored in key-value pairs, and then the structure of the original data (such as matrix, tree, graph, etc.) is identified through structural metadata. Finally, it is connected to the semantic model by attaching semantic data. The Constance system is an intelligent data lake system. In order to extract as much metadata as possible in the data intake stage, the system uses the structural metadata discovery (SMD) component, but this component mainly solves the problem of metadata structure extraction of semi-structured data sources. Sawadogo et al. proposed a method for extracting text document metadata structure in data lakes, which makes up for the problem of extracting metadata in unstructured data. Datamaran is an algorithm suitable for converting complex log files in a data lake environment. The algorithm automatically extracts metadata structure in semi-structured log data in an unsupervised manner. The Datamaran algorithm solves problems such as data boundary determination, data field determination, complex structure, redundant structure and semantic structure.

[0077] The steps to extract the data features of the target data are: principal component analysis. The PCA method is to obtain a set of new features with the largest variance from a set of features by solving the optimal orthogonal transformation. They are linear combinations of the original features and are unrelated to each other. The new features are then sorted by importance and the first few principal components are selected. Using fewer principal components to represent the data can achieve feature dimensionality reduction and eliminate noise in the data. This algorithm does not consider the category information of the sample and is an unsupervised method.

[0078] Further, the method of creating a data lake index: An index is another way to represent data. The data order it provides is different from the physical storage order of the data on the disk. The special function of an index is to rearrange the physical position of records in a table. An index can be created on a column of a data table, or on a combination of several columns of a table. The method of creating an index is:

[0079] create[unique][cluster]index<index name>

[0080] on (<column name>[<order>][,<column name>][<order>]…)

[0081] An index can be created on one or more columns of a table. Each <column name> can be followed by <order> to specify the order in which the index values ​​are sorted. You can choose ASC (ascending order) or DESC (descending order). The default value is ASCunique, which means that each index value of this index corresponds to only a unique data record.

[0082] In this embodiment, the user specifies the index field, index type, and data sorting method based on business needs and experience, completes the background creation of a manual index, and obtains an initial index. As the data gradually enters the lake, the initial index is dynamically updated by performing data feature analysis on the data entering the lake to obtain a more complete data lake index, and the change actions of the target data are recorded in the data dynamic change information. Then, based on the data dynamic change information of the target data, a data change record file is generated, wherein the data change record file includes at least one data change record of the target data, and the data change record is used to record the data dynamic changes of the target data.

[0083] Furthermore, data lake data security sharing management, data classification on the data lake platform, estimate the consequences of data damage, analyze the degree of harm caused by data damage, and classify data according to the importance of the data. For example, data is divided from high to low according to the degree of harm after damage. According to the sensitivity and importance of the data, the data classification levels are defined as primary data, secondary data, and tertiary data. Among them, primary data is highly sensitive data, such as personal privacy information, core business secrets of enterprises, etc., secondary data is general sensitive data, such as daily office documents, routine business data, etc., and tertiary data is publicly available data, such as public information, industry statistics, etc. Different levels of encryption systems are added according to different levels of data to improve data security.

[0084] The beneficial effects of the present invention are as follows: by grading data on a data lake platform, estimating the consequences of data damage, analyzing the degree of harm caused by data damage, and grading data according to the importance of the data, for example, grading data from high to low according to the degree of harm after damage, and defining data grading levels into different levels according to the sensitivity and importance of the data: primary data, secondary data, and tertiary data, wherein primary data is highly sensitive and important data, such as personal privacy information, core business secrets of an enterprise, etc., secondary data is generally sensitive and important data, such as daily office documents, routine business data, etc., and tertiary data is publicly accessible data, such as public information, industry statistics, etc., and different levels of encryption systems are added according to different levels of data to improve data security, thereby avoiding the danger that the data lake architecture, by definition, puts all eggs in one basket. If the security of the repository is compromised, then unknown parties may have access to all data.

[0085] This application also provides an Excel data sharing management system, including:

[0086] Building blocks for constructing a data lake;

[0087] Create a module to create an index for the data lake;

[0088] Add module, used to add indexed data to the data lake;

[0089] Sharing module, used to share data in the data lake;

[0090] Building a data lake includes:

[0091] Collect various types of data;

[0092] Process the data, including data cleaning, data conversion, and data aggregation;

[0093] Analyze the processed data to obtain analysis results;

[0094] Enter the analysis results into an Excel file and share them via the Excel file.

[0095] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0096] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0097] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An Excel data sharing management method, characterized in that: include: Build a data lake; Creating an index for the data lake; Adding the data with the index added to the data lake; Sharing the data in the data lake; Wherein, constructing the data lake includes: Collect various types of data; Processing the data, including data cleaning, data conversion, and data aggregation; Analyzing the processed data to obtain analysis results; The analysis results are entered into an Excel file and shared via the Excel file.

2. The Excel data sharing management method according to claim 1, characterized in that: The excel file is shared including: Select "Tools" in the menu bar of the Excel file, and select "Shared Workbook" in the pop-up function list; Click the "Shared Workbook" to open the Shared Workbook dialog box; In the Shared Workbook dialog box, check "Allow multiple users to edit simultaneously and allow workbook merging".

3. The Excel data sharing management method according to claim 2, characterized in that: The data includes structured data, semi-structured data and unstructured data.

4. The Excel data sharing management method according to claim 3, characterized in that: The data cleaning includes data quality assessment, missing value processing, outlier processing, and duplicate value processing; The data conversion includes data format conversion and data normalization; The data aggregation includes data screening and data merging.

5. The Excel data sharing management method according to claim 4, characterized in that: Analysis of the processed data includes data mining, data visualization, and data reporting.

6. The Excel data sharing management method according to claim 5, characterized in that: Creating the index of the data lake includes establishing the index on one or more columns of a table, and the index establishment method includes an index name, a table name, a column name, and an order.

7. The Excel data sharing management method according to claim 6, characterized in that: The sharing of the Excel file includes grading the data according to its importance, and the grading levels include primary data, secondary data, and tertiary data. The primary data is highly sensitive, the secondary data is generally sensitive, and the tertiary data is publicly available data.

8. An Excel data sharing management system, characterized in that: include: Building blocks for constructing a data lake; A creation module, used to create an index of the data lake; An adding module, used to add the data of adding the index to the data lake; A sharing module, used for sharing the data in the data lake; Wherein, constructing the data lake includes: Collect various types of data; Processing the data, including data cleaning, data conversion, and data aggregation; Analyzing the processed data to obtain analysis results; The analysis results are entered into an Excel file and shared via the Excel file.

9. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: It stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the method described in any one of claims 1 to 7.

Images