Self-adaptive data lossless acquisition and storage method

Abstract

The invention discloses a self-adaptive data lossless acquisition and storage method, which comprises the following steps of performing data feature extraction on existing data in a database, and associating data features with a storage medium; data feature capture is performed on data needing to be collected, and a medium suitable for storage is analyzed according to data features; in the process of data acquisition, transmission and storage, the data are verified; according to the self-adaptive data lossless collection and storage method, the storage medium is automatically analyzed through data feature analysis, a data source does not need to be manually configured, the data is verified under various operations, the integrity and consistency of the data are ensured, the data are detected, and the data recovery efficiency is improved. The efficiency of timely repairing the data is ensured, and the accuracy and value of the data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to an adaptive data lossless acquisition and storage method. Background Art

[0002] In the era of big data, the integrity and reliability of data play a vital role in supporting business decisions and driving innovation and development. However, traditional data collection and storage solutions often fail to balance data losslessness and storage efficiency. At the same time, there is a lack of effective repair methods when data is damaged, which greatly reduces the value of data. Therefore, it is particularly important to develop a system that can adapt to the characteristics of different data sources, achieve lossless data collection, efficient storage, and intelligent repair.

[0003] Traditional data management platforms use manual configuration of data sources when collecting and storing data. Different data sources have different formats. The amount of data sources that need to be configured on the big data platform is huge, which greatly increases the workload of operation and maintenance personnel.

[0004] There is no data retrieval and anomaly monitoring mechanism for incomplete data;

[0005] When dealing with incomplete data, default values ​​are mostly used to fill in. Although this ensures that no data is missed, the quality of the data is greatly reduced. Summary of the invention

[0006] In order to solve the deficiencies in the prior art, the present invention provides an adaptive data lossless acquisition and storage method, which automatically analyzes the storage medium through data feature analysis, does not require manual configuration of the data source, verifies the data under a variety of operations, ensures the integrity and consistency of the data, detects the data, ensures the efficiency of timely repair of the data, and ensures the accuracy and value of the data.

[0007] To achieve the above object, the present invention adopts the following technical solution:

[0008] The present invention provides an adaptive data lossless acquisition and storage method, comprising the following steps:

[0009] Extract data features from existing data in the database and associate the data features with the storage medium;

[0010] Capture the data features of the data to be collected, and analyze the media suitable for storage based on the data features;

[0011] Verify data during data collection, transmission and storage;

[0012] Check business data and repair data if there is any missing data.

[0013] The adaptive data lossless acquisition and storage method of the present invention first associates the data to be collected with the storage medium by means of feature extraction and analysis, without the need for manual configuration, thereby greatly improving the configuration efficiency. The data is verified during the data operation process, thereby obtaining data with higher integrity and consistency. Data detection and data repair are performed on the data in the database, thereby improving the data quality.

[0014] The adaptive data lossless acquisition and storage method of the present invention automatically analyzes the storage medium through data feature analysis, without the need for manual configuration of the data source, verifies the data under multiple operations, ensures the integrity and consistency of the data, detects the data, ensures the efficiency of timely repair of the data, and ensures the accuracy and value of the data.

[0015] In a further technical solution, a decision tree is established based on the correlation between data features and storage media, and the decision tree is used when performing data feature analysis.

[0016] Decision trees can be used for efficient and accurate classification, ensuring efficiency and accuracy.

[0017] In a further technical solution, when the number of data verification failures reaches a threshold, an alarm notification is issued.

[0018] An alarm procedure has been established to ensure the accuracy of data operations and avoid data contamination.

[0019] In a further technical solution, when repairing data, an expectation maximization algorithm is used to calculate the missing value through the data around the missing value.

[0020] By using the expectation maximization algorithm, the accuracy of data repair can be improved.

[0021] The beneficial effects are:

[0022] 1. The adaptive data lossless acquisition and storage method of the present invention automatically analyzes the storage medium through data feature analysis, without the need to manually configure the data source, verifies the data under various operations, ensures the integrity and consistency of the data, detects the data, ensures the efficiency of timely repair of the data, and ensures the accuracy and value of the data.

[0023] 2. Decision trees can be used for efficient and accurate classification, ensuring efficiency and accuracy.

[0024] 3. An alarm procedure has been established to ensure the accuracy of data operations and avoid data pollution.

[0025] 4. The accuracy of data repair can be improved by using the expected value maximization algorithm. DETAILED DESCRIPTION

[0026] The present invention will be further described below:

[0027] Example:

[0028] An adaptive data lossless collection and storage method comprises the following steps:

[0029] Extract data features from existing data in the database, including but not limited to data type, data format, and metadata, and associate data features with storage media;

[0030] Capture the data features of the data to be collected, and analyze the media suitable for storage based on the data features;

[0031] Verify data during data collection, transmission and storage;

[0032] It should be emphasized that in this step, the purpose of verification is to find out data anomalies caused by network fluctuations.

[0033] Check business data and repair data if there is any missing data.

[0034] It should be emphasized that in this step, the purpose of detection is to find out business data anomalies.

[0035] Therefore, the present invention detects network fluctuations and business data anomalies in the database respectively, further ensuring the accuracy of the data.

[0036] The adaptive data lossless acquisition and storage method of the present invention first associates the data to be collected with the storage medium by means of feature extraction and analysis, without the need for manual configuration, thereby greatly improving the configuration efficiency. The data is verified during the data operation process, thereby obtaining data with higher integrity and consistency. Data detection and data repair are performed on the data in the database, thereby improving the data quality.

[0037] The adaptive data lossless acquisition and storage method of the present invention automatically analyzes the storage medium through data feature analysis, without the need for manual configuration of the data source, verifies the data under multiple operations, ensures the integrity and consistency of the data, detects the data, ensures the efficiency of timely repair of the data, and ensures the accuracy and value of the data.

[0038] In another embodiment, a decision tree is established based on the correlation between data features and storage media, and the decision tree is used when performing data feature analysis.

[0039] Decision tree is a commonly used classification and regression algorithm that achieves classification or prediction of data sets by building a tree structure. The main idea of ​​​​the decision tree is to divide the data set into several small parts, each of which corresponds to a branch, until the predetermined termination condition is reached (such as the samples in the leaf nodes belong to the same category or reach a preset depth). The decision tree algorithm will select the best attributes for division, thereby obtaining the best classification or prediction effect.

[0040] Decision trees can be used for efficient and accurate classification, ensuring efficiency and accuracy.

[0041] In another embodiment, when the number of data verification failures reaches a threshold, an alarm notification is issued.

[0042] An alarm procedure has been established to ensure the accuracy of data operations and avoid data contamination.

[0043] In another embodiment, during data repair, an expectation maximization algorithm is used to calculate the missing value using data surrounding the missing value.

[0044] The expectation maximization algorithm is an algorithm used in statistics to iteratively solve the maximum likelihood estimation (MLE) or maximum a posteriori estimation (MAP) of the parameters of a probability model containing latent variables. It is widely used in parameter estimation of complex statistical models with missing data or latent variables.

[0045] By using the expectation maximization algorithm, the accuracy of data repair can be improved.

[0046] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. An adaptive data lossless acquisition and storage method, characterized in that: The following steps are involved: Extract data features from existing data in the database and associate the data features with the storage medium; Capture the data features of the data to be collected, and analyze the media suitable for storage based on the data features; Verify data during data collection, transmission and storage; Check business data and repair data if there is any missing data.

2. The adaptive data lossless acquisition and storage method according to claim 1, characterized in that: A decision tree is established based on the correlation between data features and storage media, and the decision tree is used when performing data feature analysis.

3. The adaptive data lossless acquisition and storage method according to claim 1, characterized in that: When the number of data verification failures reaches the threshold, an alarm notification is issued.

4. The adaptive data lossless acquisition and storage method according to claim 1, characterized in that: When repairing data, the expected value maximization algorithm is used to calculate the missing values ​​through the data around the missing values.