Data maintenance method based on big data

Abstract

The invention relates to the technical field of information engineering, and provides a data maintenance method based on big data, comprising the following steps: step 1, uploading multiple data maintenance records, analyzing multiple maintenance data, extracting data maintenance events, and classifying the events into common events and rare events; 2, generating a high-incidence event big data model by using the data of the common events, and generating a rare event big data model by using the rare events; and 3, preferentially detecting the data in the database in batches by using the high-incidence event big data model, and improving a secondary detection threshold value according to problems carried by each batch of detected data. According to the method, the security and the integrity of the data are firstly measured by means of the high-incidence event big data model, the preliminary condition of the data is judged, and the detection frequency of rare events is determined, so that the security and the integrity of the database data are rapidly measured, tedious and time-wasting comprehensive detection is avoided, the maintenance time is shortened, and the maintenance efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of information engineering technology, and in particular to a data maintenance method based on big data. Background Art

[0002] Data maintenance refers to the process of managing, updating, repairing and protecting data to ensure its accuracy, integrity and availability. Effective data maintenance can help organizations improve decision-making quality, optimize business processes and reduce data-related risks.

[0003] In the prior art, data maintenance usually involves a detailed and comprehensive check of the data within the system, so as to find problems in the data and solve the problems found in the data to achieve the maintenance effect. However, this method takes a lot of time and the process is cumbersome and lengthy, which greatly increases the maintenance time and lacks efficiency. Therefore, a data maintenance method based on big data is needed. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a data maintenance method based on big data, which solves the problems in the prior art that data maintenance requires a lot of time, the process is cumbersome and lengthy, the maintenance time is greatly increased, and there is a lack of efficiency.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data maintenance method based on big data comprises the following steps: Step 1: Upload multiple data maintenance records, analyze multiple maintenance data, extract the data maintenance events performed, and classify them into common events and rare events; Step 2: Generate a big data model of high-incidence events using data of common events, and generate a big data model of rare events using data of rare events; Step 3: Prioritize using the high-incidence event big data model to test the data in the database in batches, and increase the secondary detection threshold according to the problems of each batch of data tested; Step 4: judging the batch of data according to the secondary detection threshold in each batch of data in step 3; Step 5: Determine the next step of processing the batch of data based on the judgment result. If the judgment passes, the data is considered safe. If the judgment fails, a second data test is performed; Step 6: The second data detection is performed according to the detection scheme corresponding to the gradient allocation of the secondary detection threshold of the batch of data; Step 7: Record the data that failed the judgment in step 4, generate a report form, and repair the problems in each batch of data; Step 8. The report table data generated in step 7 is applied again to the high-incidence event big data model and the rare event big data model for iterative optimization.

[0006] Preferably, the specific steps of increasing the secondary detection threshold in step 3 are: S1. Set three gradients for the number of detected problems, and increase the gradients according to the number of detected problems in the batch of data; S2, set the number of the first gradient to 0-X, and when the number of questions is greater than 0 and less than or equal to X, the threshold is increased by 1 for each data question; S3, setting the number of the second gradient to X to Y, when the number of questions is greater than X and less than or equal to Y, the threshold is increased by 2 for each data question exceeding the number X; S4. Set the number of the third gradient to Y-∞. When the number of questions is ≥Y, the threshold is increased by 3 for each data question exceeding the number Y.

[0007] Preferably, the specific process of the determination in step four is: taking the secondary detection threshold of each batch of data in step three as a benchmark, when the secondary detection threshold is less than 0, the batch of data is considered normal, and when the secondary detection threshold is ≥1, the batch of data is considered abnormal and assigned to the second data detection process.

[0008] Preferably, the specific steps of gradient allocation in step six are: S1, set four threshold gradients that increase layer by layer; S2: When the batch of values ​​is in any one of the first gradient, the second gradient and the third gradient, secondary detection schemes with different detection frequencies are generated in sequence corresponding to the gradient; S3. When the batch values ​​are at the fourth gradient, a comprehensive secondary inspection and additional re-inspection plan is generated.

[0009] Preferably, the scheme for generating the fourth gradient in S3 needs to be manually implemented by a technician.

[0010] Preferably, the secondary detection scheme in S2 is to generate random numbers, and extract a control unit range in the rare event large model according to the random numbers for detection and comparison. The random number increases as the corresponding gradient increases, that is, the detection frequency is increased.

[0011] Preferably, the detection contents of step three and step six include data integrity, data loopholes, malicious data contained in the data, bundled data contained in the data, viruses contained in the data and other security risks related to the data.

[0012] Preferably, the repair of data in step seven is divided into intelligent repair and manual repair, and the intelligent repair is based on the virus database and the copy of the historical complete data for repair and recovery.

[0013] Preferably, the data problems are repaired using simple and fast intelligent repair methods first, and individual cases where intelligent repair fails are counted in a table, with the file path annotated with characteristic points and turned into manual repair.

[0014] Preferably, the report form of step seven is also used for uploading to the cloud and saving a copy to prevent the loss of local files and provide multiple insurances.

[0015] The present invention provides a data maintenance method based on big data. It has the following beneficial effects: 1. The present invention uses a high-incidence event big data model to first determine the security and integrity of the data, judge the preliminary situation of the data, and determine the detection frequency of rare events, so as to quickly determine the security and integrity of the database data, avoid tedious and time-consuming comprehensive detection, reduce maintenance time, and improve maintenance efficiency.

[0016] 2. The present invention ensures the security and recoverability of data by recording data that fails to pass the judgment, generating report forms, and performing intelligent or manual repair. The cloud upload and copy storage mechanism of the report form further enhances the recoverability of the data, and can ensure the security of data even in the event of local file loss.

[0017] 3. The present invention performs iterative optimization by reapplying the generated report table data to the high-incidence event big data model and the rare event big data model, thereby achieving continuous self-improvement of the model, enabling the data maintenance method to continuously adapt to new data features and changes, and improving the adaptability and predictive ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flow chart of the method of the present invention DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Example: Please refer to the attached Figure 1 The embodiment of the present invention provides a data maintenance method based on big data, comprising the following steps: Step 1: Upload multiple data maintenance records, analyze multiple maintenance data, extract the data maintenance events performed, and classify them into common events and rare events; Step 2: Generate a big data model of high-incidence events using data of common events, and generate a big data model of rare events using data of rare events; Step 3: Prioritize using the high-incidence event big data model to test the data in the database in batches, and increase the secondary detection threshold according to the problems of each batch of detected data. The specific steps for increasing the secondary detection threshold are: S1. Set three gradients for the number of detected problems, and increase the gradients according to the number of detected problems in the batch of data; S2, set the number of the first gradient to 0-X, and when the number of questions is greater than 0 and less than or equal to X, the threshold is increased by 1 for each data question; S3, setting the number of the second gradient to X to Y, when the number of questions is greater than X and less than or equal to Y, the threshold is increased by 2 for each data question exceeding the number X; S4. Set the number of the third gradient to Y-∞. When the number of questions is ≥Y, the threshold is increased by 3 for each data question exceeding the number Y.

[0021] Step 4: Determine the batch of data according to the secondary detection threshold in each batch of data in step 3. The specific process of determination is: take the secondary detection threshold of each batch of data in step 3 as a benchmark, when the secondary detection threshold is less than 0, the batch of data is considered normal, when the secondary detection threshold is ≥1, the batch of data is considered abnormal, and is assigned to the second data detection process; Step 5: Determine the next step of processing the batch of data based on the judgment result. If the judgment passes, the data is considered safe. If the judgment fails, a second data test is performed; Step 6: The second data detection is performed according to the detection scheme corresponding to the gradient allocation of the secondary detection threshold of the batch of data. The specific steps of gradient allocation are: S1, set four threshold gradients that increase layer by layer; S2: When the batch of values ​​is in any one of the first gradient, the second gradient and the third gradient, secondary detection schemes with different detection frequencies are generated in sequence corresponding to the gradient; S3. When the batch value is at the fourth gradient, a comprehensive secondary inspection and additional re-inspection plan is generated, which needs to be manually implemented by technical personnel.

[0022] The secondary detection scheme in S2 is to generate random numbers, and extract a range of control units in the rare event large model according to the random numbers for detection and comparison. The random number increases as the corresponding gradient increases, that is, the detection frequency is increased.

[0023] Step 7. Record the data that failed the judgment in step 4, generate a report form, and repair the problems in each batch of data. The data repair is divided into intelligent repair and manual repair. Intelligent repair is based on the virus database and the copy of the historical complete data. Data problems are given priority to simple and fast intelligent repair. Individual cases where intelligent repair fails will be counted in a table, and the file path and feature points will be marked and turned into manual repair. The report form is also used for cloud upload and copy storage to prevent local file loss and set up multiple insurances; The report table data generated in step eight and step seven are applied again to the high-incidence event big data model and the rare event big data model for iterative optimization.

[0024] The detection contents of step three and step six include data integrity, data loopholes, malicious data contained in the data, bundled data contained in the data, viruses contained in the data and other security risks related to the data.

[0025] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data maintenance method based on big data, characterized in that: The following steps are involved: Step 1: Upload multiple data maintenance records, analyze multiple maintenance data, extract the data maintenance events performed, and classify them into common events and rare events; Step 2: Generate a big data model of high-incidence events using data of common events, and generate a big data model of rare events using data of rare events; Step 3: Prioritize using the high-incidence event big data model to test the data in the database in batches, and increase the secondary detection threshold according to the problems of each batch of data tested; Step 4: judging the batch of data according to the secondary detection threshold in each batch of data in step 3; Step 5: Determine the next step of processing the batch of data based on the judgment result. If the judgment passes, the data is considered safe. If the judgment fails, a second data test is performed; Step 6: The second data detection is performed according to the detection scheme corresponding to the gradient allocation of the secondary detection threshold of the batch of data; Step 7: Record the data that failed the judgment in step 4, generate a report form, and repair the problems in each batch of data; Step 8. The report table data generated in step 7 is applied again to the high-incidence event big data model and the rare event big data model for iterative optimization.

2. The data maintenance method based on big data according to claim 1, characterized in that: The specific steps of increasing the secondary detection threshold in step 3 are: S1. Set three gradients for the number of detected problems, and increase the gradients according to the number of detected problems in the batch of data; S2, set the number of the first gradient to 0-X, and when the number of questions is greater than 0 and less than or equal to X, the threshold is increased by 1 for each data question; S3, setting the number of the second gradient to X to Y, when the number of questions is greater than X and less than or equal to Y, the threshold is increased by 2 for each data question exceeding the number X; S4. Set the number of the third gradient to Y-∞. When the number of questions is ≥Y, the threshold is increased by 3 for each data question exceeding the number Y.

3. The data maintenance method based on big data according to claim 2 is characterized in that: The specific process of the determination in step 4 is: taking the secondary detection threshold of each batch of data in step 3 as a benchmark, when the secondary detection threshold is less than 0, the batch of data is considered normal, and when the secondary detection threshold is ≥1, the batch of data is considered abnormal and assigned to the second data detection process.

4. The data maintenance method based on big data according to claim 1, characterized in that: The specific steps of gradient allocation in step 6 are: S1, set four threshold gradients that increase layer by layer; S2: When the batch of values ​​is in any one of the first gradient, the second gradient and the third gradient, secondary detection schemes with different detection frequencies are generated in sequence corresponding to the gradient; S3. When the batch values ​​are at the fourth gradient, a comprehensive secondary inspection and additional re-inspection plan is generated.

5. The data maintenance method based on big data according to claim 4 is characterized in that: The scheme for generating the fourth gradient in S3 needs to be manually implemented by technicians.

6. The data maintenance method based on big data according to claim 4 is characterized in that: The secondary detection scheme in S2 is to generate random numbers, and extract a control unit range in the rare event large model according to the random numbers for detection and comparison. The random number increases as the corresponding gradient increases, that is, the detection frequency is increased.

7. The data maintenance method based on big data according to claim 1, characterized in that: The detection contents of step three and step six include data integrity, data loopholes, malicious data contained in the data, bundled data contained in the data, viruses contained in the data and other security risks related to the data.

8. The data maintenance method based on big data according to claim 1, characterized in that: The data repair in step seven is divided into intelligent repair and manual repair, and the intelligent repair is based on the virus database and the copy of the historical complete data for repair and recovery.

9. The data maintenance method based on big data according to claim 8, characterized in that: The data problems will be repaired using simple and fast intelligent repair methods first. Individual cases where intelligent repair fails will be counted in a table, with the file path and feature points marked for manual repair.

10. The data maintenance method based on big data according to claim 1, characterized in that: The report form in step seven is also used for cloud upload and copy storage to prevent local file loss and provide multiple insurances.

Images