Abnormal data management method and device

Abstract

The present invention discloses an abnormal data management method, which includes the following steps: step S1: in the abnormal occurrence time period, record the basic information of the abnormal data, create a block file in a local file, and write the abnormal data record in the block file; step S2: according to the abnormal data type, open a corresponding thread for the same type of abnormal data to read the abnormal data record in the block file; step S3: according to the abnormal data recovery request, recover and retry the read abnormal data. The abnormal monitoring module detects the occurrence of abnormal situations and establishes a block file. When abnormal data occurs, the abnormal data is saved in a local file. When the abnormal information needs to be restored, the abnormal data is read and restored from the local module to avoid the situation where the data is not processed due to network instability, database downtime, etc., to ensure that the abnormal data is not lost when the system returns to normal, and to improve the efficiency of abnormal data recovery.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an abnormal data management method and device. Background Art

[0002] In a service-oriented data system architecture, in order to avoid excessive coupling of various functional modules, modules usually communicate with each other to transmit data to promote business processes or interactions. However, data transmission often involves certain risks, which may lead to data transmission anomalies or even data transmission failures; for example: database crashes, network instability, etc.

[0003] When data anomalies occur, the existing technology generally adopts the method of retrying data transmission. However, the following problems often occur in implementation: it is impossible to repeatedly try to retransmit data for a long time; when the data volume is large and the data type is complex, the data recovery efficiency is very low or difficult. Summary of the invention

[0004] The present invention aims to solve the problem that abnormal data in the data system architecture of the prior art is difficult to recover when the data volume is huge and the data type is complex, and provides an abnormal data management method and device that can efficiently recover abnormal data.

[0005] On the one hand, the present application proposes a method for managing abnormal data, the method comprising the following steps:

[0006] Step S1: During the time period when the abnormality occurs, basic information of the abnormal data is recorded, a block file is created in a local file, and the abnormal data record is written in the block file;

[0007] Step S2: according to the type of abnormal data, a corresponding thread is opened for the same type of abnormal data to read the abnormal data records in the block file;

[0008] Step S3: According to the abnormal data recovery request, the read abnormal data is recovered and retried. Using block files to record abnormal data can greatly improve the processing speed of abnormal data. When an abnormality occurs, the abnormal data can be quickly recorded in a local file so that the abnormal data can be directly read and restored from the local file after the abnormality occurs. Avoid the situation where data is not processed due to network instability, database downtime, etc., ensure that the abnormal data is not lost when the system returns to normal, and improve the efficiency of abnormal data recovery.

[0009] Preferably, the step S1 of creating a block file in a local file includes: creating a first block area of ​​continuous addresses in a local disk, each first block area corresponding to a block file; writing the starting position information and the ending position information of the block file in the header of each block file; writing the length and the ending mark of the block file in the tail of each block file, so as to support reverse query and reading of block files.

[0010] Preferably, writing abnormal data records in the block file in step S1 includes: writing abnormal data in each record, and writing length data of the record at the head and tail of each record, so as to support reverse reading of records.

[0011] Preferably, step S1 also includes establishing second block areas in the local file according to the type of abnormal data, each second block area corresponds to a second block file, writing the position index of the abnormal data record in the local file into the second block file, and updating the end position of the file at the head of the second block file, and updating the file length and the end position identifier of the file at the tail of the second block file.

[0012] Preferably, the reading of the abnormal data record in the block file described in step S2 includes: obtaining the abnormal data position index information from the second block area, finding the abnormal data record in the first block file according to the abnormal data position index information, reading the abnormal data, and recording the abnormal data number currently read, so as to facilitate the retrieval of the abnormal data.

[0013] Preferably, step S3 also includes clearing the abnormal data records that have been recovered according to the current abnormal data processing progress.

[0014] In another method, the present application proposes an abnormal data management device, in which the abnormal data management method described in the present application is built-in.

[0015] Preferably, it includes an abnormality monitoring module: monitoring and detecting the occurrence of abnormalities, and when determining that an abnormality occurs, transmitting a signal with abnormality occurrence information to the abnormal data recording module;

[0016] Abnormal data recording module: receives the signal with abnormal occurrence information sent by the abnormal monitoring module, and saves the abnormal data when the abnormality occurs to a local file;

[0017] Abnormal data reading module: It has multi-thread communication. After receiving the abnormal data recovery request signal, it starts a corresponding thread for the same type of abnormal data to read the abnormal data records in the block file;

[0018] Abnormal data recovery module: receives abnormal data recovery requests and retry recovery of the read abnormal data.

[0019] Preferably, the abnormal data recording module comprises:

[0020] Generating unit: creating first block areas for storing abnormal data in the local disk respectively; establishing second block areas for writing position index information in the local file respectively according to the abnormal data type;

[0021] Writing unit: used to write abnormal data records in the first block file, write abnormal data in each record, and write the length data of the record at the head and tail of each record; write the position index of the abnormal data record in the local file in the second block file, and update the end position of the file in the head of the second block file, and update the length of the file and the end position identifier of the file at the tail of the second block file. It is convenient to query the block files in reverse order and further improve the efficiency of abnormal data recovery.

[0022] Preferably, the generating unit is further configured to create a new first block file when the available capacity of the first block file reaches a lower limit, and to create a new second block file when the available capacity of the second block file reaches a lower limit, so as to ensure that there is always sufficient block file capacity in reserve.

[0023] Therefore, the present invention has the following beneficial effects: the present application proposes an abnormal data management method and device, in which an abnormal monitoring module detects the occurrence of abnormal situations and establishes block files. When abnormal data occurs, the abnormal data is saved in a local file. When the abnormal information needs to be restored, the abnormal data is read and restored from the local module to avoid situations in which data is not processed due to network instability, database downtime, etc., to ensure that when the system returns to normal, the abnormal data is not lost, thereby improving the efficiency of abnormal data recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flowchart of an abnormal data management method according to an embodiment of the present invention.

[0025] Figure 2 It is a flowchart of subdivided steps of step S1 of an abnormal data management method according to an embodiment of the present invention.

[0026] Figure 3 It is a schematic diagram of the first block file structure according to an embodiment of the present invention.

[0027] Figure 4 It is a schematic diagram of the structure of a single abnormal data record according to an embodiment of the present invention.

[0028] Figure 5 The present invention is a structural block diagram of an abnormal data management device according to an embodiment of the present invention.

[0029] In the figure: 1. Abnormal monitoring module 2. Abnormal data recording module 3. Abnormal data reading module 4. Abnormal data recovery module 5. Abnormal data recording 6. File header information 7. File tail information 8. Record header information 9. Record tail information 10. Abnormal data. DETAILED DESCRIPTION

[0030] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0031] Example:

[0032] like Figure 1 An abnormal data management method shown includes the following steps:

[0033] Step S1: During the time period when the abnormality occurs, basic information of the abnormal data is recorded, a block file is created in a local file, and abnormal data record 5 is written in the block file;

[0034] Step S1 includes the following subdivision steps: Figure 2 As shown:

[0035] S1001: Create block files

[0036] Create a block area of ​​a certain size with continuous addresses in the local disk, corresponding to a block file, to record abnormal data. When the data that can be accommodated in each block file reaches the upper limit, a new block file is created. Using block files to record abnormal data can greatly improve the reading and writing speed of abnormal data.

[0037] S1002: Initialize block files

[0038] The starting position and the ending position of the file are written in the header of the block file to determine the storage space position belonging to the block file.

[0039] The length of the file and the end identifier of the file are written at the end of the block file to support reverse query and reading of the block file.

[0040] S1003: Write abnormal data record 5

[0041] When an exception is detected, the exception is recorded in a block file.

[0042] The length of this record is written in the header of this record to determine the storage space belonging to this record.

[0043] Write the exception data of this record in the middle of this record.

[0044] The length of this record is written at the end of this record to support reverse reading of records.

[0045] S1004: Save abnormal data record index

[0046] According to the type of abnormal data, block areas are created respectively, and the position index of the abnormal data record 5 is written to facilitate retrieval of the abnormal data.

[0047] S1005: Update block file header and tail information

[0048] Update the end position of the file in the header of the block file.

[0049] The file length and the file end identifier are updated at the end of the block file.

[0050] Step S2: according to the type of abnormal data, a corresponding thread is opened for the same type of abnormal data to read the abnormal data record 5 in the block file; reading the abnormal data record 5 in the block file includes: obtaining the abnormal data position index information from the second block area, finding the abnormal data record 5 in the first block file according to the position index information of the abnormal data, reading the abnormal data, and recording the abnormal data number currently read.

[0051] Step S3: According to the abnormal data recovery request, the read abnormal data is recovered and retried. According to the current abnormal data processing progress, the abnormal data records 5 that have been recovered are cleared.

[0052] like Figure 3 The first block file header information 6 includes a start position and an end position, the file tail information 7 includes the length of the file and an end identifier of the file, and an abnormal data record 5 is written between the header and the tail of the first block file.

[0053] like Figure 4 The record header information 8 of the single abnormal data record 5 shown includes the length of the record, and the record tail information 9 includes the length of the record. The abnormal data 10 is written between the header and the tail.

[0054] Figure 5 is a schematic diagram of the structure of an abnormal data processing device according to an embodiment of the present application.

[0055] It includes an abnormality monitoring module 1: monitoring and detecting abnormality occurrence, and when determining that an abnormality occurs, transmitting a signal with abnormality occurrence information to an abnormality data recording module 2;

[0056] Abnormal data recording module 2: receives the signal with abnormal occurrence information sent by abnormal monitoring module 1, and saves the abnormal data when the abnormality occurs into a local file;

[0057] Abnormal data recording module 2, including:

[0058] Generation unit: creates first block areas for storing abnormal data in the local disk; establishes second block areas for writing position index information in the local file according to the type of abnormal data; the generation unit is also used to create a new first block file after the available capacity of the first block file reaches a lower limit, and to create a new second block file after the available capacity of the second block file reaches a lower limit.

[0059] Writing unit: used to write abnormal data records in the first block file, write abnormal data in each record, and write the length data of the record at the head and tail of each record; write the position index of the abnormal data record in the local file in the second block file, and update the end position of the file at the head of the second block file, and update the length of the file and the end position identifier of the file at the tail of the second block file.

[0060] Abnormal data reading module 3: with multi-thread communication, after receiving the abnormal data recovery request signal, a corresponding thread is opened for the same type of abnormal data to read the abnormal data records in the block file;

[0061] Abnormal data recovery module 4: receives abnormal data recovery request, and recovers and retry the read abnormal data.

[0062] When abnormal data occurs, this application saves the abnormal data to a local file. When the abnormal information needs to be restored, the abnormal data is read and restored from the local module to avoid situations where data is not processed due to network instability, database crashes, etc., to ensure that the abnormal data is not lost when the system returns to normal, and to improve the efficiency of abnormal data recovery.

[0063] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0064] Although the terminology such as abnormal data, block area, block file, and identification is used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.

Claims

1. A method for managing abnormal data, characterized in that: The steps include: Step S1: During the time period when the abnormality occurs, basic information of the abnormal data is recorded, a first block area with a continuous address is created in the local disk, each first block area corresponds to a first block file, the start position information and the end position information of the first block file are written in the header of each first block file, and the abnormal data record is written in the first block file; second block areas are respectively established in the local file according to the type of abnormal data, each second block area corresponds to a second block file, and the position index of the abnormal data record in the local file is written in the second block file; Step S2: opening a corresponding thread for the same type of abnormal data according to the abnormal data type to read the abnormal data record in the first block file, including: obtaining the abnormal data position index information from the second block area, finding the abnormal data record in the first block file according to the abnormal data position index information, reading the abnormal data, and recording the abnormal data number currently read; Step S3: according to the abnormal data recovery request, the read abnormal data is recovered and retried.

2. The abnormal data management method according to claim 1, characterized in that the steps S1 includes: creating first block areas with continuous addresses in the local disk, each first block area corresponds to a first block file; and writing the length and end mark of the first block file at the end of each block file.

3. The abnormal data management method according to claim 2, characterized in that: Writing abnormal data records in the first block file in step S1 includes: writing abnormal data in each record, and writing length data of the record at the head and the tail of each record.

4. The abnormal data management method according to claim 3 is characterized in that: The step S1 also includes establishing second block areas in the local file according to the abnormal data type, each second block area corresponds to a second block file, writing the position index of the abnormal data record in the local file into the second block file, and updating the end position of the file at the header of the second block file, and updating the file length and the end position identifier of the file at the tail of the second block file.

5. The abnormal data management method according to claim 1 is characterized in that: Step S3 also includes clearing the abnormal data records that have been recovered according to the current abnormal data processing progress.

6. An abnormal data management device, using the abnormal data management method according to any one of claims 1 to 5, characterized in that: include: Abnormal monitoring module: monitors and detects abnormal situations. When an abnormal situation is determined to have occurred, it transmits a signal with abnormal situation information to the abnormal data recording module; Abnormal data recording module: receives the signal with abnormal occurrence information sent by the abnormal monitoring module, and saves the abnormal data when the abnormality occurs to a local file; Abnormal data reading module: having multi-thread communication, after receiving the abnormal data recovery request signal, opening a corresponding thread for the same type of abnormal data to read the abnormal data record in the first block file; Abnormal data recovery module: receives abnormal data recovery requests and retry recovery of the read abnormal data.

7. The abnormal data management device according to claim 6, characterized in that: The abnormal data recording module includes: a generating unit: creating first block areas for storing abnormal data in the local disk respectively; and establishing second block areas for writing position index information in the local file respectively according to the type of abnormal data; The writing unit is used to write abnormal data records in the first block file, write abnormal data in each record, and write the length data of the record at the head and tail of each record; write the position index of the abnormal data record in the local file in the second block file, and update the end position of the file at the head of the second block file, and update the length of the file and the end position identifier of the file at the tail of the second block file.

8. The abnormal data management device according to claim 7, characterized in that: The generating unit is further configured to create a new first block file after the available capacity of the first block file reaches a lower limit, and to create a new second block file after the available capacity of the second block file reaches a lower limit.

Images