A method and system for fast database content inspection

Abstract

The present invention discloses a method and system for quickly checking the contents of a database, the method comprising: connecting to a database to determine whether the data needs to be checked; when data checking is required, obtaining database records in batches from the database, allocating fixed-size memory blocks to the records obtained in batches, and storing the table data obtained in batches in two-bit arrays of fixed length in the memory blocks; re-aggregating the data to form a continuous data; and checking the data through a multi-mode matching algorithm after the aggregation is completed, and generating a final inspection result after the inspection is completed. The present invention solves the problem of difficulty in quickly screening and checking sensitive information in a database.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for quickly checking database content. Background Art

[0002] With the popularization of informatization, information management systems (such as OA, ERP, CRM, etc.) are used by more and more enterprises and institutions. Information management systems generally use databases to store data. As the amount of data stored in the database increases, some important business secrets or sensitive information will be accidentally stored in the database. If the data in the database is not checked regularly, it is easy for these business secrets or sensitive information to be stored in the database for a long time and be obtained by some people with ulterior motives.

[0003] An effective way to prevent commercial secrets or sensitive information from being stored in a database is to conduct regular and comprehensive checks on the entire database. Since the amount of data stored in a database is generally large, it is necessary to be able to quickly check the database content. Summary of the invention

[0004] To this end, the present invention provides a method and system for quickly checking database content to solve the problem that it is difficult to quickly screen and check sensitive information in a database.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] According to a first aspect of the present invention, a method for quickly checking database content is disclosed, the method comprising:

[0007] Connect to the database and determine whether the data needs to be checked;

[0008] When data checking is required, database records are obtained in batches from the database;

[0009] Allocate the batch-acquired records to a fixed-size memory block, which stores the batch-acquired table data in a fixed-length two-bit array;

[0010] Re-aggregate the table data into a continuous data;

[0011] After the aggregation is completed, the data is checked through a multi-mode matching algorithm, and the final inspection result is generated after the inspection is completed.

[0012] Furthermore, the specific process of allocating fixed-size memory blocks to the batch-acquired records is as follows:

[0013] The pre-allocated memory size is: n*(sum(max(len)), where n is the number of records obtained in batches; len is the maximum length of each field, and the memory block stores the table data obtained in batches in a two-dimensional array of fixed length.

[0014] Furthermore, before re-aggregating the data, since there is a large amount of useless data in the data stored in the two-dimensional memory block, the useless data needs to be eliminated, and the two-dimensional data stored by record and field is aggregated into a continuous data.

[0015] Furthermore, when the data is aggregated into a continuous data, each field is separated by a line break character \n, and a collection variable fieldsMap is used to store the memory head address corresponding to each record and each field.

[0016] Furthermore, after the aggregation is completed, the data is checked by a multi-mode matching algorithm. After the check is completed, the records and fields corresponding to the check results are found from fieldsMap to generate the final check results.

[0017] According to a second aspect of the present invention, a system for quickly checking database content is disclosed, the system comprising:

[0018] Data acquisition module, data distribution module, data aggregation module, data inspection module;

[0019] The data acquisition module acquires the data to be checked from the database;

[0020] The data allocation module allocates the acquired data to fixed-size memory blocks;

[0021] The data aggregation module aggregates the two-dimensional data stored in records and fields into a continuous data;

[0022] The data checking module checks the data through a multi-mode matching algorithm and outputs the checking result.

[0023] Furthermore, the data allocation module allocates the batch-acquired records to a fixed-size memory block, and the memory block stores the batch-acquired table data in a two-bit array of fixed length.

[0024] Furthermore, the data aggregation module removes useless data from the data, separates each field with a line break character \n, and uses a collection variable fieldsMap to store the memory head address corresponding to each record and each field.

[0025] Furthermore, after the inspection is completed, the data inspection module finds the records and fields corresponding to the inspection results from fieldsMap to generate the final inspection results.

[0026] The present invention has the following advantages:

[0027] The present invention discloses a method and system for quickly checking database content. Through unified memory allocation, batch query, data aggregation and two-dimensional field positioning technology, memory allocation and release operations in the checking process are greatly reduced, while the speed of multi-mode matching is greatly improved, thereby improving the overall performance of database checking. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0029] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0030] Figure 1 A flowchart of a method for quickly checking database content provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0031] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Example 1

[0033] This embodiment discloses a method for quickly checking database content, which is:

[0034] Connect to the database and determine whether the data needs to be checked;

[0035] When data checking is required, database records are obtained in batches from the database;

[0036] Allocate the batch-acquired records to a fixed-size memory block, which stores the batch-acquired table data in a fixed-length two-bit array;

[0037] Re-aggregate the table data into a continuous data;

[0038] After the aggregation is completed, the data is checked through a multi-mode matching algorithm, and the final inspection result is generated after the inspection is completed.

[0039] The specific process of allocating fixed-size memory blocks to batch-acquired records is as follows:

[0040] The pre-allocated memory size is: n*(sum(max(len)), where n is the number of records obtained in batches; len is the maximum length of each field, and the memory block stores the table data obtained in batches in a two-dimensional array of fixed length.

[0041] Before re-aggregating the data, since there is a lot of useless data in the two-dimensional memory block, it is necessary to remove the useless data and aggregate the two-dimensional data stored by record and field into a continuous data. Each field is separated by a line break character \n, and a collection variable fieldsMap is used to store the memory first address corresponding to each record and each field.

[0042] After the aggregation is completed, the data is checked through the multi-mode matching algorithm. After the check is completed, the records and fields corresponding to the check results are found from the fieldsMap to generate the final check results.

[0043] In general, when a multi-mode matching algorithm matches keywords in a text, it needs to calculate each keyword one by one during the matching process. In other words, each time a keyword is matched, it needs to scan the document from the beginning to the end again. The idea of ​​the AC automaton is to cache the following three situations through the word list at the beginning:

[0044] 1. Jump according to the successful character transfer (success table);

[0045] 2. Jump according to character transfer failure (fail table);

[0046] 3. Output table of successful matches (output table) Therefore, during the matching process, there is no need to match from the beginning of the document again, but to jump directly through the cache, thereby achieving a time complexity close to linear.

[0047] The construction process is divided into three steps, which are to build the success table, the fail table, and the output table. The output table is supplemented in the construction of the success and fail tables. The fail table is one-to-one, and the output table is one-to-many.

[0048] Jump according to the successful character transfer (success table). The success table is actually a trie tree, and its construction method is the same as that of the trie tree.

[0049] Jump according to character transfer failure (fail table). Let the letter on this node be C. Follow the failure pointer of its father until you reach a node whose son also has the letter C. Then point the failure pointer of the current node to the son with the letter C. If you have walked all the way to the root and still cannot find it, then point the failure pointer to the root. Use breadth-first search BFS to traverse the nodes hierarchically to process the failure path of each node.

[0050] In view of the characteristics of data storage in the database, the use of data aggregation and field two-dimensional positioning technology has greatly improved the performance of database data inspection and realized rapid data inspection.

[0051] Example 2

[0052] This embodiment discloses a system for quickly checking database content, the system comprising:

[0053] Data acquisition module, data distribution module, data aggregation module, data inspection module;

[0054] The data acquisition module acquires the data to be checked from the database;

[0055] The data allocation module allocates the acquired data to fixed-size memory blocks;

[0056] The data aggregation module aggregates the two-dimensional data stored in records and fields into a continuous data;

[0057] The data checking module checks the data through a multi-mode matching algorithm and outputs the checking result.

[0058] The data allocation module allocates fixed-size memory blocks to the records obtained in batches. The memory blocks store the table data obtained in batches in two-dimensional arrays with fixed lengths. The data aggregation module removes useless data from the data, separates each field with a line break character \n, and uses a collection variable fieldsMap to store the memory first address corresponding to each record and field. After the data inspection module completes the inspection, it finds the records and fields corresponding to the inspection results from fieldsMap and generates the final inspection results.

[0059] The present embodiment discloses a system for rapid database content inspection, which greatly reduces memory allocation and release operations during the inspection process through unified memory allocation, batch query, data aggregation and two-dimensional field positioning technology, and at the same time greatly improves the speed of multi-mode matching, thereby improving the overall performance of database inspection.

[0060] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for checking database content, characterized in that: The method is: Connect to the database and determine whether the data needs to be checked; When data checking is required, database records are obtained in batches from the database; Allocate the batch-acquired records to a fixed-size memory block, which stores the batch-acquired table data in a fixed-length two-bit array; Reaggregate the table data into a continuous data. Specifically, before reaggregating the table data, remove useless data, aggregate the two-dimensional data stored by records and fields into a continuous data, separate the fields with line breaks \n, and use a collection variable fieldsMap to store the memory first address corresponding to each record and field. After the aggregation is completed, the data is checked through a multi-mode matching algorithm, and the final inspection result is generated after the inspection is completed.

2. A method for checking database content according to claim 1, characterized in that: The specific process of allocating fixed-size memory blocks to the batch-acquired records is as follows: The pre-allocated memory size is: n*(sum(max(len)), where n is the number of records obtained in batches; len is the maximum length of each field, and the memory block stores the table data obtained in batches in a two-dimensional array of fixed length.

3. A method for checking database content according to claim 1, characterized in that: Before the data is re-aggregated, since there is a large amount of useless data in the data stored in the two-dimensional memory block, the useless data needs to be eliminated, and the two-dimensional data stored by record and field is aggregated into a continuous data.

4. A method for checking database content as claimed in claim 3, characterized in that: When the data is aggregated into a continuous data, each field is separated by a line break character \n, and a collection variable fieldsMap is used to store the memory first address corresponding to each record and each field.

5. A method for checking database content as claimed in claim 1, characterized in that: After the aggregation is completed, the data is checked by a multi-mode matching algorithm. After the check is completed, the records and fields corresponding to the check results are found from fieldsMap to generate the final check results.

6. A system for checking database content, characterized in that: The system comprises: Data acquisition module, data distribution module, data aggregation module, data inspection module; The data acquisition module acquires the data to be checked from the database; The data allocation module allocates the acquired data to fixed-size memory blocks; The data aggregation module aggregates the two-dimensional data stored by records and fields into a continuous data; specifically: before re-aggregating the table data, the useless data is removed, and the two-dimensional data stored by records and fields are aggregated into a continuous data, each field is separated by a line break character \n, and a collection variable fieldsMap is used to store the memory first address corresponding to each record and each field; The data checking module checks the data through a multi-mode matching algorithm and outputs the checking result.

7. A system for checking database contents as claimed in claim 6, characterized in that: The data allocation module allocates fixed-size memory blocks to the records acquired in batches, and the memory blocks store the table data acquired in batches in two-bit arrays of fixed length.

8. A system for checking database contents as claimed in claim 6, characterized in that: The data aggregation module removes useless data from the data, separates each field with a line break character \n, and uses a collection variable fieldsMap to store the memory head address corresponding to each record and each field.

9. A system for checking database contents as claimed in claim 6, characterized in that: After the inspection is completed, the data inspection module finds the records and fields corresponding to the inspection results from fieldsMap and generates the final inspection results.

Images