A universal deleted data positioning method and device based on SQLite fuzzy matching

Abstract

One aspect of the present invention discloses a universal deleted data locating method and device based on SQLite fuzzy matching. The method comprises: establishing a field header description template according to a recovered SQLite database; reading data in an original data file by block, fuzzily matching each block of data with the field header description template, and locating the data offset of the record; in response to a successful match between the data and the field header description template, reading the complete field data according to the location of the data offset and the data length described in the field header description template to obtain a data record result; and arranging the data record result by using a screening mechanism to locate the deleted data. The disclosed aspect of the present invention can realize the recovery capability of deleted data in different data structure forms, different operations, and different business scenarios, and realize the universality, comprehensiveness, effectiveness, and accuracy of SQLite database data recovery.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic evidence collection, and in particular to a universal deleted data locating method and device based on SQLite fuzzy matching. Background Art

[0002] With the application of information technologies such as network storage, Internet of Things, and video storage in people's daily work, life, and study, various storage media have become an indispensable part of people's lives. At the same time, as more and more criminal activities use related technologies to implement network production, copying and dissemination of pornographic and obscene audio and video, network fraud, network pyramid schemes, and various types of criminal cases that use the Internet to endanger national security, the criminal suspects' anti-detection awareness is constantly increasing, and they actively delete some key information involved in the case. Therefore, whether it can be successfully restored becomes crucial, and even affects whether the case can be successfully solved.

[0003] The latest revised Criminal Procedure Law and Civil Procedure Law have both listed electronic data as a new type of evidence. It is easy to imagine the important role of electronic data forensics and judicial appraisal in criminal and civil cases. Then, "how to recover deleted electronic data involved in the case" has become a difficult problem in obtaining evidence. In the information age, the recovery of electronic data has also become one of the most important technical means for various administrative and law enforcement agencies.

[0004] Due to the significant role of data extraction from mobile smart devices in electronic forensics, and the fact that core application data within these devices requires SQLite database storage technology, embedded devices are the primary application for SQLite, exemplified by Google's Android. Furthermore, desktop applications and websites are also increasingly using SQLite. The mobile application market is booming, and apps on nearly all mobile platforms, such as Android, iOS, Windows Mobile, Symbion, and Palm, use SQLite databases to store data. To address the problem of deleted data recovery on smartphones, particularly structured data generated by various applications, a common approach currently employed is to obtain the SQLite database file storing the relevant data, perform data analysis based on the database file, and then recover residual data from previously deleted tables by searching the unallocated space within the database file. This method relies on the integrity of the database file, and can only detect and recover data if the database does not immediately release storage space after deletion.

[0005] During normal use, an application generates a lot of data that is stored in SQLite storage, inserting data records into data tables. These data types generally include content that needs to be permanently saved, such as application settings and chat history, as well as temporarily generated data such as cookies, cache, network status, etc. Permanently saved data will remain saved without additional operations, while temporary data will be continuously added, updated, and deleted as the application is used. During the forensics process, both actively deleted data and temporary data that is deleted due to continuous updates and changes include records that need to be recovered. Therefore, traditional SQLite database data recovery methods have the following disadvantages:

[0006] In different application data, the data tables are different and the data formats are also different. If you follow the conventional method, you need to use feature codes. However, using feature codes to recover SQLite data requires adaptation and compatibility of each database file, which will be a huge amount of development and maintenance work.

[0007] The use of applications is often accompanied by high-frequency data deletion, which can cause disorder in the record-level data sequence stored in the database. In addition, new data is overwritten, making traditional SQLite database recovery methods difficult to implement.

[0008] While traditional SQLite database recovery techniques can relatively completely recover deleted data, numerous studies have shown that the recovered data often has a high duplication rate and contains some unrecognizable characters, commonly known as garbled characters. Therefore, even if data is successfully recovered, its usability is significantly reduced, significantly increasing the user's workload.

[0009] Traditional SQLite database recovery technology often misjudges deleted data, and the use of traditional data recovery technology will result in inaccuracies in the presentation of results. Electronic data forensics focuses on scientificity and rigor to ensure the objectivity, relevance and legality of converted evidence. Inaccuracies in data recovery results will ultimately affect the determination of facts.

[0010] Therefore, there is an urgent need to solve the problems of existing SQLite database recovery technology, such as the non-universality of data recovery, the incompleteness, low efficiency and ambiguity of data recovery results. Summary of the Invention

[0011] In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a universal deleted data locating method and device based on SQLite fuzzy matching.

[0012] To achieve the above object, the present invention proposes a universal deleted data locating method based on SQLite fuzzy matching, comprising the following steps:

[0013] Create a field header description template based on the restored SQLite database;

[0014] Read the data in the original data file in blocks, fuzzy match each block of data with the field header description template, and locate the data offset of the record;

[0015] In response to the data successfully matching the field header description template, reading the complete field data according to the positioning of the data offset and the data length described in the field header description template to obtain a data record result;

[0016] The data recording results are sorted using a screening mechanism to locate and delete data.

[0017] Preferably, the step of establishing a field header description template based on the restored SQLite database includes:

[0018] Establishing a field header information list, in response to manually describing the data type and additional information of the field header information in the field header information list, saving the described field header information list as a field header template; or,

[0019] A field header template is generated by automatically inferring field header information through a table creation statement, wherein the table creation statement includes but is not limited to field information, and the field header information includes but is not limited to data type and additional information.

[0020] Preferably, the step of reading the data in the original data file in blocks, fuzzy matching each block of data to the field header description template, and locating the data offset of the record includes:

[0021] Open the original data file, which includes the main db file and transaction cache file of the SQLite database;

[0022] Sequentially scan and read the data in the original data file in blocks, and match them with the field header description templates in sequence;

[0023] In response to a matching error of any field header information in the field header description template, the block of data is skipped and subsequent data is scanned until all field header information in the field header description template of the data is matched successfully.

[0024] Preferably, the use of a screening mechanism to sort the field data results and locate and delete data includes:

[0025] Query the rowid list of all undeleted records through the SQLite database, and scan the data record results to obtain the rowids of all data record results;

[0026] Compare the rowid of the data record result with the rowid list to determine whether the rowid of the data record result exists in the rowid list;

[0027] If a rowid of the data record result exists in the rowid list, compare the data content corresponding to the rowid of the data record result with the data content corresponding to the rowid in the rowid list corresponding to the rowid of the data record result to see whether they are the same;

[0028] In response to the compared data contents being identical, marking the data record result as a non-deleted record;

[0029] In response to the compared data contents being different, marking the data record result as a modified record;

[0030] If the rowid of a data record result does not exist in the rowid list, mark the data record result as a deleted record, and obtain the location of the deleted data;

[0031] The data records marked for deletion include records with missing rowids.

[0032] Preferably, the method of using a screening mechanism to sort the data recording results and locate and delete data further includes:

[0033] According to the rowid loss record, compare the rowid values ​​of the undeleted records on the left and right sides of the rowid loss record;

[0034] If the rowid-missing record is deleted individually, select a starting point on either side based on the offset of the leftmost or rightmost data describing the record in the data subpage, read the records sequentially according to their rowids, and infer the location of the rowid-missing record;

[0035] If multiple records with missing rowids are deleted in batches or are frequently written or deleted continuously, select the leftmost and rightmost starting points, read the records sequentially according to their rowids, and infer two positions respectively. If the two positions are the same, either result is the position of the record with missing rowid.

[0036] Preferably, the method of using a screening mechanism to sort the data recording results and locate and delete data further includes:

[0037] In response to the deleted data being overwritten by the new data, a secondary screening is performed using the field information of the business data.

[0038] Preferably, the method of using a screening mechanism to sort the data recording results and locate and delete data further includes:

[0039] In response to a record update operation, field data of a new record is generated, and the field data of the undeleted record is compared with the field data of the new record. If the field data of the undeleted record is different from the field data of the new record, the undeleted record is marked as a residual record before the record update, and the data location before the record update is obtained.

[0040] Preferably, the data in the original data file also includes a user data record table in the FTS extension module.

[0041] Before reading the data in the original data file in blocks, fuzzy matching each block of data with the field header description template, and locating the data offset of the record, the method further includes:

[0042] When reading sub-page data, the sub-page data is dynamically decrypted and used directly, and then released directly after use.

[0043] To achieve the above-mentioned object, the present invention further proposes a general deleted data location device based on SQLite fuzzy matching, comprising:

[0044] Field header description module creation module, used to create a field header description template based on the restored SQLite database;

[0045] The field type fuzzy matching module is used to read the data in the original data file in blocks, fuzzy match each block of data with the field header description template, and locate the data offset of the record

[0046] The data record result reading module is used to respond to the successful matching of the data with the field header description template, read the complete field data according to the positioning of the data offset and the data length described in the field header description template, and obtain the data record result

[0047] The data record result screening module is used to sort the data record results by using a screening mechanism and locate and delete data.

[0048] Compared with the prior art, the beneficial effects brought about by one aspect disclosed in the present invention are:

[0049] (1) Through fuzzy matching, the deleted data is scanned and located comprehensively, effectively solving the problem of difficulty in recovering deleted records caused by high-frequency data deletion and new data writing and overwriting, without losing any relevant residual fields, thereby fundamentally ensuring the comprehensiveness and completeness of data recovery results.

[0050] (2) More screening conditions are added. By screening the recovered data layer by layer, duplicate data and unrecognizable character data (garbled characters) are reasonably discarded, making the final recovered data effective and improving its availability.

[0051] (3) While integrating the recovered data, the recovered data is verified from multiple aspects, and the verified data is recorded and finally presented in the recovery results to ensure the accuracy of the recovery results, effectively avoid the uncertainty that may arise when database-based recovery technology is used in electronic data forensics, and improve traditional recovery technical means. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flowchart of the steps of a general deleted data location method based on SQLite fuzzy matching disclosed by the present invention;

[0053] Figure 2 This is a flowchart of the steps of locating and deleting data according to an embodiment of the present invention;

[0054] Figure 3 This is a flowchart of the steps for screening duplicate data according to an embodiment disclosed in the present invention;

[0055] Figure 4 A flowchart of the steps of mining potential duplicate data according to an embodiment of the present invention;

[0056] Figure 5 This is a flowchart of the steps of a block-by-block matching field header description module according to an embodiment disclosed in the present invention;

[0057] Figure 6 This is a flowchart of the steps for restoring residual data of an FTS extension module according to an embodiment of the present invention;

[0058] Figure 7 This is a structural diagram of a universal deleted data locating device based on SQLite fuzzy matching disclosed by the present invention. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention using specific examples and accompanying drawings. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through other different specific examples, and the details in this specification may be modified and altered based on different viewpoints and applications without departing from the spirit of the present invention.

[0060] Figure 1 This is a flowchart of a general method for locating deleted data based on SQLite fuzzy matching disclosed by the present invention. Figure 1As shown, the present invention provides a universal deleted data location method based on SQLite fuzzy matching, comprising the following steps:

[0061] Step S101: Create a field header description template based on the restored SQLite database.

[0062] In one embodiment disclosed in the present invention, manually generating a field header description template specifically includes the following steps:

[0063] A field header information list is created, and in response to manually describing the data type and additional information of the field header information in the field header information list, the described field header information list is saved as a field header description template. In other words, the field header description information is manually generated and described, and after the field header information list is created, its type is described one by one and the template is saved.

[0064] In another embodiment disclosed in the present invention, a field header description template is automatically generated by inferring a table creation statement, specifically comprising the following steps:

[0065] A field header description template is generated by automatically inferring field header information through a table creation statement. The table creation statement can be abstracted as "create table [condition] table name (field information) [additional information]", and the abstract format of the field header information is "field name [data type] [additional information]".

[0066] Among them, in the abstract format of the field header information, "field name" is a required item, "type" is usually filled in, and "additional information" is usually default values, constraints, primary keys and other information that can be filled in and used.

[0067] It should be noted that since SQLite is a database with weak data types, the type of data actually stored may not match the field type described in the table creation statement. It is even possible to describe only the field name without describing the field type when creating the table. When inserting data, the actual type is dynamically confirmed and written into the data file for storage. For example, the table creation statement describes the field type as int integer, but it is also feasible to directly store string data in the field during use without any error. In the SQLite database, the field type and field length are combined to form the field header information, which in turn forms a field header description template. When SQLite reads data, the field header information of the field header description template can be parsed to determine the actual type of the field and the actual length of the data. Combined with the subsequent data, a record can be queried.

[0068] If a data anomaly causes byte errors, a normal read will result in unrecognized data or a direct data error, preventing further querying. Using the summarized field header value type, you can reverse-infer the data type and length. In a complete record, the types and lengths of multiple fields comprise the field header data. The types and lengths of fields 1 through 9 correspond to the following data 1 through 9. Even if the record has been deleted and the record data structure header has been overwritten, the field header can still be used to match the data type and length, accurately reconstructing the complete record content.

[0069] Based on this, in an embodiment of the present invention, by creating a field header description template, there is no need to perform additional compatibility operations to accommodate various data residue situations.

[0070] Referring to Table 1, the field types used in one embodiment of the present invention include the following:

[0071] type describe NULL The field content is empty and matches alone Primary key (PRIMARY) The auto-increment primary key is usually rowid Integer (INT) Integer types Floating-point numbers (FLOAT) Floating-point type STRING String type, empty string is not NULL Binary data (BLOB) Binary data types Any type (ANY) Used when the field type is default, matching any type of data

[0072] Table 1

[0073] In an embodiment of the present invention, the field type of certain fields can be modified to ANY to address data type differences between new and old application versions and improve compatibility. Furthermore, in addition to field types, additional restrictions such as non-empty, non-NULL, ranges, or fixed values ​​can be added to field descriptions to improve scan matching efficiency and success rate.

[0074] Step S102 , reading data from the original data file in blocks, fuzzy matching each block of data with the field header description template, and locating the data offset of the record.

[0075] Optionally, a multi-threaded approach is used for fuzzy matching to improve scanning efficiency.

[0076] The present invention discloses an embodiment of locating and deleting data. Figure 2 This is a flowchart of the steps of locating and deleting data according to an embodiment of the present invention. Figure 2 As shown, step S2 further includes the following steps:

[0077] Step S1021: Open the original data file, which includes the main db file and transaction cache file of the SQLite database.

[0078] Optionally, open the main db file and transaction cache file of the SQLite database in read-only + shared mode to prevent the original data from being modified due to careless operations.

[0079] Step S1022 , sequentially scan and read the data in the original data file in blocks, and perform fuzzy matching with the field header description templates in sequence.

[0080] Step S1023: In response to a matching error of any field header information in the field header description template, skip the block of data and continue to scan subsequent data until all field header information in the field header description template of the data is matched successfully.

[0081] Step S103 , in response to the data successfully matching the field header description template, the complete field data is read according to the location of the data offset and the data length described in the field header description template to obtain the location of the deleted data.

[0082] Specifically, the field header information is based on a combination of field type and field length. When reading data, the data type and data length can be obtained by reading the field header information, and a record can be queried.

[0083] Step S104: sorting the data record results using a screening mechanism to locate and delete data.

[0084] Specifically, through step S102 to step S103, fuzzy matching is performed, and the obtained data record results are all data records in the SQLite data table, including all deleted and non-deleted records. In addition, in the case of some data page changes and caches, the matched records also contain a lot of duplicate data. In this case, it is necessary to filter and dedupe the data record results, extract some non-deleted records, and accurately locate the deleted data.

[0085] The present invention discloses an embodiment of filtering duplicate data. Figure 3 This is a flowchart of the steps of screening duplicate data according to an embodiment of the present invention. Figure 3 As shown, step S104 includes:

[0086] Step S10411, query the rowid list of all undeleted records through the SQLite database, scan the data record results to obtain the rowids of all data record results.

[0087] Step S10412: compare the rowid of the data record result with the rowid list to determine whether the rowid of the data record result exists in the rowid list.

[0088] Step S10413: If a rowid of the data record result exists in the rowid list, compare the data content corresponding to the rowid of the data record result with the data content corresponding to the rowid in the rowid list corresponding to the rowid of the data record result to see if they are the same.

[0089] Step S10414: In response to the compared data contents being identical, the data record is marked as a non-deleted record.

[0090] Step S10415: In response to the fact that the compared data contents are different, the data record result is marked as a modified record.

[0091] Step S10416: If the rowid of a certain data record result does not exist in the rowid list, mark the data record result as a deleted record, and obtain the location of the deleted data.

[0092] The data records marked for deletion include records with missing rowids.

[0093] That is to say, by utilizing the uniqueness of rowid, the rowid list of all non-deleted records is normally queried through SQLite, and then the data file is scanned to match all records and obtain the rowid of all records, and then compared with the existing rowid list. If the rowid exists, the data content is compared again. If it is consistent, it means that the record has not been deleted, and if it is inconsistent, it means that the record content may have been modified. If the rowid of the scanned record is not in the existing rowid list, then the record is marked as deleted. In addition, in some cases, the deleted record header is lost due to being overwritten, and the record is also marked as a deleted record. This embodiment can screen duplicate data and eliminate a large amount of duplicate deleted data through steps S10411 to S10416.

[0094] In an embodiment of the present invention, mining of potential duplicate data is also disclosed. Figure 4 This is a flowchart of the steps of mining potential duplicate data according to an embodiment of the present invention. Figure 4 As shown, step S104 further includes the following steps:

[0095] Step S10421: Based on the rowid missing record, compare the rowid values ​​of the undeleted records on the left and right sides of the rowid missing record.

[0096] In step S10422, if the rowid missing record is deleted individually, select a starting point on either side according to the offset of the leftmost data or the rightmost data describing the record in the data subpage, read the records in sequence according to their rowids, and infer the location of the rowid missing record.

[0097] In step S10423, if multiple records with missing rowids are deleted in batches or are frequently written or deleted continuously, select the leftmost and rightmost starting points, read the records in sequence according to their rowids, and infer two positions respectively. When the two positions are the same, either result is the position of the missing rowid record.

[0098] Specifically, in some cases, the missing rowid's business-related conditions may cause the record data to become garbage data. Therefore, it is necessary to find the original rowid for business matching. In other words, it is necessary to locate the location of the record with the missing rowid. On the same data subpage, to the left of the missing rowid record is a record with a larger rowid, and to the right is a record with a smaller rowid. The data subpage contains offsets describing the leftmost and rightmost data of the record. Finding the starting points on both sides, read sequentially according to the record rowid until the location of the missing rowid record is inferred. The inferred rowid is the missing rowid value.

[0099] Optionally, in the case of a single deleted record, the missing rowid value can be inferred from either the left or right side. However, in the case of batch deletion of some records or frequent "write-delete-write-delete" operations (deleting while chatting), the rowid available for inference on one side may be intermittent or have significant differences, and the resulting value is less reliable. Rigorousness is strictly emphasized in electronic data forensics, so this embodiment uses simultaneous inference on both sides. When the rowid inferred from the left and right sides is the same, it is temporarily marked as the temporary rowid for the record. Mutual verification of the results of simultaneous inference on both sides can improve the accuracy of the results, reduce the rowid error rate, and avoid data logic processing errors.

[0100] In this embodiment, steps S10421 to S10423 utilize a deleted data rowid inference mechanism to mine potential duplicate data and improve the quality of deleted data recovery.

[0101] The present invention also discloses an embodiment of filtering invalid data, which includes:

[0102] In response to the deleted data being overwritten by the new data, a secondary screening is performed using the field information of the business data.

[0103] Specifically, since the deleted data is an idle address and can be overwritten by new data, the matched content is usually incomplete or partially overwritten, resulting in garbled characters, etc., and due to the interference of cache and other data, even after the preliminary screening of duplicate data in steps S10411 to S10416, there are still data record results that contain some garbled characters and duplicate records. During the recovery process, the role of the target data table and the meaning of some fields are very clear, and according to the actual use of the application, there are many fields that represent fixed meanings, so the field information of the business data can be used as a secondary screening and verification to increase the data quality of the recovery results. For example, if some fields store timestamps, you can add a legal time condition judgment on the basis of the fuzzy matching integer. If a field stores a mobile phone number, you can add a verification of the phone number format on the basis of the fuzzy matching string. In actual use, for example, in the scenario of recovering a mobile phone short message database, you can filter out garbled records by checking whether the other party's number field is legal, and then filter out duplicate content by the time and content of the short message. Only minimal adaptation and filtering are required to remove obvious errors, garbled characters and other junk data in the result data, making the forensic results clearer and more effective.

[0104] The present invention also discloses an embodiment of restoring the record of the update operation, wherein step S104 further includes:

[0105] In response to a record update operation, field data of a new record is generated, and the field data of the undeleted record is compared with the field data of the new record. If the field data of the undeleted record is different from the field data of the new record, the undeleted record is marked as a residual record before the record update, and the data location before the record update is obtained.

[0106] Specifically, the record update operations are operations such as withdrawal, refresh, transaction cache, and page type conversion. For example, in WeChat, when a message is withdrawn, the database operation level is to update the data in the original chat content field to "You withdrew a message". By tracking the withdrawal operation, the database usually has the following two situations:

[0107] Case 1: If the length of the new value is less than or equal to the length of the original value, the original data area of ​​the field to be updated will be directly operated on, and the original data will be overwritten with the new data. In this case, the original field content is completely lost, the data of other unchanged fields remains unchanged, the record rowid remains unchanged, and the portion of the length not occupied by the new value becomes fragmented.

[0108] Case 2: The length of the new value is greater than that of the original value. In this case, the original data block is not large enough to store the new value. A complete new record is inserted, and the original data record becomes a free block. This is equivalent to inserting a complete new record and deleting the original record. The original record data is completely retained, and the rowid of the new record remains unchanged.

[0109] In this embodiment, based on fuzzy matching, the rowid of the record marked as undeleted after fuzzy matching is compared with the field data of the new record. If there is any inconsistency, the undeleted record is the residual content before the modification and update. In the general recovery matching process, multiple groups of record data with the same rowid may be located. If the data content is inconsistent, it may be an updated message. At this time, the current data and the data before the update can be distinguished by comparing them based on the actual queried data, which can further improve the efficiency of obtaining data using this recovery method.

[0110] In this embodiment, the above steps can achieve the retention of field data before the record is updated, thereby improving the ability to recover deleted data under specific business operations.

[0111] In another embodiment disclosed in the present invention, a block matching field header description module is provided. Figure 5 This is a flowchart of the steps of the block matching field header description module according to an embodiment of the present invention. Figure 5 As shown, the following steps are included:

[0112] Step S201: Automatically infer and generate a field header description template through table creation statements.

[0113] serial number type Additional conditions 1 primary key Not empty 2 Integer Not empty, fixed length is 4 3 Integer 4 String Not empty 5 Binary data 6 Any type Not NULL 7 Integer

[0114] Table 2

[0115] Step S202: Open the original data file in a read-only + shared mode, wherein the original data file includes the main db file and transaction cache file of the SQLite database.

[0116] Step S203 , sequentially scan and read the data in the original data file in blocks, and perform fuzzy matching with the field header description template in sequence.

[0117] Step S204: When any field header type or additional condition in the field header description template matches incorrectly, the block of data is skipped and the subsequent data is scanned.

[0118] Step S205: When all seven fields are matched successfully, the location of a record data is completed.

[0119] The present invention also discloses an embodiment of recovering residual data in the FTS extension module. Figure 6 This is a flowchart of the steps for restoring residual data of the FTS extension module according to an embodiment of the present invention. Figure 6 As shown, this embodiment also includes:

[0120] Step S301, creating a field header description template based on the restored SQLite database;

[0121] Step S302, reading the user data record table in the FTS extension module, fuzzy matching each block of data in the user data record table with the field header description template, and locating the data offset of the record;

[0122] Step S303: In response to the data successfully matching the field header template, the complete field data is read according to the data length described in the field header template to obtain the FTS data record result;

[0123] Step S304: sorting the FTS data record results using a screening mechanism to locate data residues in the FTS extension module.

[0124] Specifically, the FTS extension module, which is the FTS full-text search extension module built into SQLite, has the function of accelerating full-text searches when the amount of data stored is too large. Compared to ordinary tables, FTS is a virtual table. When an FTS virtual table is created, several ordinary data tables are actually created in the database to store physical data. These are called shadow tables, and these shadow tables are independent database files. After the extension is started, new shadow tables will be created in the newly generated ftsdb, named t_content, t_messageize, t_segdir, t_segments, etc., among which the t_content table is a table containing user data records. The t_content table can be used to locate data residues in the FTS extension module according to the fuzzy matching method and screening mechanism described above.

[0125] The present invention further discloses a method for dynamically decrypting sub-page data according to an embodiment. In this embodiment, the method includes all steps from step S101 to step S104. The same steps will not be repeated here. Between step S101 and step S102, the method further includes:

[0126] When reading sub-page data, the sub-page data is dynamically decrypted and used directly, and then released directly after use.

[0127] Specifically, mobile applications often encrypt their own application data to ensure user data security. After encryption is enabled, the application performs encryption and decryption operations during reading and writing, respectively. The data is stored in the encrypted ciphertext form. The ciphertext directly read cannot be directly used for fuzzy matching. Therefore, it is necessary to decrypt the encrypted DB file (including the transaction cache and FTS extension file) into the original data, and ensure that the page numbers are not out of order. Then, fuzzy matching is performed on the original SQLite storage data. In an embodiment, by dynamically decrypting the page data when reading the sub-page data and then using it directly, and releasing it directly after use, it is possible to achieve full memory direct operation without the need for hard disk caching.

[0128] The present invention also provides a general deleted data positioning device based on SQLite fuzzy matching, Figure 7 This is a structural diagram of a general deleted data positioning device based on SQLite fuzzy matching disclosed by the present invention, such as Figure 7 As shown, the device includes:

[0129] The field header description module creation module 71 is used to create a field header description template according to the restored SQLite database.

[0130] The field type fuzzy matching module 72 is used to read the data in the original data file in blocks, fuzzy match each block of data with the field header description template, and locate the data offset of the record.

[0131] The data record result reading module 73 is used to read the complete field data according to the data offset and the data length described in the field header description template in response to the successful matching of the data with the field header description template, and obtain the data record result.

[0132] The data record result screening module 74 is used to sort the data record results using a screening mechanism and locate and delete data.

[0133] Preferably, the field header description module creation module 71 is further used to establish a field header information list, and in response to manually describing the data type and additional information of the field header information in the field header information list, save the described field header information list as a field header template.

[0134] Preferably, the field header description module creation module 71 is further used to automatically infer field header information through table creation statements to generate field header templates, wherein the table creation statements include but are not limited to field information, and the field header information includes but is not limited to data types and additional information.

[0135] Preferably, the field type fuzzy matching module 72 is further configured to:

[0136] Open the original data file, which includes the main db file and transaction cache file of the SQLite database;

[0137] Sequentially scan and read the data in the original data file in blocks, and match them with the field header description templates in sequence;

[0138] In response to a matching error of any field header information in the field header description template, the block of data is skipped and subsequent data is scanned until all field header information in the field header description template of the data is matched successfully.

[0139] Preferably, the data record result screening module 74 is further configured to:

[0140] Query the rowid list of all undeleted records through the SQLite database, and scan the data record results to obtain the rowids of all data record results;

[0141] Compare the rowid of the data record result with the rowid list to determine whether the rowid of the data record result exists in the rowid list;

[0142] If a rowid of the data record result exists in the rowid list, compare the data content corresponding to the rowid of the data record result with the data content corresponding to the rowid in the rowid list corresponding to the rowid of the data record result to see whether they are the same;

[0143] In response to the compared data contents being identical, marking the data record result as a non-deleted record;

[0144] In response to the compared data contents being different, marking the data record result as a modified record;

[0145] If the rowid of a data record result does not exist in the rowid list, mark the data record result as a deleted record, and obtain the location of the deleted data;

[0146] The data records marked for deletion include records with missing rowids.

[0147] Preferably, the data record result screening module 74 is further configured to:

[0148] According to the rowid loss record, compare the rowid values ​​of the undeleted records on the left and right sides of the rowid loss record;

[0149] If the rowid-missing record is deleted individually, select a starting point on either side based on the offset of the leftmost or rightmost data describing the record in the data subpage, read the records sequentially according to their rowids, and infer the location of the rowid-missing record;

[0150] If multiple records with missing rowids are deleted in batches or are frequently written or deleted continuously, select the leftmost and rightmost starting points, read the records sequentially according to their rowids, and infer two positions respectively. If the two positions are the same, either result is the position of the record with missing rowid.

[0151] Preferably, the data record result screening module 74 is further configured to:

[0152] In response to the deleted data being overwritten by the new data, a secondary screening is performed using the field information of the business data.

[0153] Preferably, the data record result screening module 74 is further configured to:

[0154] In response to a record update operation, field data of a new record is generated, and the field data of the undeleted record is compared with the field data of the new record. If the field data of the undeleted record is different from the field data of the new record, the undeleted record is marked as a residual record before the record update, and the data location before the record update is obtained.

[0155] Preferably, in the data record result screening module 74, the data in the original data file also includes the user data record table in the FTS extension module.

[0156] Preferably, the device also includes a dynamic decryption sub-page data module, which is used to read the data in the original data file in blocks, fuzzy match each block of data with the field header description template, and locate the data offset of the record before reading the sub-page data. When reading the sub-page data, the sub-page data is dynamically decrypted for direct use and released directly after use.

[0157] It should be noted that the above-mentioned device corresponds to the universal deleted data positioning method based on SQLite fuzzy matching provided by the present invention. Other descriptions can refer to the description of the universal deleted data positioning method based on SQLite fuzzy matching provided by the present invention, and will not be repeated here.

[0158] It can be seen that the present invention provides a universal deleted data locating method and device based on SQLite fuzzy matching, by establishing a field header description template, fuzzy matching the data in the original data file with the field header description template, obtaining a data record result after a successful match, and then screening and sorting the data record result to accurately locate the deleted data, and can match and locate various types of data residues; wherein, the field header description template can be manually generated or automatically generated using a table creation statement; using the uniqueness of rowid, duplicate data is preliminarily screened to obtain records marked as deleted, records marked as modified, and records marked as not deleted, and the records marked as deleted are the deleted data that have undergone initial deduplication; By utilizing the deleted data rowi inference mechanism, the location of the rowid-lost record can be inferred based on the rowid values ​​of the undeleted records on the left and right sides of the rowid-lost record, thereby realizing the mining of potential duplicate data; by utilizing the invalid data filtering mechanism, when the deleted data is overwritten by new data, secondary screening is performed through the field information of the business data; by utilizing the updated record recovery technology, based on fuzzy matching, the rowid marked as the undeleted record after fuzzy matching is compared with the field data of the new record to obtain the residual content before the record was updated; if the FTS extension module is enabled, the user data record table in the FTS extension module will be used to locate the data residue in the FTS extension module through the above-mentioned fuzzy matching and screening mechanism. In addition, the method of dynamically decrypting subpages is used as an encrypted database preprocessing technology, without the need for external caching. The above aspects disclosed by the present invention can achieve the comprehensiveness, effectiveness and accuracy of recovered data.

[0159] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Any skilled artisan may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be as set forth in the appended claims.

Claims

1. A general deleted data location method based on SQLite fuzzy matching, comprising the following steps: Create a field header description template based on the restored SQLite database; Read the data in the original data file in blocks, fuzzy match each block of data with the field header description template, and locate the data offset of the record; In response to the data successfully matching the field header description template, reading the complete field data according to the positioning of the data offset and the data length described in the field header description template to obtain a data record result; Using a screening mechanism to sort out the data record results and locate and delete the data; The step of establishing a field header description template based on the restored SQLite database includes: Establishing a field header information list, in response to manually describing the data type and additional information of the field header information in the field header information list, saving the described field header information list as a field header template; or, Generating a field header template by automatically inferring field header information through a table creation statement, wherein the table creation statement includes but is not limited to field information, and the field header information includes but is not limited to data type and additional information; reading data in the original data file in blocks, fuzzy matching each block of data with the field header description template, and locating the data offset of the record, including: Open the original data file, which includes the main db file and transaction cache file of the SQLite database; Sequentially scan and read the data in the original data file in blocks, and match them with the field header description templates in sequence; In response to a matching error of any field header information in the field header description template, the block of data is skipped and subsequent data is scanned until all field header information in the field header description template of the data is matched successfully.

2. A general deleted data location method based on SQLite fuzzy matching as claimed in claim 1, characterized in that: The method of using a screening mechanism to sort the field data results and locate and delete data includes: Query the rowid list of all undeleted records through the SQLite database, and scan the data record results to obtain the rowids of all data record results; Compare the rowid of the data record result with the rowid list to determine whether the rowid of the data record result exists in the rowid list; If a rowid of the data record result exists in the rowid list, compare the data content corresponding to the rowid of the data record result with the data content corresponding to the rowid in the rowid list corresponding to the rowid of the data record result to see whether they are the same; In response to the compared data contents being identical, marking the data record result as a non-deleted record; In response to the compared data contents being different, marking the data record result as a modified record; If the rowid of a data record result does not exist in the rowid list, mark the data record result as a deleted record, and obtain the location of the deleted data; The data records marked for deletion include records with missing rowids.

3. A general deleted data location method based on SQLite fuzzy matching as claimed in claim 2, characterized in that: The method of using a screening mechanism to sort out the data recording results and locate and delete data also includes: According to the rowid loss record, compare the rowid values ​​of the undeleted records on the left and right sides of the rowid loss record; If the rowid-missing record is deleted individually, select a starting point on either side based on the offset of the leftmost or rightmost data describing the record in the data subpage, read the records sequentially according to their rowids, and infer the location of the rowid-missing record. If multiple records with missing rowids are deleted in batches or are frequently written or deleted continuously, select the leftmost and rightmost starting points, read the records sequentially according to their rowids, and infer two positions respectively. If the two positions are the same, either result is the position of the record with missing rowid.

4. A general deleted data location method based on SQLite fuzzy matching as claimed in claim 1, characterized in that: The method of using a screening mechanism to sort out the data recording results and locate and delete data also includes: In response to the deleted data being overwritten by the new data, a secondary screening is performed using the field information of the business data.

5. A general deleted data location method based on SQLite fuzzy matching as claimed in claim 2, characterized in that: The method of using a screening mechanism to sort out the data recording results and locate and delete data also includes: In response to a record update operation, field data of a new record is generated, and the field data of the undeleted record is compared with the field data of the new record. If the field data of the undeleted record is different from the field data of the new record, the undeleted record is marked as a residual record before the record update, and the data location before the record update is obtained.

6. A general deleted data location method based on SQLite fuzzy matching as claimed in claim 1, characterized in that: The data in the original data file also includes a user data record table in the FTS extension module.

7. A general deleted data location method based on SQLite fuzzy matching as claimed in claim 1, characterized in that: Before reading the data in the original data file in blocks, fuzzy matching each block of data with the field header description template, and locating the data offset of the record, the method further includes: When reading sub-page data, the sub-page data is dynamically decrypted and used directly, and then released directly after use.

8. A universal deleted data locating device based on SQLite fuzzy matching, comprising: A field header description module creation module is used to establish a field header description template based on the restored SQLite database: establish a field header information list, and in response to manually describing the data type and additional information of the field header information in the field header information list, save the described field header information list as a field header template; or automatically infer the field header information to generate the field header template through a table creation statement, the table creation statement including but not limited to field information, and the field header information including but not limited to data type and additional information; A field type fuzzy matching module is used to read data in the original data file in blocks, fuzzy match each block of data with the field header description template, and locate the data offset of the record: open the original data file, which includes the main db file and transaction cache file of the SQLite database; Sequentially scan and read the data in the original data file in blocks, and match them with the field header description templates in sequence; in response to a match error in any field header information in the field header description template, skip the block of data and continue scanning subsequent data until all field header information in the field header description template of the data is successfully matched; a data record result reading module, configured to read the complete field data according to the data offset and the data length described in the field header description template in response to a successful match between the data and the field header description template, to obtain a data record result; The data record result screening module is used to sort the data record results by using a screening mechanism and locate and delete data.

Images