Method and device for parsing multiple tables of SQLite based on policy registry

Abstract

The application relates to a policy registry-based SQLite multi-table parsing method and device, and is applied to the technical field of data processing, and comprises the following steps: a parsing policy object is obtained according to a table name to execute a uniform life cycle hook, the parsing logic of each table is separated from core scheduling, the newly-added table structure only needs to add a policy object and register, and therefore the expandability of multi-table parsing is greatly improved; a large file table is separated from initial parsing and is transferred into a large file message queue for asynchronous processing through large file determination, memory overflow and timeout caused by full loading of a GB-level large table are avoided; a ROWID cursor is used for batch loading for a table supporting batch processing, the cursor paging and the parsing batch processing depth are coupled, and performance collapse of offset-based paging in deep paging is avoided; a deletion condition determination is performed when three conditions of no unprocessed large file table, no error table and configuration opening deletion are simultaneously established, it is ensured that a source file will not be mistakenly deleted, and a data source is reserved for failure retry.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method and apparatus for parsing multiple tables in SQLite based on a policy registry. Background Technology

[0002] In the data platform scenario of instant messaging applications, the local SQLite database on the mobile device typically contains multiple heterogeneous data tables, such as message tables, friend tables, avatar tables, Moments tables, red envelope tables, group chat tables, application message tables, and mass message tables. The backend service needs to download this SQLite database from Object Storage Service (OSS), and parse, transform, and persist each of its internal tables to heterogeneous storage such as MongoDB, MySQL, Elasticsearch, and OLAP to support subsequent statistics, retrieval, and relational queries.

[0003] Existing technologies for SQLite multi-table parsing and large file processing mainly suffer from the following three shortcomings:

[0004] First, the scalability of multi-table parsing is poor: While some existing solutions employ a combination of mapping tables and templated functions, the mapping key is the protocol message ID rather than the table name. The parsing logic is concentrated in the core module, and adding a new table structure requires modification of the core code. Other technologies use a registration strategy based on content type (text / table / image), which cannot be directly migrated to relational database scenarios where table names are used for registration. When the database contains multiple heterogeneous tables and the field structure evolves with each version, existing solutions suffer from high expansion costs and are prone to introducing regression defects.

[0005] Secondly, the large file parsing suffers from insufficient resistance to OutOfMemoryError (OOM) and timeouts: existing large file parsing solutions either rely on multi-node distributed architectures (unavailable on single machines) or GPU hardware acceleration (poor versatility). When a single table contains 50 million to 100 million data entries, occupying GB-level storage, a full load will inevitably lead to an OutOfMemoryError (OOM). Furthermore, existing solutions cannot retain and accumulate the initial parsing time in subsequent parsing sessions after the initial interruption due to a large file, resulting in distorted operational observations.

[0006] Third, cursor pagination is decoupled from parsing batch processing and lacks orchestration capabilities: Existing cursor pagination solutions are mainly used for query scenarios and are not combined with multi-table parsing batch processing. They also generally lack engineering availability constraints such as table order priority orchestration, error table isolation, and consistency of object storage source file deletion timing.

[0007] Therefore, there is an urgent need for a SQLite multi-table parsing and large file hierarchical asynchronous processing method that can solve the above problems. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide a SQLite multi-table parsing method and apparatus based on a policy registry, aiming to solve the problems in the prior art, such as poor scalability of multi-table parsing, insufficient OOM resistance for large file parsing, decoupling of cursor pagination and parsing batch processing, and lack of orchestration capabilities.

[0009] According to a first aspect of the present invention, a method for parsing multiple SQLite tables based on a policy registry is provided, the method comprising: Download the SQLite file from the object storage service to your local working directory; The set of table names to be processed is parsed from the SQLite file, and each table in the set of table names is sorted according to a preset priority function, which is used to indicate the priority of each table in the parsing order. Traverse each table in the sorted set of table names, and for the target table currently being traversed, obtain the parsing policy object of the target table through the policy registry; The initial parsing is performed based on the parsing strategy object of the target table, including: determining whether the target table is a large file table based on the parsing strategy object of the target table; if so, and the current state is not in large file processing mode, then the target table is added to the large file table set and the current parsing is skipped; if not, determining whether the target table supports batch parsing based on the parsing strategy object of the target table; if so, performing cursor batch loading parsing on the target table; if not, performing full loading parsing on the target table. After the initial parsing is completed, a second parsing of the large file is performed, including: when the large file table set is not empty, adjusting the current state to large file processing mode; sending the large file table set and the initial parsing time to the large file message queue; and, based on the large file message queue, using the current large file to be parsed as the target table, performing cursor-based batch loading and parsing on the target table. When the large file table set is empty and the error table set is empty, it is determined whether the source file deletion condition of the SQLite file is met. If the deletion condition is met, the source file of the SQLite file is deleted. The error table set is used to store the table names of the initial parsing and large file secondary parsing errors. The deletion condition is that all three conditions are met simultaneously: no unprocessed large file table, no error table, and deletion is enabled.

[0010] Preferably, it further includes: After the initial parsing is completed, obtain the initial parsing time. After the secondary parsing of the large file is completed, the time taken for the secondary parsing of the large file is obtained; The total time is obtained by adding the initial parsing time to the secondary parsing time.

[0011] Preferably, The step of obtaining the parsed policy object of the target table through the policy registry includes: The corresponding parsing policy object is found in the policy registry using the table name of the target table as the key. The policy registry uses the table name as the key and the parsing policy object as the value. The parsing policy object implements a unified lifecycle hook interface, which includes at least the following methods: skip judgment method, large file judgment method, batch support judgment method, full load method, cursor batch load method, pre-save conversion method, save method, and post-save processing method.

[0012] Preferably, The step of performing cursor-based batch loading and parsing on the target table includes: The ROWID cursor is used to execute the path in batches, looping through the following steps with a preset page size: Data is loaded in batches according to the current cursor, and the pre-save conversion method, the save method, and the post-save processing method are executed sequentially on the currently loaded batches of data. Update the cursor to the ROWID of the last record in this batch, and continue until the batch loading result is empty or the cursor has not advanced; complete the batch loading and parsing of the cursor. The process of performing a full load and parsing of the target table includes: After loading all the data in the target table at once, the pre-save conversion method, the save method, and the post-save processing method are executed sequentially.

[0013] Preferably, The priority function is as follows: when the table belongs to the friend information table, the priority value is 1, otherwise it is 0. The tables are arranged in descending order of priority value so that the friend information table is parsed before the other tables.

[0014] Preferably, it further includes: When traversing each table in the sorted set of table names, determine whether to skip the current table according to the skip determination method defined in the parsing strategy object. If skipped, add the current table to the skipped table set and skip this parsing.

[0015] Preferably, When any target table encounters an error during the initial or secondary parsing process, the table name of that target table is appended to the set of error tables, and the subsequent tables in the sorted set of table names or the large file message queue are processed. After the initial parsing or secondary parsing of a large file is completed, the table names in the error table set are summarized and written to the parsing log.

[0016] Preferably, The parsing strategy object also implements a strategy attribute bag, which is used to write custom key-value attributes during the execution of the parsing strategy object for use in time accumulation and log statistics. The method further includes: when the initial parsing of the target table or the secondary parsing of a large file contains a friend information table, triggering post-processing of friend association tasks and / or statistical tasks, wherein anomalies in the post-processing do not block the initial parsing or secondary parsing of the large file process.

[0017] Preferably, Among the deletion conditions, the option to enable deletion is a configurable option that is disabled by default.

[0018] According to a second aspect of the present invention, an SQLite multi-table parsing apparatus based on a policy registry is provided, the apparatus comprising: Initialization module: Used to download SQLite files from the object storage service to the local working directory; Table name resolution module: used to parse the set of table names to be processed from the SQLite file, and sort each table in the set of table names according to a preset priority function, wherein the priority function is used to indicate the priority of each table in the parsing order; The strategy object parsing module is used to traverse each table in the sorted set of table names and, for the target table currently being traversed, obtain the parsing strategy object of the target table through the strategy registry. Initial parsing module: Used to perform initial parsing based on the parsing strategy object of the target table, including: determining whether the target table is a large file table based on the parsing strategy object of the target table; if so, and the current state is not in large file processing mode, then add the target table to the large file table set and skip this parsing; if not, then determine whether the target table supports batch parsing based on the parsing strategy object of the target table; if it supports batch parsing, then perform cursor batch loading parsing on the target table; if it does not support batch parsing, then perform full loading parsing on the target table. Secondary parsing module: Used to perform secondary parsing of large files after the initial parsing is completed, including: when the large file table set is not empty, adjusting the current state to large file processing mode; sending the large file table set and the initial parsing time to the large file message queue; according to the large file message queue, taking the current large file to be parsed as the target table, and performing cursor batch loading and parsing on the target table; Source file deletion module: When the large file table set is empty and the error table set is empty, it determines whether the source file deletion condition of the SQLite file is met. If the deletion condition is met, the source file of the SQLite file is deleted. The error table set is used to store the table names of the initial parsing and large file secondary parsing errors. The deletion condition is that all three conditions are met simultaneously: no unprocessed large file table, no error table, and deletion is enabled.

[0019] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This application significantly improves the scalability of multi-table parsing by retrieving the parsing strategy object by table name and executing a unified lifecycle hook, thus decoupling the parsing logic of each table from the core scheduling. This allows adding a new table structure to only require adding a new strategy object and registering it. Large file judgment in the lifecycle hook separates large file tables from the initial parsing and transfers them to a large file message queue for asynchronous processing, avoiding memory overflow and timeouts caused by full loading of GB-level large tables. Batch loading is achieved by using ROWID cursors for tables that support batch loading based on the judgment result, deeply coupling cursor paging with parsing batch processing, avoiding performance collapse during deep paging based on offset. Priority functions are used to sort the table set, ensuring that tables with dependencies (such as friend information tables) are parsed first, allowing subsequent tasks to obtain dependent data. An error table set isolates single-table parsing anomalies, preventing single-table failures from blocking the overall parsing process. Deletion conditions—no unprocessed large file tables, no error tables, and deletion enabled by configuration—ensure that source files are not mistakenly deleted, preserving the data source for retrying failures.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0022] Figure 1 This is a flowchart illustrating a policy registry-based SQLite multi-table resolution method according to an exemplary embodiment. Figure 2 This is a system schematic diagram of a policy registry-based SQLite multi-table resolution apparatus according to another exemplary embodiment; In the attached diagram: 1-Initialization module, 2-Table name resolution module, 3-Resolution strategy object acquisition module, 4-Initial resolution module, 5-Secondary resolution module, 6-Source file deletion module. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0024] Example 1 Figure 1 This is a flowchart illustrating a policy registry-based SQLite multi-table resolution method according to an exemplary embodiment, such as... Figure 1 As shown, the method includes: S1, download the SQLite file from the object storage service to the local working directory; S2, parse the set of table names to be processed from the SQLite file, and sort each table in the set of table names according to a preset priority function, wherein the priority function is used to indicate the priority of each table in the parsing order; S3, traverse each table in the sorted set of table names, and for the target table currently being traversed, obtain the parsing policy object of the target table through the policy registry; S4, perform initial parsing according to the parsing strategy object of the target table, including: determining whether the target table is a large file table according to the parsing strategy object of the target table; if so, and the current state is not in large file processing mode, then add the target table to the large file table set and skip this parsing; if not, determine whether the target table supports batch parsing according to the parsing strategy object of the target table; if so, perform cursor batch loading parsing on the target table; if not, perform full loading parsing on the target table. S5, after the initial parsing is completed, a second parsing of the large file is performed, including: when the large file table set is not empty, the current state is adjusted to the large file processing mode; the large file table set and the initial parsing time are sent to the large file message queue; according to the large file message queue, the large file to be parsed is used as the target table, and the cursor is used to perform batch loading and parsing of the target table; S6, when the large file table set is empty and the error table set is empty, determine whether the source file deletion condition of the SQLite file is met. If the deletion condition is met, delete the source file of the SQLite file. The error table set is used to store the table names of the initial parsing and the large file secondary parsing exceptions. The deletion condition is that all three conditions are met simultaneously: no unprocessed large file table, no error table, and the configuration to enable deletion. It is understood that this embodiment specifically includes: Download the SQLite file to be parsed from the Object Storage Service (OSS) to your local working directory; Check if the file exists. If the file exists, proceed with the next steps; otherwise, report an error and terminate the process.

[0025] Set parsing status: Initialize parsing log records and set the parsing status to "parsing" to ensure that the status of any subsequent step is traceable.

[0026] Parse the set of table names: Parse the set of table names T to be processed from the SQLite file. If the current mode is large file secondary parsing (large file processing mode is marked as true), only the subset of large file table names is parsed to avoid repeatedly parsing small tables that have already been completed.

[0027] Sort by priority function: according to the preset priority function. σ (t) Sort the tables in the set of table names T, with the priority function defined as follows: when t belongs to the friend information table... σ (t)=1, otherwise σ (t)=0, sorted in descending order. This step ensures that the friend information table is placed before all other tables, thus guaranteeing that subsequent friend association and statistics tasks can obtain the dependent data.

[0028] Obtain the parsing strategy object: for the target table t currently being traversed. i Retrieve the table name t from the policy registry S. i The corresponding parsing strategy object S(t) i The policy registry uses table names as keys and parsed policy objects as values, providing a lookup interface for policy objects by table name. The parsed policy object implementation has a unified lifecycle hook interface, which includes at least: skip judgment method, large file judgment method, batch support judgment method, full load method, batch load method, pre-save conversion method, save method, and post-save processing method.

[0029] It should be noted that the parsing strategy object also implements a strategy attribute bag, including: custom key-value attributes (such as "actual number of messages in the wrapper object strategy" and "duplicate message deduplication count") that can be written to the context during strategy execution, for use by the end-of-line layer for time accumulation and log statistics; and to solve custom counting needs such as the number of messages saved in the wrapper object strategy not being equal to the actual number of messages.

[0030] Determine whether to skip this table: Call the skip determination method of the current parsing strategy object. If it returns true, add the table to the skip table set T. skip The process continues processing the next table; if the return value is false, it proceeds to the next step. Generally, reasons for skipping a table include scenarios such as configuration being disabled or version incompatibility. To determine if a file is a large file and whether the current processing mode is large file: Call the large file determination method LF(t) of the current parsing strategy object. If it is determined to be a large file and the current large file processing mode flag handleBigFile is false (i.e., not in large file processing mode), then place the table into the large file table collection L and continue processing the next table. Note that the large file determination is completed before the lifecycle hooks are executed to avoid performing a full load on the large file table, which could lead to an OutOfMemoryError (OOM). If it is determined to be a large file and handleBigFile is true (i.e., already in large file processing mode), then perform secondary parsing of the large file. If it is determined not to be a large file, then perform the initial parsing.

[0031] Initial parsing: Determine if batching is supported, and call the batching support determination method of the current parsing strategy object. If batching is supported, execute the ROWID cursor to load and parse in batches; if batching is not supported, use full loading and parsing.

[0032] ROWID cursor batch loading parsing: The following steps are executed iteratively with a preset page size P (e.g., 1000 records): Data is loaded using the batch loading method LB(C,P) based on the current cursor C; pre-save transformation method, save method, and post-save processing method are executed; cursor C is updated to the ROWID of the last record in this batch, i.e., C_next. The termination condition is: LB(C,P) is null or C_next has not been advanced (i.e., C_next=⊥). This step avoids the performance collapse of offset-based pagination during deep page turns. It should be noted that: the pre-save transformation method includes: pre-save transformation, filtering, and deduplication; the save method is the actual storage in the target database such as MongoDB / MySQL / ES / OLAP; the post-save processing method is post-save processing (such as updating indexes, pushing events, and accumulating deduplication counts). Full Load Analysis: The full load method is called to load all data at once, followed by the pre-save conversion method, the save method, and the post-save processing method.

[0033] Determine if the large file table set L is empty: If L is not empty (i.e. there are unprocessed large file tables), then perform secondary parsing of the large files.

[0034] Secondary parsing of large files: Send a message to the large file queue, carrying parameters including the large file table set L, the large file processing mode marked as true (handleBigFile:=true), and the initial parsing time Δ_prev, to trigger asynchronous secondary parsing, and set the parsing status to "partially completed".

[0035] Waiting for the second parsing of large files to complete: The consumer pulls messages from the queue for the second parsing of large files and re-enters the parsing process. At this time, handleBigFile is true, L is set to an empty set, and the large file message queue is parsed in batches using ROWID cursors. It should be noted that when the initial parsing of the target table or the second parsing of large files contains a friend information table, the post-processing of friend association tasks and / or statistics tasks is triggered. However, exceptions in the post-processing do not interrupt the initial parsing or the second parsing of large files. Cumulative time consumption: When entering the secondary parsing of a large file, read the initial parsing time Δ_prev recorded in the parsing log, add the current secondary parsing time Δ_current to Δ_prev, and get the total time consumption Δ_total = Δ_prev + Δ_current, which is written back as the final time consumption.

[0036] Set the status to "All Completed": When L is empty and all tables have been processed, set the parsing status to "All Completed".

[0037] Determine OSS deletion conditions: Call the OSS deletion timing determination module to check if the following three conditions are met simultaneously: ① No unprocessed large file table (i.e., L is empty or handleBigFile is true); ② Error table set E is empty; ③ cfg_delete is configured to enable deletion and is true. If all three conditions are met, delete the original OSS file; if any condition is not met, retain the original OSS file to ensure that there is a source available for download during retry.

[0038] Error handling: When any target table encounters an error during the initial or secondary parsing process, the table name of the target table is appended to the error table set E, and the subsequent tables in the sorted table name set or large file message queue are processed. After the initial parsing or large file secondary parsing is completed, the table names in the error table set E are summarized and written to the parsing log for manual or automatic retries.

[0039] To facilitate understanding of the above-described solution in this embodiment, the following example is also provided: Local library parsing for mobile instant messaging applications: Taking a data platform for an instant messaging application as an example, the mobile client uploads the entire local SQLite database to OSS, and the backend service downloads it from OSS and adopts the technical solution of this embodiment.

[0040] Specific steps: Download and initialization: Download the OSS file to the / wechat / WECHAT_FILES / WECHAT_ORIG_FILES / directory, initialize the parsing log, and set the status to "parsing"; Table Name Parsing and Sorting: Parse to obtain a set of table names T (message table, friend table, avatar table, Moments table, red envelope table, group chat table, application message table, mass message table); sort by \(\ σ (t)\) After sorting, the friend list and group chat list (including friend relationships) are placed at the top; Friend table parsing (priority): When the batch support judgment method of the friend table strategy returns true, enter the batch path of the ROWID cursor; each batch contains 1000 records, and execute "batch loading -> conversion before saving (MD5 deduplication) -> saving (writing to MongoDB + synchronizing ES) -> post-saving processing (pushing friend changes)"; the cursor loops until it is empty; Message table parsing (large file classification): The message table strategy's large file determination method returns true (data volume reaches hundreds of millions), and handleBigFile is false; the message table is collected into the large file table set L, and this parsing is skipped; Other table parsing: Avatar table, Moments table (OLAP database entry), Red Packet table, Application Message table, and Mass Message table are parsed according to their respective strategies; single table exceptions are appended to the error table E without interruption; Finishing: Post-processing of friend information triggers friend association tasks and statistics updates; time accumulated; if the state machine determines that the large file table set L is not empty, then set "partially completed" and large file flag = 1; send ((L, large file processing mode = true, \Delta_{prev})\) to the large file queue; Secondary parsing of large files: The consumer pulls large file messages and re-enters the parsing process, marking the large file processing mode as true; at this time, the large file table set L is empty, and the message table strategy is executed normally (ROWID cursor batching); after completion, the status advances to "all completed", and the total time Δ_total = Δ_prev + Δ_current; OSS deletion criteria: ① No unprocessed large file table (i.e., L is empty or handleBigFile is true); ② Error table set E is empty; ③ cfg_delete is enabled in the configuration and is true; if all three conditions are met, the original OSS file is deleted; otherwise, it is retained.

[0041] Technical effects: In a real-world deployment of a data platform: The overall parsing time for a single GB-level SQLite database has been reduced from the original full load timeout (>30 minutes) to approximately 8 minutes (including secondary parsing of large files); By using ROWID cursors for batching and asynchronous hierarchical processing of message tables (50 million to 100 million records), the OOM (Out of Memory) occurrence rate was reduced from approximately 12% to below 0.1%. The parsing logic for adding a new table has been reduced from "modifying the core scheduling code + full regression" to "adding a new strategy class + registration", reducing the extended working time by approximately 70%. The cumulative time consumption of secondary parsing of large files across multiple sessions makes the accuracy of the "total time consumption" field in operation and maintenance observation reach 100% (previously it would be overwritten and distorted). The OSS deletion timing consistency constraint reduces the "original file has been deleted" type of fault to 0 when retrying due to failure.

[0042] Example 2 Figure 2 This is a system schematic diagram of a policy registry-based SQLite multi-table resolution apparatus according to another exemplary embodiment, the apparatus comprising: Initialization module 1: Used to download SQLite files from the object storage service to the local working directory; Table name resolution module 2: used to parse the set of table names to be processed from the SQLite file, and sort each table in the set of table names according to a preset priority function, wherein the priority function is used to indicate the priority of each table in the parsing order; Module 3 for obtaining parsing strategy objects: It is used to traverse each table in the sorted set of table names, and for the target table currently being traversed, it obtains the parsing strategy object of the target table through the policy registry. Initial parsing module 4: Used to perform initial parsing based on the parsing strategy object of the target table, including: determining whether the target table is a large file table based on the parsing strategy object of the target table; if so, and the current state is not in large file processing mode, then add the target table to the large file table set and skip this parsing; if not, then determine whether the target table supports batch parsing based on the parsing strategy object of the target table; if it supports batch parsing, then perform cursor batch loading parsing on the target table; if it does not support batch parsing, then perform full loading parsing on the target table. Secondary parsing module 5: Used to perform secondary parsing of large files after the initial parsing is completed, including: when the large file table set is not empty, adjusting the current state to large file processing mode; sending the large file table set and the initial parsing time to the large file message queue; according to the large file message queue, taking the current large file to be parsed as the target table, and performing cursor batch loading and parsing on the target table; Source file deletion module 6: When the large file table set is empty and the error table set is empty, it determines whether the source file deletion condition of the SQLite file is met. If the deletion condition is met, the source file of the SQLite file is deleted. The error table set is used to store the table names of the initial parsing and large file secondary parsing errors. The deletion condition is that all three conditions are met simultaneously: no unprocessed large file table, no error table, and deletion is enabled.

[0043] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0044] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0045] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0046] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0047] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0048] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0049] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0050] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0051] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A SQLite multi-table parsing method based on a policy registry, characterized in that, The method includes: Download the SQLite file from the object storage service to your local working directory; The set of table names to be processed is parsed from the SQLite file, and each table in the set of table names is sorted according to a preset priority function, which is used to indicate the priority of each table in the parsing order. Traverse each table in the sorted set of table names, and for the target table currently being traversed, obtain the parsing policy object of the target table through the policy registry; The initial parsing is performed based on the parsing strategy object of the target table, including: determining whether the target table is a large file table based on the parsing strategy object of the target table; if so, and the current state is not in large file processing mode, then the target table is added to the large file table set and the current parsing is skipped; if not, determining whether the target table supports batch parsing based on the parsing strategy object of the target table; if so, performing cursor batch loading parsing on the target table; if not, performing full loading parsing on the target table. After the initial parsing is completed, a second parsing of the large file is performed, including: when the large file table set is not empty, adjusting the current state to large file processing mode; sending the large file table set and the initial parsing time to the large file message queue; and, based on the large file message queue, using the large file to be parsed as the target table, performing cursor-based batch loading and parsing on the target table. When the large file table set is empty and the error table set is empty, it is determined whether the source file deletion condition of the SQLite file is met. If the deletion condition is met, the source file of the SQLite file is deleted. The error table set is used to store the table names of the initial parsing and large file secondary parsing errors. The deletion condition is that all three conditions are met simultaneously: no unprocessed large file table, no error table, and deletion is enabled.

2. The SQLite multi-table parsing method based on a policy registry according to claim 1, characterized in that, Also includes: After the initial parsing is completed, obtain the initial parsing time. After the secondary parsing of the large file is completed, the time taken for the secondary parsing of the large file is obtained; The total time is obtained by adding the initial parsing time to the secondary parsing time.

3. The SQLite multi-table parsing method based on a policy registry according to claim 1, characterized in that, The step of obtaining the parsed policy object of the target table through the policy registry includes: The corresponding parsing policy object is found in the policy registry using the table name of the target table as the key. The policy registry uses the table name as the key and the parsing policy object as the value. The parsing policy object implements a unified lifecycle hook interface, which includes at least the following methods: skip judgment method, large file judgment method, batch support judgment method, full load method, cursor batch load method, pre-save conversion method, save method, and post-save processing method.

4. The SQLite multi-table parsing method based on a policy registry according to claim 3, characterized in that, The step of performing cursor-based batch loading and parsing on the target table includes: The ROWID cursor is used to execute the path in batches, looping through the following steps with a preset page size: Data is loaded in batches according to the current cursor, and the pre-save conversion method, the save method, and the post-save processing method are executed sequentially on the currently loaded batches of data. Update the cursor to the ROWID of the last record in this batch, and continue until the batch loading result is empty or the cursor has not advanced; complete the batch loading and parsing of the cursor. The process of performing a full load and parsing of the target table includes: After loading all the data in the target table at once, the pre-save conversion method, the save method, and the post-save processing method are executed sequentially.

5. The SQLite multi-table parsing method based on a policy registry according to claim 1, characterized in that, The priority function is as follows: when the table belongs to the friend information table, the priority value is 1, otherwise it is 0. The tables are arranged in descending order of priority value so that the friend information table is parsed before the other tables.

6. The SQLite multi-table parsing method based on a policy registry according to claim 3, characterized in that, Also includes: When traversing each table in the sorted set of table names, determine whether to skip the current table according to the skip determination method defined in the parsing strategy object. If skipped, add the current table to the skipped table set and skip this parsing.

7. The SQLite multi-table parsing method based on a policy registry according to claim 3, characterized in that, When any target table encounters an error during the initial or secondary parsing process, the table name of that target table is appended to the set of error tables, and the subsequent tables in the sorted set of table names or the large file message queue are processed. After the initial parsing or secondary parsing of a large file is completed, the table names in the error table set are summarized and written to the parsing log.

8. The SQLite multi-table parsing method based on a policy registry according to claim 3, characterized in that, The parsing strategy object also implements a strategy attribute bag, which is used to write custom key-value attributes during the execution of the parsing strategy object for use in time accumulation and log statistics. The method further includes: when the initial parsing of the target table or the secondary parsing of a large file contains a friend information table, triggering post-processing of friend association tasks and / or statistical tasks, wherein anomalies in the post-processing do not block the initial parsing or secondary parsing of the large file process.

9. The SQLite multi-table parsing method based on a policy registry according to claim 1, characterized in that, Among the deletion conditions, the option to enable deletion is a configurable option that is disabled by default.

10. A SQLite multi-table parsing device based on a policy registry, characterized in that, The device includes: Initialization module: Used to download SQLite files from the object storage service to the local working directory; Table name resolution module: used to parse the set of table names to be processed from the SQLite file, and sort each table in the set of table names according to a preset priority function, wherein the priority function is used to indicate the priority of each table in the parsing order; The strategy object parsing module is used to traverse each table in the sorted set of table names and, for the target table currently being traversed, obtain the parsing strategy object of the target table through the strategy registry. Initial parsing module: Used to perform initial parsing based on the parsing strategy object of the target table, including: determining whether the target table is a large file table based on the parsing strategy object of the target table; if so, and the current state is not in large file processing mode, then add the target table to the large file table set and skip this parsing; if not, then determine whether the target table supports batch parsing based on the parsing strategy object of the target table; if it supports batch parsing, then perform cursor batch loading parsing on the target table; if it does not support batch parsing, then perform full loading parsing on the target table. Secondary parsing module: Used to perform secondary parsing of large files after the initial parsing is completed, including: when the large file table set is not empty, adjusting the current state to large file processing mode; sending the large file table set and the initial parsing time to the large file message queue; according to the large file message queue, taking the current large file to be parsed as the target table, and performing cursor batch loading and parsing on the target table; Source file deletion module: When the large file table set is empty and the error table set is empty, it determines whether the source file deletion condition of the SQLite file is met. If the deletion condition is met, the source file of the SQLite file is deleted. The error table set is used to store the table names of the initial parsing and large file secondary parsing errors. The deletion condition is that all three conditions are met simultaneously: no unprocessed large file table, no error table, and deletion is enabled.

Images