Log collection and intelligent retrieval method based on columnar data warehouse supporting inverted index

Abstract

The application discloses a log collection and intelligent retrieval method based on a columnar data warehouse supporting an inverted index, and belongs to the technical field of log data management and analysis. The method simultaneously improves storage utilization, retrieval performance and operation and maintenance efficiency in a massive log scenario by means of configured table building and retrieval strategies, centralized management and control and health detection of collection instances, unified query and visualization processes at the front end, and the introduction of intelligent agents to assist in generating query conditions. The method solves the problems of high storage occupation, insufficient complex retrieval performance, complex collection end operation and maintenance, high use threshold and fragmented multi-data source / multi-module management in the prior art, thereby simultaneously improving storage utilization, retrieval performance and operation and maintenance efficiency, and reducing the use threshold.

Description

Technical Field

[0001] This invention relates to the field of log data management and analysis technology, specifically to a log collection and intelligent retrieval method based on a columnar data warehouse that supports inverted indexes. It is an integrated log collection, retrieval, and visualization system and method based on a columnar data warehouse that supports inverted indexes, combined with intelligent agents to generate query conditions. Background Technology

[0002] With the widespread adoption of microservices and containerization, the scale of logs generated by enterprise applications continues to rise, with daily increments reaching billions. Existing solutions, such as ELK (Elasticsearch-Logstash-Kibana), expose the following problems in massive scenarios:

[0003] (1) High storage consumption: In order to meet the retrieval needs, it is often necessary to build an inverted index for all fields and maintain a copy. The column storage compression is insufficient and the long-term retention cost is high.

[0004] (2) Insufficient performance of complex retrieval: Under the combined query of large time range and multiple conditions (structured fields + keywords), the scanning volume and merging overhead increase significantly, the user experience is unstable, and it takes a long time to locate the problem.

[0005] (3) Complex operation and maintenance of the data collection end: Logstash instances are distributed on multiple machines. Starting, stopping, scaling and configuration changes rely on manual or script-based methods. There is a lack of unified remote control and health detection, resulting in low efficiency in fault detection and tracing.

[0006] (4) The threshold for use is relatively high: queries often require understanding the underlying table structure and retrieval syntax; the analytical capabilities such as field distribution and time histogram are scattered in many places, making it impossible to form a consistent process from input conditions to result visualization.

[0007] (5) Fragmented management of multiple data sources / multi-modules: The lack of unified modular management and access control for log tables and collection tasks in different systems or environments leads to difficulties in cross-team collaboration and auditing. Summary of the Invention

[0008] To address the problems of existing technologies, this invention provides a log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes. Through configurable table creation and retrieval strategies, centralized management and health monitoring of collection instances, a unified front-end query and visualization process, and the introduction of intelligent agents to assist in generating query conditions, it simultaneously improves storage utilization, retrieval performance, and operational efficiency in massive log scenarios. This solves problems in existing technologies such as high storage consumption, insufficient performance in complex retrieval, complex collection-end operation and maintenance, high usage threshold, and fragmented management of multiple data sources / modules, thereby simultaneously improving storage utilization, retrieval performance, and operational efficiency while lowering the usage threshold.

[0009] Glossary

[0010] 1. SQL: refers to Structured Query Language, a standardized language used to perform data definition, data query, and data update operations in relational or relational databases.

[0011] 2. WHERE: Refers to the conditional clause in an SQL statement, used to apply Boolean filtering conditions to records when querying a table or view, returning only records that satisfy the given conditional expression.

[0012] 3. SELECT: Refers to a query clause or query statement in an SQL statement, used to specify the set of fields to be read from one or more data tables, and can be combined with clauses such as WHERE to form a complete query.

[0013] 4. Columnar data warehouse: refers to a data warehouse system that adopts a column-based physical organization method and is oriented towards large-scale analytical queries. In this specification, it is limited to a columnar database that supports data definition and querying using SQL syntax.

[0014] 5. MATCH_ANY: Refers to an arbitrary word matching retrieval method based on an inverted index. If at least one of the multiple search terms derived from the query string appears in the target record, the record is considered to meet the query criteria. (For ease of description, this manual uses the illustrative designations "MATCH_ANY", "MATCH_ALL", and "MATCH_PHRASE" for the three inverted index-based retrieval methods. These English names are for technical purposes only and do not limit the specific database product, query syntax, or keyword names used in their implementation.)

[0015] 6. MATCH_ALL: This refers to a search method based on the inverted index, where all terms obtained from the decomposition of the query string appear in the same target record. Only when all terms appear in the same target record will the record be considered to meet the query conditions.

[0016] 7. MATCH_PHRASE: This refers to a phrase matching retrieval method based on inverted index and position information. That is, when the ordered sequence of search terms corresponding to the query string appears consecutively in the target record in a predetermined order or sequentially within a preset word distance range, the record is determined to meet the query conditions.

[0017] 8. LIKE: Refers to a retrieval method based on string pattern matching. It uses preset wildcards (such as "%" and "_") in the query pattern to perform fuzzy matching on the text content of the target field to determine whether the record meets the query conditions.

[0018] 9. Logstash: Refers to an open-source data pipeline component used to collect, parse, transform and forward log or event data. It usually acts as a "collector" to read logs from multiple input sources, and outputs them to search engines, message queues or storage systems after filtering and formatting.

[0019] 10. Logstash Process: In this system, for the sake of consistent terminology, the Logstash process is used to refer to the Logstash log collection and processing task itself in an abstract sense. It represents a type of logical processing unit defined within the system, rather than specifically referring to the actual Logstash process running in a particular machine's operating system.

[0020] 11. Logstash process instance: In this system, Logstash process instance refers to the actual running entity deployed on a specific machine that belongs to a certain Logstash process (logical task), that is, the logstash process at the operating system level. This process has a unique process identifier (PID) in the corresponding operating system.

[0021] 12. JVM: Refers to the Java Virtual Machine, a virtualized execution engine used to load and execute Java bytecode and provide a unified runtime environment.

[0022] 13. SSH: Secure Shell is a remote login and command execution protocol based on an encrypted channel. It is used to securely connect to servers, transfer files, and perform administrative operations in insecure network environments through key or password authentication.

[0023] 14. PID: A unique process identifier (ProcessID) assigned to each running process in the operating system. It is used to locate, monitor, and control resources of processes in the kernel and various process management tools.

[0024] To achieve the above objectives, the present invention provides the following technical solution:

[0025] The log collection and intelligent retrieval method based on a columnar data warehouse that supports inverted indexes includes the following steps:

[0026] Step 1: In the module management phase, establish the binding relationship between the module and the data source and save the module query configuration. The module query configuration includes a time field, a retrieval strategy for keyword queries, and a list of excluded fields.

[0027] Step 2: Perform log collection by remotely distributing and starting the collection instance, and register the mapping relationship between the collection instance and the machine, module, and target log table;

[0028] Step 3: Receive the retrieval request containing the time range, structured WHERE clause, and keyword expression, and determine the target log table for query routing based on the selected module and the binding relationship between the module and the data source; perform collaborative parsing of the keyword expression and WHERE clause, and generate matching clauses equivalent to these module query configurations according to the time field configured in the module query configuration, the retrieval strategy used for keyword query, and the exclusion field list;

[0029] Step 4: Construct the final query statement during the SQL generation stage. This involves removing the excluded fields when building the SELECT column set, and merging the time-filtered, collaboratively parsed matching clause with the WHERE clause to form an equivalent query. The equivalent query is then executed in the columnar data warehouse, and the query results are returned.

[0030] Step 5: During the query construction phase, when agent collaboration is enabled, only module query configuration, module and target log table metadata, and module query configuration metadata are provided to the agent. The agent calls the system's internal tool to actively push data to the front end via SSE push streaming to generate action data to trigger the front end to query the backend interface. The agent does not directly access log detail data.

[0031] Furthermore, the keyword expression is composed of Boolean operators represented by ||, &&, and -. During parsing, the expression is split into several clauses. For each clause, the corresponding predicate generation method is selected according to the configured keyword query retrieval strategy: when MATCH_ANY is specified, the keyword fields MATCH_ANY word1, word2, ... are generated; when MATCH_ALL is specified, the keyword fields MATCH_ALL word1, word2, ... are generated; when MATCH_PHRASE is specified, the keyword field MATCH_PHRASE word string is generated; and when LIKE is specified, the keyword field LIKE pattern is generated. The predicates of each clause are connected and incorporated into the WHERE clause using the SQL Boolean operators OR, AND, and NOT according to the combination relationship of ||, &&, and - to form an equivalent query.

[0032] Furthermore, during the SQL generation phase, a difference operation is performed between the field set of the target log table and the list of excluded fields in the module query configuration. The field set obtained by the difference is used as the SELECT column set, thereby completing the removal of the excluded fields during the query construction phase.

[0033] Furthermore, before the collaborative parsing, the retrieval request is standardized and security verified. The security verification includes parameter whitelisting and type verification, length and range restrictions, and special character escaping and dangerous statement interception to prevent SQL injection. In terms of syntax compatibility, it supports parsing and unifying variant or nested fields according to the equivalent addressing mode of ab and a['b']. And according to the query configuration strategy in the module query configuration, the keyword expression is translated into a matching clause consistent with the query configuration strategy and incorporated into the WHERE clause, thereby generating an equivalent query for execution.

[0034] Furthermore, the query routing is based on the N:1 binding relationship between the module and the data source, and the target data source and target log table are determined according to the module query configuration; when the binding is missing, the configuration is incomplete, or the connection is unavailable, a controlled error is returned and SQL generation and execution are not initiated; and role-based access permission verification is performed before routing, allowing only requests with the corresponding module query permissions to continue processing.

[0035] Furthermore, the remote distribution and startup of the collection instance are executed via SSH commands, including: distributing the Logstash package and the collection configuration corresponding to the target module to the installation directory of the target machine and recording the installation directory and instance identifier; establishing the mapping relationship between the module and the Logstash process, the Logstash process and the Logstash process instance, and the Logstash instance and the machine; registering the process identifier (PID) and startup result status after the front end triggers startup; and registering the above information in the instance status ledger for subsequent health detection and operation and maintenance auditing based on the PID.

[0036] Furthermore, the collection instance is managed by a state machine, which includes the states INITIALIZING, NOT_STARTED (initialization completed but not started), STARTING (starting), RUNNING (running), START_FAILED (startup failed), INITIALIZE_FAILED (initialization failed), STOPPING (stopping), and STOP_FAILED (stopping failed). The startup path is NOT_STARTED→STARTING→RUNNING and the PID is registered. Startup failure is marked as START_FAILED, and initialization failure is marked as INITIALIZE_FAILED. Stopping is part of the complete state machine but is not shown in the distribution and startup sequence. Its process is RUNNING→STOPPING. After normal completion, it returns to NOT_STARTED, and stopping failure is marked as STOP_FAILED. The system only performs periodic liveness detection on instances in the RUNNING state based on the PID. When the system detects that the process does not exist or is unresponsive, it captures the most recently running or startup log segment and dispatches an alarm, while updating the instance state to avoid duplicate alarms.

[0037] Furthermore, the intelligent agent collaboratively obtains metadata for query construction through the system's tool interface. This metadata includes module identifiers, module query configurations, field and type information of the target log table, and query parameter constraints. Based on this, the intelligent agent generates a standardized query parameter set that conforms to the front-end action data specifications. This standardized query parameter set includes module identifiers, time ranges, WHERE clauses, keyword expressions, and pagination parameters. The front-end then calls the back-end interface based on the action data to trigger the equivalent query. Throughout the entire process, the intelligent agent does not directly access the log detail results.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] More efficient storage and retrieval: During the table creation phase, an inverted index is built for the keyword field and combined with columnar execution, reducing the scan volume and latency of complex condition retrieval.

[0040] Integrated and easy to use: Module → table creation → query configuration → query and visualization are completed in the same process. The main interface links histograms with details and field distribution, making it easy to use.

[0041] Operations are manageable and observable: unified management of machine access, process distribution, instance start / stop and ledger; periodic detection by PID, automatic log capture and email alerts for anomalies.

[0042] Clear and auditable routing: Module-data source N:1 binding, clear query construction and distribution chain, facilitating governance and traceability.

[0043] Good compatibility and scalability: It supports consuming external log platforms (Kafka by default) and storing them in the database. Instances can be scaled horizontally, and the underlying columnar data warehouse can be replaced. Attached Figure Description

[0044] Figure 1 The diagram shown is a schematic diagram of the overall system structure of the present invention, which is described as "log collection—database entry—retrieval and intelligent agent collaboration".

[0045] Figure 2 The diagram shown is an overview of the system's functional flow (module management / table configuration—log query—Logstash instance state machine).

[0046] Figure 3 The diagram shown is a hierarchy and mapping relationship diagram of the "module-data source-log table-Logstash process / instance-machine" of this invention.

[0047] Figure 4 The image shown is a schematic diagram of the "Module Query Configuration Interface" of this invention;

[0048] Figure 5 The diagram shown is a flowchart of the "Log Retrieval Condition Parsing and SQL Generation Process (Keyword and WHERE Collaboration)" of this invention.

[0049] Figure 6 The diagram shown is a timing diagram of the Logstash process distribution and startup control according to the present invention.

[0050] Figure 7 The diagram shown is a timing diagram of "Intelligent Agent Triggering Equivalence Queries" in this invention. Detailed Implementation

[0051] The present invention will now be further described with reference to the accompanying drawings.

[0052] The log collection and intelligent retrieval method based on a columnar data warehouse that supports inverted indexes includes the following steps:

[0053] S1 module and data source, table creation and query configuration (through the system's visual interface)

[0054] S11 Module Creation: Create a new module in "Module Management", fill in the module name, bind the data source, and specify the target table name.

[0055] S12 Table Creation Execution: Select the newly created module, click "Create Table," and paste / enter the table creation statement required for the business logic. The system then sends this statement to the underlying columnar data warehouse (such as Doris) to actually create the table. When creating the table, you can configure inverted indexes, tokenizers, and phrase matching switches on the keyword field, and specify the time base field.

[0056] S13 Module Query Configuration: In "Module Query Configuration", set:

[0057] (1) Time field (used for time range filtering and histogram bucketing);

[0058] (2) Exclude display fields (redundant columns that are not displayed by default on the main interface);

[0059] (3) Keyword search fields and methods (select LIKE / MATCH_ANY / MATCH_ALL / MATCH_PHRASE by field, in coordination with the index configuration when creating the table).

[0060] S2 Data Acquisition Instance Orchestration and Remote Control (Machine → Process → Instance → Start / Stop)

[0061] S21 Machine Management and Access: In the "Machine Management" interface, add the target machine that will be used to run Logstash, and enter the SSH username and password / key pair; the system attempts to perform connectivity verification via SSH and registers it as a candidate node.

[0062] S22 Create a Logstash process: On the "Logstash Management" page, create a new Logstash process, selecting one of the previously connected machines (multiple machines are allowed); and specify:

[0063] (1) Data collection configuration (such as parameters for Kafka input, parsing / filtering, and output to the data warehouse)

[0064] (2) JVM configuration (optional)

[0065] (3) System custom configuration (optional)

[0066] After submission, the system will distribute the Logstash package and configuration to a specific directory on the selected machine (the path can be specified, or it will be randomly generated if not specified), generate a process instance for each machine, and establish a mapping relationship between "module and Logstash process, Logstash process and Logstash process instance, Logstash instance and machine".

[0067] S23 Instance Initialization: After the instance is created, its status is changed to INITIALIZING → NOT_STARTED to enter the pending startup state; at this time, the installation path and instance identifier are recorded (PID has not yet been generated).

[0068] S24 Start / Stop Control (Front-end Button Trigger): Click Start in the instance list, the status will cycle through NOT_STARTED → STARTING → RUNNING, and the PID will be registered; if stopped, RUNNING → STOPPING → START_FAILED will indicate a startup failure and retain the startup error information.

[0069] S25 Consume external logs from the platform and store them in the database: The instance consumes logs from the external log storage platform (Kafka by default) according to the collection configuration, and writes them to the target log table in the bound data source after parsing / cleaning (e.g., through batch import / StreamLoad, etc.) to form an end-to-end data stream.

[0070] S26 Configuration Change and Distribution (Optional): When the collection rules change, remote configuration updates are supported and rolling restarts can be selected. Change history and operator information are retained for auditing.

[0071] S27 Operational Status Log: The system continuously maintains an instance log (machine, installation directory, instance status, PID, version / configuration fingerprint, most recent operator and time) for retrieval and operation and maintenance auditing.

[0072] S3 Survival Detection and Anomaly Alarm (Polling by PID)

[0073] S31 Probe Plan Generation: The system creates a probe plan for each Logstash instance (triggered by default every 2 minutes). The plan items include: instance identifier, machine to which it belongs, installation directory, last probe time, and last status.

[0074] S32 Anomaly Evidence Collection: In the anomaly branch, the system retrieves the recent running logs and the most recent startup logs from the instance installation directory, and records a snapshot of machine resources.

[0075] S33 Alarm Dispatch: Packages instance identifier, machine information, timestamp, and key log fragments into an email alarm and sends it to the preset alarm recipients for that module or instance.

[0076] S4 Query Triggering and Routing (Module → Data Source N:1)

[0077] S41 Condition Input: The main query interface receives the time range, module identifier, structured WHERE condition and / or keyword condition (keyword condition takes effect according to the LIKE / MATCH_ANY / MATCH_ALL / MATCH_PHRASE selected for each field in the module query configuration).

[0078] S42 Module Resolution and Routing: Based on the many-to-one binding relationship between modules and data sources, the target data source and log table are resolved; if a user does not have permission to access the module, access is denied and a message is displayed.

[0079] S43 Query Construction: The system assembles time filtering, WHERE conditions, excluded fields, and keyword conditions into an equivalent query (the keyword clause works in conjunction with the inverted index, tokenizer, and phrase matching switch during table creation).

[0080] S44 Request delivery and timeout control: Execute queries to the bound data source; set query timeout and concurrency limit (configurable on the system side).

[0081] S45 Result Reception and Display: While receiving the result set, the column mapping and type are verified, and the data details and data time distribution histogram are rendered on the front end.

[0082] S5 Agent-Assisted Query

[0083] S51 Context Assembly: When a user enables an agent, the system provides the agent with a summary of module configuration (time fields, exclusion fields, keyword fields and their retrieval methods, table structure highlights) and the user's current intent (natural language).

[0084] S52 Condition Generation: The agent interprets natural language into equivalent front-end query parameters (time range, WHERE clause, keyword clause, etc.) based on context, without directly bypassing the front-end or going through the back-end "privileged channel".

[0085] S53 Triggering and Echoing: The S4 process is called through the query triggering interface exposed on the front end; the result is echoed on the main query interface, and the user can continue to add natural language follow-up questions, and the agent incrementally generates new conditions.

[0086] Example

[0087] I. System Overall Structure and Process (corresponding to) Figure 1 , Figure 2 )

[0088] S1 Overall Structure

[0089] S11 This embodiment adopts an end-to-end link of "collection → (message middleware) → columnar storage / query → front-end retrieval and display". Log data is received by the collection component, buffered and smoothed by the message middleware, and then written to the columnar storage and query engine. Users initiate retrieval and obtain results through the front-end interface (see...). Figure 1 , Figure 2 The system supports distributed deployment and remote orchestration of the data acquisition process.

[0090] S2 Technology Stack and Middleware Selection

[0091] S21 Backend: The server is built based on Java and Spring Boot 3.x, providing REST interfaces for retrieval, configuration, and data collection orchestration; the project is built with Maven.

[0092] S22 Frontend: A search and management interface based on React (v19) and TypeScript, implementing a frontend and backend separation.

[0093] S23 Storage and Query: The columnar storage and query engine uses Apache Doris as its core engine to achieve millisecond-level retrieval and analysis of massive log data.

[0094] S24 Acquisition: Logstash is used as the log acquisition and forwarding component. The system remotely orchestrates and manages the Logstash process via SSH commands.

[0095] S25 Message Middleware: In this embodiment, Apache Kafka is selected as the message queue for buffering and decoupling the log stream. A typical approach is "Logstash ↔ Kafka ↔ Downstream Storage / Query". (Note: Kafka is the chosen implementation for this example and can be replaced with a messaging system with equivalent capabilities; Doris is the preferred implementation and can also be used.)

[0096] S3 Deployment Topology and Data Flow (Overview)

[0097] S31 Collection Side: Each business host or edge node runs a Logstash instance, which pushes logs to a specified topic in Kafka by module / application to achieve high-throughput buffering and peak shaving.

[0098] S32 Storage / Query Side: Backend services are configured by module and execute queries from Doris; Doris, as a columnar MPP engine, carries structured log details and aggregate queries.

[0099] S33 Management / Interaction Side: The front-end (React + TS) provides a search and operation interface, and queries, configurations and data collection orchestration are performed through the API exposed by the back-end (SpringBoot); the system supports distributed deployment and visualized Logstash process management.

[0100] S4 Key Implementation Constraints and Substitutability

[0101] S41 Doris and Logstash can be integrated through the official ecosystem (such as using Logstash output to connect to Doris's Stream Load / HTTP capabilities), or other equivalent access links can be used; Kafka is the preferred message middleware, but it can also be replaced according to the scenario.

[0102] II. Module Management and Table Configuration (corresponding) Figure 2 , Figure 3 , Figure 4 )

[0103] S1 module establishment

[0104] S11 users create new modules in the "Module Management" interface. When creating a module, a module name must be entered, and a unique data source must be bound. The system automatically establishes a many-to-one correspondence between modules and data sources (see [link]). Figure 3 ).

[0105] S2 table creation execution

[0106] After the module is established, users can perform table creation operations. The system supports directly pasting or entering the table creation statement required for business operations in the interface, and sends the statement to the columnar storage database (preferably Doris) for execution to complete the actual creation of the log table.

[0107] During the table creation phase, S22 allows configuring inverted indexes, tokenizers, and phrase matching switches for key fields, while also specifying a time base field for time range filtering during subsequent retrieval (see [link]). Figure 2 ).

[0108] S3 Query Configuration

[0109] After S31 is created, users can access the "Module Query Configuration" interface (see [link]). Figure 4 ), set the query parameters associated with this module.

[0110] The S32 configuration includes: a time field (used to retrieve time ranges and histogram binning), an exclusion display field (used to hide redundant columns), and a keyword search field and its search methods (supporting LIKE, MATCH_ANY, MATCH_ALL, MATCH_PHRASE, etc.).

[0111] After saving the configuration, the S33 system establishes a correspondence between it and the indexing strategy used when creating the table, ensuring that subsequent queries can be automatically parsed and generate the optimal execution statement.

[0112] III. Log Collection and Instance Control (corresponding) Figure 2 , Figure 3 , Figure 6 )

[0113] S1 machine access

[0114] In the "Machine Management" interface of S11, maintenance personnel can enter information about the target machine, including IP address, SSH username and authentication method (password or key pair).

[0115] After data entry, the S12 system performs connectivity verification via SSH. Machines that pass the verification are registered as candidate nodes and can serve as the runtime environment for Logstash instances (see [link]). Figure 3 ).

[0116] S2 process creation and distribution

[0117] S21 users can create a new process in the "Logstash Management" interface and select the previously connected target machine, which can be one or more machines.

[0118] When creating a process, users need to specify the collection configuration (such as Kafka input, parsing / filtering rules, and parameters for output to Doris), and can choose to configure JVM parameters or system-defined parameters.

[0119] After S23 is submitted, the system automatically distributes the Logstash package and configuration files to the installation directory of the specified machine. If no installation directory is specified, the system will generate a random path.

[0120] S24 Generates a process instance for each machine and records the mapping relationship of "module / process / machine / instance" (see [link]). Figure 3 ).

[0121] S3 instance initialization and state transition

[0122] The newly created Logstash instance on S31 has its installation package distributed to various machine nodes via SSH. After distribution, its initial state is INITIALIZING. When the initialization process is successfully completed, the state changes to NOT_STARTED, waiting for user startup.

[0123] After an S32 user clicks the "Start" button on the front end, the instance status transitions to STARTING. If the startup completes successfully, it enters the RUNNING state and registers the corresponding PID. If an error occurs during startup, it is marked as START_FAILED, and the exception information is recorded.

[0124] S33 If an error occurs during the initialization phase of the instance, it is recorded as INITIALIZE_FAILED (initialization failed), and can be repaired by re-initializing the process.

[0125] S34 Stop operation not available Figure 6 Within the scope of the demonstration, but in the complete state machine of the system (see...) Figure 2 When the user clicks "Stop", the instance status transitions to STOPPING (stopping); if it completes normally, it returns to NOT_STARTED; if the stop fails, it is marked as STOP_FAILED (stopping failed), and the exception information is recorded.

[0126] S4 Log Collection and Storage

[0127] The running Logstash instance on S41 consumes log data from an external log platform (default Kafka) according to the preset collection configuration.

[0128] After the logs consumed by S42 are parsed, cleaned, and formatted, they are written to the data source target table of the bound module (preferably Doris), completing the log ingestion process (see [link]). Figure 2 ).

[0129] S5 Configuration Changes and Remote Control

[0130] When the collection rules are adjusted, the S51 system supports remote update of instance configuration and can choose to restart in a rolling manner to reduce business interruption.

[0131] S52 records all operations with change history and operator information for subsequent auditing and tracking.

[0132] IV. Operation Monitoring and Anomaly Alarms (corresponding to) Figure 2 )

[0133] S1 Scheduled Probe: A system-defined scheduled task that runs every two minutes by default to check the running status of Logstash instances. The scheduled task only scans instances currently in the RUNNING state and determines whether the process is alive based on its PID.

[0134] S2 anomaly detection and log collection: When a process is found to be missing or has exited, the system determines that the instance has crashed and immediately performs the following operations.

[0135] S21 invokes the log command, collecting only the most recent 50 lines of runtime logs for this instance.

[0136] S22 records the identifier of the abnormal instance, its module, the machine it is on, and the collected log content to form an abnormal snapshot.

[0137] S3 alarm dispatch: The system will summarize all downtime instances detected in this scan and send them via email to the designated alarm handler for each instance. The email content includes: a list of downtime instances, the time of the anomaly, machine information, and the corresponding log fragments.

[0138] S4 status update and anti-duplicate mechanism: To avoid duplicate alarms, after sending an alarm, the system updates the status of a failed instance from RUNNING to NOT_STARTED. This ensures that these instances will not trigger alarms again in the next round of scheduled detection until the instance is manually or automatically restarted.

[0139] V. Query Construction and Route Execution (corresponding) Figure 2 , Figure 5 )

[0140] S1 Query Request Input

[0141] S11 Users select the target module and enter the time range in the front-end query interface.

[0142] S12 users can simultaneously enter structured WHERE conditions (such as field filtering) and unstructured keyword conditions. Both types of conditions can be used individually or in combination.

[0143] S2 Query Condition Analysis

[0144] The S21 system parses out the target data source and the target log table based on the many-to-one binding relationship between the module and the data source.

[0145] The S22 system retrieves the definition of the time field from the module query configuration and constructs a range filtering SQL for the time field based on the user-input time range.

[0146] S3 Keyword Expression Parsing

[0147] The S31 system performs syntax parsing on the input keyword conditions and supports three operators: || (OR), && (AND), and - (NOT), with a maximum of two levels of nesting.

[0148] The S32 parsing result will be converted into an SQL clause for inverted index retrieval. For example, when the user enters "test-data && - error" and the target field "message" is configured with the MATCH_PHRASE retrieval method, the system will generate the SQL: ((message MATCH_PHRASE 'test-data') AND NOT (message MATCH_PHRASE 'error')).

[0149] S4 WHERE conditional concatenation

[0150] The S41 system converts user-input structured WHERE conditions into Boolean clauses.

[0151] The final SQL query in S42 is composed of exclusion fields, time range filtering clauses, keyword search clauses, and WHERE condition clauses, forming a complete query statement.

[0152] S5 Query Execution and Result Display

[0153] The S51 system routes the generated SQL to the bound columnar storage database for execution and sets query timeout and concurrency limits.

[0154] After the S52 query is executed, the results are directly returned to the front-end interface for display.

[0155] VI. Agent-Assisted Query (corresponding) Figure 7 )

[0156] S1 User Request Input

[0157] S11 users input their query requirements in natural language on the front end.

[0158] When the S12 system invokes an AI, it only provides the AI ​​with role settings and a list of available tools, without including any log details.

[0159] S2 Tool Invocation and Information Retrieval

[0160] Based on user input and a tool list, S21 AI autonomously calls upon tools within the system to obtain the necessary module metadata.

[0161] The metadata that the S22 tool can return includes: module query configuration (time field, keyword field and search method, excluded fields, target data source and target table), underlying table structure, table creation statement, etc.

[0162] S3 Query Parameter Generation

[0163] S31 AI generates standardized query parameters based on user input and metadata obtained by S2.

[0164] The S32 query parameters include a time range, keyword expressions, and WHERE conditions. Keyword expressions support three operators: ||, &&, and -, with a maximum of two levels of nesting, and are mapped to corresponding SQL clauses.

[0165] S4 Action Data Trigger

[0166] S41 AI calls the action triggering tool within the system, packages the standardized query parameters generated by S3 into action data, and sends it to the front end.

[0167] After receiving the action data, the S42 frontend requests the backend to query the execution log. The execution process is the same as in Example 5.

[0168] S5 Results Display and Interaction

[0169] The S51 query results are returned by the backend and rendered by the frontend to display log details.

[0170] Simultaneously, the AI ​​in the S52 dialog box notifies the user that a query has been triggered and provides a summary of the query criteria. The user can then continue with natural language input to trigger the next query.

[0171] This application proposes an integrated log collection, retrieval, and intelligent query method and system based on a columnar data warehouse supporting inverted indexes. Log tables are created for modules through visual configuration, setting time bases and exclusion fields, and inverted indexes are built for keyword fields during table creation. Logstash instances are distributed to multiple machines in an N:M ratio and controlled via SSH, recording PIDs and probing for liveness every 2 minutes. Instance logs are captured and email alerts are sent upon failure. The query client is bound one-to-one between modules and data sources, receiving WHERE clauses and keyword conditions for cascading / parallel execution, returning details, and generating histograms by time buckets. An AI assistant reads module configurations and table structures, triggering front-end equivalent queries. SQL editing, result visualization, and role-based access control are also provided. This solution reduces storage and maintenance costs and improves the efficiency and ease of use for massive log retrieval.

[0172] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes, characterized in that, Includes the following steps: Step 1: In the module management phase, establish the binding relationship between the module and the data source and save the module query configuration. The module query configuration includes a time field, a retrieval strategy for keyword queries, and a list of excluded fields. Step 2: Perform log collection by remotely distributing and starting the collection instance, and register the mapping relationship between the collection instance and the machine, module, and target log table; Step 3: Receive the retrieval request containing the time range, structured WHERE clause, and keyword expression, and determine the target log table for query routing based on the selected module and the binding relationship between the module and the data source; perform collaborative parsing of the keyword expression and WHERE clause, and generate matching clauses equivalent to these module query configurations according to the time field configured in the module query configuration, the retrieval strategy used for keyword query, and the exclusion field list; Step 4: Construct the final query statement during the SQL generation stage. This involves removing the excluded fields when building the SELECT column set, and merging the time-filtered, collaboratively parsed matching clause with the WHERE clause to form an equivalent query. The equivalent query is then executed in the columnar data warehouse, and the query results are returned. Step 5: During the query construction phase, when agent collaboration is enabled, only module query configuration, module and target log table metadata, and module query configuration metadata are provided to the agent. The agent calls the system's internal tool to actively push data to the front end via SSE push streaming to generate action data to trigger the front end to query the backend interface. The agent does not directly access log detail data.

2. The log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes as described in claim 1, characterized in that: The keyword expression is composed of Boolean operators represented by ||, &&, and -. During parsing, the expression is split into several clauses. For each clause, the corresponding predicate generation method is selected according to the configured keyword query retrieval strategy: when MATCH_ANY is specified, the keyword fields MATCH_ANY_word1, word2, ... are generated; when MATCH_ALL is specified, the keyword fields MATCH_ALL_word1, word2, ... are generated; when MATCH_PHRASE is specified, the keyword field MATCH_PHRASE_wordstring is generated; and when LIKE is specified, the keyword field LIKE_pattern is generated. The predicates of each clause are then connected and incorporated into the WHERE clause using the SQL Boolean operators OR, AND, and NOT according to the combination relationship of ||, &&, and - to form an equivalent query.

3. The log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes as described in claim 1, characterized in that: During the SQL generation phase, a difference operation is performed between the field set of the target log table and the exclusion field list in the module query configuration. The field set obtained by the difference is used as the SELECT column set, thereby completing the removal of the exclusion fields during the query construction phase.

4. The log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes as described in claim 1, characterized in that: Before the collaborative parsing, the retrieval request is standardized and security verified. The security verification includes parameter whitelist and type verification, length and range restrictions, and special character escaping and dangerous statement interception to prevent SQL injection. In terms of syntax compatibility, it supports parsing and unifying variant or nested fields according to the equivalent addressing mode of ab and a['b']. The keyword expression is translated into a matching clause consistent with the query configuration strategy in the module query configuration and incorporated into the WHERE clause, thereby generating an equivalent query for execution.

5. The log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes as described in claim 1, characterized in that: The query routing is based on the N:1 binding relationship between the module and the data source, and the target data source and target log table are determined according to the module query configuration; When binding is missing, configuration is incomplete, or connection is unavailable, a controlled error is returned without proceeding to SQL generation and execution; and role-based access permission verification is performed before routing, allowing only requests with the corresponding module query permissions to continue processing.

6. The log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes as described in claim 1, characterized in that: Remote distribution and startup of the data collection instance are executed via SSH commands, including: distributing the Logstash package and the data collection configuration corresponding to the target module to the installation directory of the target machine and recording the installation directory and instance identifier; establishing mapping relationships between modules and Logstash processes, Logstash processes and Logstash process instances, and Logstash instances and machines; registering the process identifier (PID) and startup result status after startup is triggered by the front end; and registering the above information in the instance status ledger for subsequent health detection and operation and maintenance auditing based on the PID.

7. The log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes as described in claim 6, characterized in that: The data acquisition instance is managed by a state machine, which includes the states INITIALIZING, NOT_STARTED, STARTING, RUNNING, START_FAILED, INITIALIZE_FAILED, STOPPING, and STOP_FAILED. The startup path is NOT_STARTED→STARTING→RUNNING and the PID is registered. Startup failure is marked as START_FAILED, and initialization failure is marked as INITIALIZE_FAILED. Stopping is part of the complete state machine but is not shown in the dispatch and startup sequence. Its process is RUNNING→STOPPING. After normal completion, it returns to NOT_STARTED. If the stop fails, it is marked as STOP_FAILED. The system only performs periodic liveness detection on instances in the RUNNING state based on the PID. When the system detects that the process does not exist or is unresponsive, it captures the most recently run or started log segment and dispatches an alarm. At the same time, it updates the instance state to avoid duplicate alarms.

8. The log collection and intelligent retrieval method based on a columnar data warehouse supporting inverted indexes as described in claim 1, characterized in that: The intelligent agent collaboratively obtains metadata for query construction through the system's tool interface. This metadata includes module identifiers, module query configurations, field and type information of the target log table, and query parameter constraints. Based on this, the intelligent agent generates a standardized query parameter set that conforms to the front-end action data specifications. This standardized query parameter set includes module identifiers, time ranges, WHERE clauses, keyword expressions, and pagination parameters. The front-end then calls the back-end interface based on the action data to trigger the equivalent query. Throughout the entire process, the intelligent agent does not directly access the log detail results.

Images