Database log processing method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a database log processing method and device, a storage medium and electronic device.The method comprises the steps that based on the total number of logs within a specified log range requested to be queried by a current log query request and the number of query threads in a plurality of query threads, the query threads are queried according to the total number of the logs within the specified log range; segmenting a log requested to be queried by the log query request into a plurality of log ranges, and respectively constructing corresponding query statements to obtain a plurality of query statements; executing the corresponding query statements in parallel through a plurality of query threads, and querying log data in a corresponding log range from a log database; receiving at least one batch of log data returned by the log database through the data receiving thread, and writing the log data into the buffer block queue in a circular writing mode; and executing processing operation on the log data written in the buffer block in a circular processing mode through a data processing thread, and summarizing and sorting the processed log data to obtain a log query result corresponding to the log query request.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of database technology, and in particular to a database log processing method and device, a storage medium, and an electronic device. Background Art

[0002] With the advent of the big data era, the scale of enterprise data continues to expand, and the amount of data is increasing dramatically. Archived log information in some databases is stored in large object type fields. Large object type fields have large capacity and can store very large amounts of data. However, due to the large amount of data in large object type fields, each query may require reading a large amount of data from disk. In a network environment, the transmission of this large volume of data significantly increases query time and reduces query efficiency, resulting in reduced efficiency of remote queries. Therefore, the database log processing methods in the related art suffer from low query processing efficiency. Summary of the Invention

[0003] The embodiments of the present application provide a database log processing method and apparatus, a storage medium, and an electronic device to at least solve the technical problem of low query processing efficiency in database log processing methods in related technologies.

[0004] According to one aspect of an embodiment of the present application, a database log processing method is provided, comprising: based on the total number of log entries within a specified log range requested by a current log query request and the number of query threads in multiple query threads, dividing the log requested by the log query request into multiple log ranges, and constructing a corresponding query statement for each log range in the multiple log ranges to obtain multiple query statements, wherein one query statement in the multiple query statements corresponds to one query thread in the multiple query threads; executing the corresponding query statements in parallel by the query threads in the multiple query threads to query a log database for log data within the log range corresponding to each query statement in the multiple query statements; receiving at least one batch of log data returned by the log database in response to each query statement by a data receiving thread, and writing the received log data to a buffer block queue by the data receiving thread in a cyclic writing manner, wherein the buffer block queue is a queue formed by dividing a buffer allocated for the log query request into multiple buffer blocks; performing processing operations on the log data written to the buffer blocks in the buffer block queue by the data processing thread in a cyclic processing manner, and aggregating and sorting the processed log data to obtain a log query result corresponding to the log query request.

[0005] According to another aspect of an embodiment of the present application, a database log processing device is also provided, including: a first execution unit, for dividing the log requested by the log query request into multiple log ranges based on the total number of logs in the specified log range requested by the current log query request and the number of query threads in multiple query threads, and constructing corresponding query statements for each log range in the multiple log ranges to obtain multiple query statements, wherein one query statement in the multiple query statements corresponds to one query thread in the multiple query threads; a query unit, for executing corresponding query statements in parallel through query threads in the multiple query threads to query the multiple query threads from a log database. The log data within the log range corresponding to each query statement in the statement; a second execution unit, used to receive at least one batch of log data returned by the log database in response to each query statement through a data receiving thread, and write the received log data into a buffer block queue through the data receiving thread in a cyclic writing manner, wherein the buffer block queue is a queue formed by a plurality of buffer blocks divided from a buffer area allocated for the log query request; a third execution unit, used to perform processing operations on the log data written into the buffer block in the buffer block queue in a cyclic processing manner through a data processing thread, and summarize and sort the processed log data to obtain a log query result corresponding to the log query request.

[0006] In an exemplary embodiment, the device also includes: a sending unit for sending an aggregate query statement to the log database based on the specified log range before dividing the log requested to be queried by the log query request into multiple log ranges based on the total number of logs within the specified log range requested to be queried by the current log query request and the number of query threads in multiple query threads, wherein the aggregate query statement is used to query the total number of logs within the specified log range; and a determining unit for determining the number of logs indicated by the aggregate query result as the total number of logs within the specified log range when receiving the aggregate query result returned by the log database in response to the aggregate query statement.

[0007] In an exemplary embodiment, one of the multiple log ranges is assigned to one of the multiple query threads; the first execution unit includes: a query module, used to query the maximum column length of the logs within the current log range through the current query thread, wherein the current query thread is any query thread among the multiple query threads, and the current log range is the log range assigned to the current query thread; a construction module, used to construct a corresponding query statement for the current log range through the current query thread when the maximum column length of the logs within the current log range is greater than or equal to a specified length threshold; a conversion module, used to convert each large object column of the at least one large object column into a corresponding non-large object column when the maximum column length of the logs within the current log range is less than the specified length threshold and the logs within the current log range include at least one large object column, and construct a corresponding query statement for the current log range based on the converted non-large object column.

[0008] In an exemplary embodiment, the second execution unit includes: a writing module for writing the current batch of log data into the first buffer block of the buffer block queue according to the first block pointer of the buffer block queue through the data receiving thread when the current batch of log data is received through the data receiving thread, wherein the first block pointer is used to indicate the free block in the buffer block queue to which the log data is to be written; wherein, when the writing of the current batch of log data is completed, the first block pointer is moved back to the second buffer block, when the first buffer block is not the last buffer block of the buffer block queue, the second buffer block is the next buffer block of the first buffer block, and when the first buffer block is the last buffer block of the buffer block queue, the second buffer block is the first buffer block of the buffer block queue.

[0009] In an exemplary embodiment, when the current batch of log data is written to the first buffer block of the buffer block queue, the block state of the first buffer block is modified from an idle state to a pending state; the third execution unit includes: a first execution module, which is used to perform the processing operation on the current batch of log data written in the first buffer block according to the second block pointer through the data processing thread when the second block pointer of the buffer block queue points to the first buffer block and the block state of the first buffer block is a pending state, wherein the second block pointer is used to indicate the buffer block in the buffer block queue on which the processing operation is to be performed on the written log data, and when the processing operation is completed on the current batch of log data written in the first buffer block, the block state of the first buffer block is modified from the pending state to the idle state.

[0010] In an exemplary embodiment, each query thread among the multiple query threads corresponds to a log data linked list; the device also includes: a saving unit, which is used to save the processed current batch of log data in the form of a newly added linked list node to the log data linked list corresponding to a specified query thread among the multiple query threads after the data processing thread performs the processing operation on the log data in the first buffer block according to the second block pointer, wherein the current batch of log data is log data returned in response to a specified query statement among the multiple query statements, and the specified query thread is the query thread corresponding to the specified query statement; wherein, the summarizing and sorting of the processed log data is performed on the log data in the log data linked list corresponding to each query thread.

[0011] In an exemplary embodiment, each query thread in the multiple query threads corresponds to a log data linked list, and after the processing operation is performed on the log data returned in response to the query statement sent by each query thread, the processed log data is saved in the form of a newly added linked list node to the log data linked list corresponding to each query thread; the third execution unit includes: a second execution module, which is used to perform multiple rounds of linked list merging operations through multiple execution threads until the log data linked lists corresponding to the multiple query threads have been merged, wherein, when executing the linked list merging operation of the current round, the log data linked lists to be merged are The table comprises at least two current log data linked lists; the at least two current log data linked lists are divided into at least one current log data linked list pair with two current log data linked lists as a group, and one current log data linked list pair in the at least one current log data linked list pair is assigned to a current execution thread among the multiple execution threads; the linked list nodes of the two current log data linked lists in the current log data linked list pair assigned to the current execution thread are synchronously traversed by the current execution thread, and the traversed linked list nodes are merged into one log data linked list based on the attribute values of the specified attribute column in a descending order or a descending order.

[0012] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to perform the steps of any of the above method embodiments when executed by a processor.

[0013] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the above-described method embodiments.

[0014] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the steps of any of the above method embodiments through the computer program.

[0015] According to the present application, based on the total number of logs within the specified log range requested to be queried by the current log query request and the number of query threads in multiple query threads, the log requested to be queried by the log query request is divided into multiple log ranges, and a corresponding query statement is constructed for each log range in the multiple log ranges to obtain multiple query statements, wherein one query statement in the multiple query statements corresponds to one query thread in the multiple query threads; the corresponding query statements are executed in parallel by the query threads in the multiple query threads to query the log data within the log range corresponding to each query statement in the multiple query statements from the log database; at least one batch of log data returned by the log database in response to each query statement is received by the data receiving thread, and the received log data is written to a buffer block queue in a cyclic writing manner by the data receiving thread, wherein the buffer block queue is a queue formed by dividing a buffer allocated for the log query request into multiple buffer blocks; the log data written into the buffer block in the buffer block queue is processed by the data processing thread in a cyclic processing manner, and the processed log data is summarized and sorted to obtain a log query result corresponding to the log query request. Since the log query task is assigned to multiple query threads for parallel execution, the time spent on the query process can be reduced. In addition, the data receiving thread and the data processing thread asynchronously receive and process the queried log data, which can efficiently complete a series of processes of query, reception and processing. Therefore, it can solve the technical problem of low query processing efficiency in the database log processing method in the related art, and achieve the effect of improving the query and processing efficiency of the log. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of an application scenario of a database log processing method according to an embodiment of the present application.

[0017] Figure 2 It is a flowchart of an optional database log processing method according to an embodiment of the present application.

[0018] Figure 3 This is a schematic diagram of an optional database log processing method according to an embodiment of the present application.

[0019] Figure 4 It is a flowchart of another optional database log processing method according to an embodiment of the present application.

[0020] Figure 5 This is a flowchart of another optional database log processing method according to an embodiment of the present application.

[0021] Figure 6 This is a structural block diagram of an optional database log processing device according to an embodiment of the present application.

[0022] Figure 7 This is a block diagram of a computer system structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to one aspect of the embodiment of the present application, a database log processing method is provided. Optionally, in this embodiment, the above database log processing method can be applied to, but is not limited to, Figure 1The hardware environment shown includes a terminal device 102 and a server 104. The server 104 can be connected to the terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) for the terminal device 102 or a client installed on the terminal device 102. A database can be set on the server 104 or independently of the server 104 to provide data storage services for the server 104.

[0026] The aforementioned network may include, but is not limited to, at least one of the following: a wired network and a wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, or a local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: wireless fidelity (Wi-Fi) and Bluetooth. The terminal device 102 may be, but is not limited to, a personal computer (PC), a mobile phone, a tablet computer, etc. The server 104 may be, but is not limited to, a cloud server, a server cluster, or other server types.

[0027] The database log processing method of the embodiment of the present application can be executed by the server 104, or by the terminal device 102, or by both the server 104 and the terminal device 102. The database log processing method of the embodiment of the present application can also be executed by the client installed on the terminal device 102.

[0028] Taking the database log processing method of this embodiment executed by the terminal device 102 as an example, Figure 2 FIG. 1 is a flow chart of an optional database log processing method according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include the following steps S202 to S208.

[0029] Step S202, based on the total number of logs in the specified log range requested by the current log query request and the number of query threads in multiple query threads, divide the log requested by the log query request into multiple log ranges, and construct corresponding query statements for each log range in the multiple log ranges to obtain multiple query statements, wherein one query statement in the multiple query statements corresponds to one query thread in the multiple query threads.

[0030] Step S204 : executing corresponding query statements in parallel by query threads in the multiple query threads to query the log database for log data within a log range corresponding to each query statement in the multiple query statements.

[0031] Step S206, receiving at least one batch of log data returned by the log database in response to each query statement through the data receiving thread, and writing the received log data into the buffer block queue in a circular writing manner through the data receiving thread, wherein the buffer block queue is a queue formed by multiple buffer blocks divided by the buffer area allocated for the log query request.

[0032] Step S208 : The data processing thread performs processing operations on the log data written into the buffer blocks in the buffer block queue in a cyclic processing manner, and summarizes and sorts the processed log data to obtain a log query result corresponding to the log query request.

[0033] The database log processing method in this embodiment can be applied to the field of database technology, and applied to the scenario of remotely querying logs in a database.

[0034] With the advent of the big data era, enterprise data scale continues to expand, and the amount of data is increasing dramatically. Some databases, such as the AS500 database, store archived log information in the form of large objects (LOBs). LOBs are a data type used in database systems to store large data such as text, images, and videos. They have large capacity and can store very large amounts of data, supporting up to 128TB of storage capacity.

[0035] In related technologies, a single-threaded, full-query mode is usually used to query database logs. However, when querying log data in LOB fields, due to the huge amount of data in LOB fields, the query needs to read a large amount of data from the disk. In the case of remote queries, the log data needs to be transmitted to the local computer via the network, which will consume a lot of time and resources during the network transmission process, thereby significantly increasing the query time and reducing the query efficiency.

[0036] In order to at least partially solve the above technical problems, in this embodiment, multiple query threads are used to query logs in parallel. Each query thread queries a part of the logs, and then the data receiving thread and the data processing thread asynchronously receive and process the log data returned by the query threads, which can improve query efficiency.

[0037] A log query request does not necessarily query all logs in the entire database. Instead, it may query a specified range of logs, such as a portion of logs within a specified time period or a portion of logs carrying specified parameters. In this embodiment, the total number of log entries within the specified log range requested by the current log query request is first determined based on the current log query request, and the number of query threads among the multiple query threads to be used to query the logs is determined. The number of query threads among the multiple query threads can be set based on current computer resources (such as processor performance, processor utilization, memory utilization, etc.) or the total number of log entries within the specified log range, but this is not determined in this embodiment.

[0038] After determining the total number of log entries within a specified log range and the number of query threads, the logs requested by the log query request are segmented into multiple log ranges, and a corresponding query statement is constructed for each of the multiple log ranges. The number of log ranges (and, therefore, the number of query statements) matches the number of query threads; that is, each segmented log range corresponds one-to-one to a query thread. It should be noted that each query statement only queries the logs within the corresponding log range, thus preventing unnecessary data reading and network transmission.

[0039] Optionally, the logs requested by the above log query request can be divided evenly. For example, if the log query request needs to query all access logs within a specific time period, including 1 million logs, if the system is configured with 10 query threads, the 1 million logs can be evenly divided into 10 log ranges, each log range containing 100,000 logs, and then a corresponding query statement is constructed for each log range, and the query statement is assigned to each query thread.

[0040] After the query statement is constructed, multiple query threads execute the corresponding query statement in parallel to retrieve the log data from the log database within the log range corresponding to each of the multiple query statements. Since the log data still needs to be received and processed after being retrieved from the log database, this parallel query approach can start receiving and processing partial data as soon as it is retrieved. Compared to querying all logs within a specified log range at once, this can reduce the waiting time of the threads involved in receiving and processing logs, thereby improving query efficiency.

[0041] After the query thread executes the query statement, the log database will return the corresponding log data, and the returned log data needs to be received. In this embodiment, as long as there is at least one batch of returned log data, the data receiving thread can receive the log data returned by the log database and write the received log data into the buffer block queue in a circular write manner. Here, the buffer block queue is a queue formed by dividing the buffer zone allocated for the log query request into multiple buffer blocks, which is used to temporarily store received and pending data. The buffer block queue is set to a circular structure, that is, the next buffer block of the buffer block at the end of the queue is the buffer block at the head of the queue. Optionally, for multiple batches of returned log data, the data receiving thread can receive them into different buffer blocks in sequence according to the batches of log data. For example, for log data batch 1, batch 2 and batch 3, the data receiving thread receives them into different buffer blocks respectively. Even if the buffer block is not fully used after the reception of one of the batches is completed, the next batch of log data is received in a new buffer block; the queue can also be received according to the principle of filling one buffer block first, that is, for different batches of log data, the next buffer block is used to receive log data only when the current buffer block is filled.

[0042] After the data receiving thread finishes writing to a buffer block, it automatically switches to the next buffer block to wait or continue writing. At this time, the buffer block that has completed writing requires the data processing thread to process the log data written into it. Here, the data receiving thread and the data processing thread adopt an asynchronous execution method. That is, as long as there is a buffer block that has completed writing, the data processing thread will process the log data in the buffer block without waiting for the data receiving thread to complete all logs within the specified log range before processing. This asynchronous processing method can start the data processing process earlier, which can reduce the waiting time of the data processing thread, thereby improving the efficiency of log processing.

[0043] In this embodiment, the data processing thread processes the log data written to the buffer blocks in the buffer block queue in a loop. Loop processing means that after the data processing thread finishes processing a buffer block, it can switch to the next buffer block in the queue to process the data until all buffer blocks have been processed. Optionally, after the data processing thread completes processing all log data in a buffer block, it can set the buffer block to free, allowing the data receiving thread to write data to the buffer block again.

[0044] Since the time sequence of different batches of log data returned by the log database is not necessarily the same as the order of different batches of log data in the log database, and the data receiving thread and the data processing thread receive and process the log data in the time sequence returned, the processed log data needs to be summarized and sorted. Here, the sorting method can be set according to experience. For example, each query statement can be numbered when constructing the corresponding query statement, and the number can be retained when receiving and processing the corresponding log data, and sorted according to the above number; it can also be sorted according to specific parameters in the log, such as the timestamp order of the log or the log number, etc., which is not limited in this embodiment. After the processed log data is summarized and sorted, the log query result corresponding to the log query request can be obtained.

[0045] For example, Figure 3 As shown, after the query starts, the logs to be queried are first divided based on the query threads and assigned to several query threads (query thread 1 to query thread N). After the query threads complete the query, the data receiving threads complete the writing of the data, and the data processing threads complete the processing of the data, the query results are summarized and sorted to obtain the log query results corresponding to the log query request.

[0046] Alternatively, in this embodiment, a log query request can be passed to the main control thread by another thread. The main control thread then completes the log data query based on the incoming log query request and the number of available query threads. After executing a log query request, if the exit condition is met, the main control thread will not wait for other threads to pass in new log query requests. If the exit condition is not met, the main control thread will wait for other threads to pass in new log query requests.

[0047] For example, Figure 4 As shown, the main control thread can wait for query information (corresponding to log query requests) passed in by other threads through its thread entry. In response to the incoming query information, the main control thread performs the query operation, retrieves the corresponding log data from the database, and sends the query results to other threads. If the exit condition is met, the thread exits and ends. If the exit condition is not met, it continues to wait for other threads to pass in query information.

[0048] Through the embodiment provided by the present application, based on the total number of logs in the specified log range requested by the current log query request and the number of query threads in multiple query threads, the log requested by the log query request is divided into multiple log ranges, and a corresponding query statement is constructed for each log range in the multiple log ranges to obtain multiple query statements, wherein one query statement in the multiple query statements corresponds to one query thread in the multiple query threads; the corresponding query statements are executed in parallel by the query threads in the multiple query threads to query the log data in the log range corresponding to each query statement in the multiple query statements from the log database; the data is received by the data receiving thread The log database responds to at least one batch of log data returned by each query statement, and writes the received log data into a buffer block queue in a circular writing manner through a data receiving thread, wherein the buffer block queue is a queue formed by a plurality of buffer blocks divided by a buffer zone allocated for the log query request; a data processing thread performs processing operations on the log data written into the buffer blocks in the buffer block queue in a circular processing manner, and summarizes and sorts the processed log data to obtain a log query result corresponding to the log query request, thereby solving the problem of low query processing efficiency existing in the database log processing method in the related art and improving the query and processing efficiency of the database log.

[0049] In an exemplary embodiment, before dividing the log requested by the log query request into multiple log ranges based on the total number of logs within the specified log range requested by the current log query request and the number of query threads in multiple query threads, the above method also includes: sending an aggregate query statement to the log database based on the specified log range, wherein the aggregate query statement is used to query the total number of logs within the specified log range; in the case where the received log database returns an aggregate query result in response to the aggregate query statement, the number of logs indicated by the aggregate query result is determined as the total number of logs within the specified log range.

[0050] To determine the total number of log entries within a specified log range and facilitate log segmentation within that range, an aggregate query statement is sent to the log database to query the total number of log entries within the specified log range. An aggregate query is an operation method that uses simplified Structured Query Language (SQL) aggregate functions to perform statistical analysis on a dataset. Its core approach is to use functions such as COUNT(), SUM(), AVG(), MAX(), and MIN() to perform batch calculations on numerical data. In this embodiment, the COUNT() function can be used to query the total number of log entries within a specified log range.

[0051] Upon receiving an aggregate query result from the log database in response to the aggregate query statement, the number of log entries indicated by the aggregate query result is determined as the total number of log entries within the specified log range. Optionally, if no aggregate query result is received within a specified period of time, the query may be deemed to have failed, and the aggregate query statement may be resent to the log database.

[0052] Through this embodiment, the total number of logs within the specified log range is determined through an aggregate query statement. Since the aggregate query statement only focuses on the number of logs and does not involve reading the log content, it can be executed quickly and the results can be returned, thereby reducing the query time and reducing the impact of network delay on query efficiency.

[0053] In an exemplary embodiment, a log range of multiple log ranges is assigned to a query thread among multiple query threads; a corresponding query statement is constructed for each log range in the multiple log ranges to obtain multiple query statements, including: querying the maximum column length of the logs in the current log range through the current query thread, wherein the current query thread is any query thread among the multiple query threads, and the current log range is the log range assigned to the current query thread; when the maximum column length of the logs in the current log range is greater than or equal to a specified length threshold, constructing a corresponding query statement for the current log range through the current query thread; when the maximum column length of the logs in the current log range is less than the specified length threshold and the logs in the current log range include at least one large object column, each large object column in the at least one large object column is converted into a corresponding non-large object column, and constructing a corresponding query statement for the current log range based on the converted non-large object column.

[0054] Because different SQL statements can exhibit significant performance differences when querying result sets, network performance must be considered when querying database logs remotely. Special SQL statement processing can be performed to improve query performance. For example, when querying large amounts of data on columns containing LOB fields, each result returned requires a network round-trip with the database, resulting in a significant amount of query time being wasted on network communications. Therefore, before performing log queries, specific optimizations can be performed on the SQL statement construction based on actual conditions to at least partially address this issue.

[0055] In this embodiment, a query statement can be constructed based on the maximum column length of the logs within the log range corresponding to each query thread. If the maximum column length of the logs within the current log range is greater than or equal to a specified length threshold, it can be determined that the current log range contains logs that need to be saved as large object columns. In this case, the corresponding query statement is constructed normally using the large object field query method.

[0056] When the maximum column length of the logs within the current log range is less than the specified length threshold and the logs within the current log range include at least one large object column, it can be determined that all logs within the current log range do not need to be saved in the form of large object columns, and there are unnecessary large object columns within the current log range. In this case, the large object columns can be converted into corresponding non-large object columns, and then the corresponding query statements can be constructed for the current log range based on the converted non-large object columns, thereby reducing the number of communications with the database in the query and saving query time.

[0057] It should be noted that large object columns can include Character Large Object (CLOB), Binary Large Object (BLOB), National Language Large Object (NLOB), and Document Large Object (DLOB). CLOBs can be used to store large amounts of text data, such as articles and reports; BLOBs can be used to store binary data, such as images, audio, and video; NLOBs can be used to store text data supporting multiple languages; and DLOBs are LOB types specifically designed for document storage. In this embodiment, converting each large object column to a corresponding non-large object column means converting the large object column into a non-large object column suitable for storing the specific type of data stored in the large object column. For example, a BLOB can be converted into a VARCHAR (variable character) type suitable for storing character strings.

[0058] Through this embodiment, by converting unnecessary large object columns into non-large object columns, network transmission of a large amount of data can be avoided, query time can be effectively reduced, and query efficiency can be improved.

[0059] In an exemplary embodiment, the received log data is written into the buffer block queue in a circular writing manner through a data receiving thread, including: when the current batch of log data is received through the data receiving thread, the current batch of log data is written into the first buffer block of the buffer block queue according to the first block pointer of the buffer block queue through the data receiving thread, wherein the first block pointer is used to indicate the free block in the buffer block queue to which the log data is to be written; wherein, when the writing of the current batch of log data is completed, the first block pointer is moved back to the second buffer block, when the first buffer block is not the last buffer block in the buffer block queue, the second buffer block is the next buffer block of the first buffer block, and when the first buffer block is the last buffer block in the buffer block queue, the second buffer block is the first buffer block in the buffer block queue.

[0060] In order to ensure that the data receiving thread can correctly write log data, a first block pointer can be set in the buffer block queue. This pointer is used to indicate the free block in the buffer block queue where the log data is to be written, that is, the first buffer block. When the current batch of log data is received by the data receiving thread, the data receiving thread will write the current batch of log data to the buffer block pointed to by the above-mentioned first block pointer, and after completing the writing of the current batch of log data, the first block pointer will point to the next buffer block in the buffer block queue. In particular, since the buffer block queue is a circular queue, when the first buffer block is the buffer block at the end of the queue, the next buffer block of the first buffer block is the first buffer block at the head of the queue.

[0061] Before the data receiving thread writes log data to the buffer block queue, the buffer block queue can be initialized by pre-allocating a fixed-size buffer and evenly splitting the buffer into multiple buffer blocks of equal size. These blocks are then abstracted into a circular queue to form the buffer block queue. The number of buffer blocks can be set based on the number of query threads or the buffer size, and this is not limited in this embodiment.

[0062] After obtaining the buffer block queue, the offset of the first block pointer can be set according to the size of a buffer block. The above offset is consistent with the size of a buffer block to ensure that the pointer can correctly point to the next buffer block after one offset. The buffer block initially pointed to by the first block pointer can be the buffer block at the head of the queue or other buffer blocks, which is not limited in this embodiment.

[0063] Through this embodiment, the data receiving thread is instructed to write log data to the first buffer block through the first block pointer, which can ensure that the data receiving thread can correctly write log data into several buffer blocks. The cyclic writing mechanism can also ensure the continuous operation of the data receiving thread and improve the efficiency of data reception and writing.

[0064] In an exemplary embodiment, when the current batch of log data is written to the first buffer block of the buffer block queue, the block state of the first buffer block is modified from the idle state to the pending state; a processing operation is performed on the log data written in the buffer block in the buffer block queue in a loop processing manner through a data processing thread, including: when the second block pointer of the buffer block queue points to the first buffer block and the block state of the first buffer block is the pending state, the data processing thread performs a processing operation on the current batch of log data written in the first buffer block according to the second block pointer, wherein the second block pointer is used to indicate the buffer block in the buffer block queue on which the processing operation is to be performed on the written log data, and when the processing operation is completed on the current batch of log data written in the first buffer block, the block state of the first buffer block is modified from the pending state to the idle state.

[0065] In this embodiment, the buffer block has two states: idle and pending. In the idle state, the data processing thread can write data to the buffer block. In the pending state, the data processing thread can perform operations on the log data in the buffer block. Setting these two states of the buffer block ensures that the data receiving thread and the data processing thread can correctly handle their respective tasks.

[0066] The data processing thread can locate the buffer block that needs to be processed through the second block pointer. It should be noted that the initialization setting of the above-mentioned second block pointer is consistent with the first block pointer, that is, the offset setting and the initial pointing position of the second block pointer are consistent with the first block pointer. For the first buffer block, it is initially in an idle state. After the data receiving thread completes writing data to the first buffer block, the first buffer block is modified to a pending state, and the first block pointer points to the next buffer block. At this time, if the other buffer blocks in the buffer block queue are all in an idle state, that is, the data processing thread has completed the data processing of all other buffer blocks or the other buffer blocks have not yet written data, the second block pointer points to the first buffer block, and the data processing thread performs processing operations on the log data in the first buffer block. After the processing is completed, the second block pointer also points to the next buffer block, and the buffer block where the data processing is completed is modified to an idle state, allowing the data receiving thread to write data to it.

[0067] It should be noted that as long as the first buffer block state is idle, the data receiving thread will write the received log data into the buffer block; as long as the first buffer block state is pending, the data processing thread will process the data in the buffer block.

[0068] Through this embodiment, the data processing thread cyclically processes the data in the buffer block queue, and the buffer block switches between the idle state and the pending state, so that data can be received and processed in real time, shortening the waiting time for data processing and avoiding idle resources.

[0069] In an exemplary embodiment, each query thread among multiple query threads corresponds to a log data linked list; after the data processing thread performs a processing operation on the log data in the first buffer block according to the second block pointer, the above method also includes: saving the processed current batch of log data in the form of a newly added linked list node to the log data linked list corresponding to a specified query thread among the multiple query threads, wherein the current batch of log data responds to the log data returned by a specified query statement among the multiple query statements, and the specified query thread is the query thread corresponding to the specified query statement; wherein summarizing and sorting the processed log data is performed on the log data in the log data linked list corresponding to each query thread.

[0070] To facilitate the aggregation and sorting of processed log data, the processed current batch of log data is saved as a newly added linked list node in the log data linked list corresponding to a specific query thread among multiple query threads. The log data linked list can be used to store and manage logs within the log range queried by its corresponding query thread. There is a one-to-one correspondence between the log data linked list and the query thread, and the linked list nodes of the log data linked list correspond one-to-one to the logs: that is, one linked list node corresponds to one log record. A linked list node contains a complete log record and a link to the next node.

[0071] In this embodiment, the log data is sequentially added to the log data linked list in the form of newly added linked list nodes, which can ensure that the log data can maintain correct data in the log data, facilitating subsequent aggregation and sorting operations.

[0072] Through this embodiment, a data linked list structure is used to save processed log data, providing an independent data storage space for each query thread, which is helpful for data management and tracking; saving new log data in the log data linked list in the form of a newly added linked list node can ensure that the order of log data in a log data linked list will not be disordered, which can reduce the time consumed in subsequent log sorting and improve sorting efficiency.

[0073] In an exemplary embodiment, each query thread in a plurality of query threads corresponds to a log data linked list. After processing the log data returned in response to the query statement sent by each query thread, the processed log data is saved in the form of a newly added linked list node to the log data linked list corresponding to each query thread; the processed log data is aggregated and sorted to obtain a log query result corresponding to the log query request, including: executing the following multiple rounds of linked list merging operations through multiple execution threads until the log data linked lists corresponding to the multiple query threads have been merged, wherein, when executing the linked list merging operation of the current round, The merged log data linked list is at least two current log data linked lists; the at least two current log data linked lists are divided into at least one current log data linked list pair with two current log data linked lists as a group, and one current log data linked list pair of the at least one current log data linked list pair is assigned to a current execution thread in multiple execution threads; the linked list nodes of the two current log data linked lists in the current log data linked list pair assigned to the current execution thread are synchronously traversed by the current execution thread, and the traversed linked list nodes are merged into one log data linked list based on the attribute values of the specified attribute column in a descending order or a descending order.

[0074] Because the logs are segmented and queried, the resulting logs are distributed across multiple log data linked lists, requiring sorting and merging. To improve the efficiency of aggregating and sorting log data, in this embodiment, a multi-threaded merge sorting approach is employed to merge log chains. Each execution thread sorts and merges a set of log data linked lists. After multiple rounds of linked list merging operations, the multiple log data linked lists can be gradually merged into a single globally sorted linked list. In each round of linked list merging operations, at least two log data linked lists are grouped together for pairing, and each pair of linked lists is assigned to an execution thread.

[0075] It's important to note that the merging and sorting of log data linked lists is performed in parallel by multiple execution threads over multiple rounds. After each round, multiple, merged and sorted, larger linked lists of log data are obtained. These linked lists are then regrouped and assigned to execution threads for a new round of merging and sorting, until a globally sorted linked list is retained.

[0076] During the merging and sorting process, the execution thread will traverse all linked list nodes in the log data linked list and merge them into a new log data linked list according to the attribute value of the specified attribute column of each linked list node. Here, the specified attribute column can be the timestamp of the log, or the log number in the log database, or other attributes, which are not limited in this embodiment.

[0077] Through this embodiment, by executing the linked list merging operation through multiple threads, the time required for sorting can be reduced, the sorting efficiency can be improved, and thus the overall log processing efficiency can be improved.

[0078] The following is an explanation of the database log processing method in the embodiment of the present application with reference to an optional example. In this optional example, the entire process from query to processing of one query thread among multiple query threads is shown.

[0079] Figure 5 This is a flowchart of the database log processing method in this optional example. Figure 5 As shown, the process of the database log processing method may include the following steps:

[0080] Step S502: Single query starts.

[0081] Step S504: The query thread queries the maximum column length of the logs within its corresponding log range.

[0082] Step S506 , determining whether the maximum column length is less than a specified length threshold. If so, executing step S508 ; if not, executing step S510 .

[0083] Step S508 : converting the large object column into a non-large object column, and constructing a corresponding special query statement for the current log range based on the converted non-large object column.

[0084] Step S510: Construct a normal query statement normally.

[0085] Step S512: Send the query statement to the database and start the data acquisition program simultaneously.

[0086] In step S514, the database returns the query result to the data receiving thread. After obtaining the data, the data receiving thread writes the data into the buffer block and continues to obtain the next batch of data.

[0087] Step S516: The data receiving thread and the data processing thread exchange data, and the data processing thread processes the data in the buffer block.

[0088] Step S518: The single query ends.

[0089] Optionally, you can use the Open Database Connectivity (ODBC) database interaction method to receive and process data. ODBC is a database access technology that allows queries to access different database management systems through a unified standard interface, and supports setting pointers and searching circular queues to facilitate asynchronous processing of data receiving threads and data processing threads.

[0090] This optional example, by constructing efficient query statements, can significantly reduce the number of interactions between the query program and the database, thereby significantly reducing the impact of network latency on log queries and significantly improving log query efficiency. Using a method based on ODBC column binding offsets and a virtual circular queue for asynchronous data processing can minimize the interval between database interactions, improve query speed, and enhance data transmission efficiency.

[0091] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as a read-only memory (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0093] According to another aspect of the embodiments of the present application, a database log processing device is also provided. The database log processing device can be used to implement the database log processing method provided in the above embodiments. The details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0094] Figure 6 This is a structural block diagram of an optional database log processing device according to an embodiment of the present application, such as Figure 6 As shown in , the database log processing device includes a first execution unit 602 , a query unit 604 , a second execution unit 606 and a third execution unit 608 .

[0095] A first execution unit 602 is configured to divide the log requested by the log query request into multiple log ranges based on the total number of logs within the specified log range requested by the current log query request and the number of query threads in the multiple query threads, and construct a corresponding query statement for each log range in the multiple log ranges to obtain multiple query statements, wherein each query statement in the multiple query statements corresponds to one query thread in the multiple query threads;

[0096] A query unit 604 is configured to execute corresponding query statements in parallel through query threads in the plurality of query threads, so as to query the log database for log data within a log range corresponding to each query statement in the plurality of query statements;

[0097] The second execution unit 606 is configured to receive at least one batch of log data returned by the log database in response to each query statement through a data receiving thread, and write the received log data into a buffer block queue in a circular writing manner through the data receiving thread, wherein the buffer block queue is a queue formed by dividing a buffer allocated for the log query request into a plurality of buffer blocks;

[0098] The third execution unit 608 is used to perform processing operations on the log data written into the buffer blocks in the buffer block queue in a loop processing manner through the data processing thread, and summarize and sort the processed log data to obtain a log query result corresponding to the log query request.

[0099] It should be noted that the first execution unit 602 in this embodiment can be used to execute the above step S202, the query unit 604 in this embodiment can be used to execute the above step S204, the second execution unit 606 in this embodiment can be used to execute the above step S206, and the third execution unit 608 in this embodiment can be used to execute the above step S208.

[0100] Through the embodiment provided by the present application, based on the total number of logs in the specified log range requested by the current log query request and the number of query threads in multiple query threads, the log requested by the log query request is divided into multiple log ranges, and a corresponding query statement is constructed for each log range in the multiple log ranges to obtain multiple query statements, wherein one query statement in the multiple query statements corresponds to one query thread in the multiple query threads; the corresponding query statements are executed in parallel by the query threads in the multiple query threads to query the log data in the log range corresponding to each query statement in the multiple query statements from the log database; the data is received by the data receiving thread The log database responds to at least one batch of log data returned by each query statement, and writes the received log data into a buffer block queue in a circular writing manner through a data receiving thread, wherein the buffer block queue is a queue formed by a plurality of buffer blocks divided by a buffer zone allocated for the log query request; a data processing thread performs processing operations on the log data written into the buffer blocks in the buffer block queue in a circular processing manner, and summarizes and sorts the processed log data to obtain a log query result corresponding to the log query request, thereby solving the problem of low query processing efficiency existing in the database log processing method in the related art and improving the query and processing efficiency of the database log.

[0101] In an exemplary embodiment, the above-mentioned device also includes: a sending unit, which is used to send an aggregate query statement to the log database based on the specified log range before dividing the log requested by the log query request into multiple log ranges based on the total number of logs within the specified log range requested by the current log query request and the number of query threads in multiple query threads, wherein the aggregate query statement is used to query the total number of logs within the specified log range; and a determination unit, which is used to determine the number of logs indicated by the aggregate query result as the total number of logs within the specified log range when receiving the aggregate query result returned by the log database in response to the aggregate query statement.

[0102] In an exemplary embodiment, a log range of multiple log ranges is assigned to a query thread among multiple query threads; the first execution unit includes: a query module, which is used to query the maximum column length of the logs in the current log range through the current query thread, wherein the current query thread is any query thread among the multiple query threads, and the current log range is the log range assigned to the current query thread; a construction module, which is used to construct a corresponding query statement for the current log range through the current query thread when the maximum column length of the logs in the current log range is greater than or equal to a specified length threshold; a conversion module, which is used to convert each large object column of the at least one large object column into a corresponding non-large object column when the maximum column length of the logs in the current log range is less than the specified length threshold and the logs in the current log range include at least one large object column, and construct a corresponding query statement for the current log range based on the converted non-large object columns.

[0103] In an exemplary embodiment, the second execution unit includes: a writing module, which is used to write the current batch of log data into the first buffer block of the buffer block queue according to the first block pointer of the buffer block queue through the data receiving thread when the current batch of log data is received through the data receiving thread, wherein the first block pointer is used to indicate the free block in the buffer block queue to which the log data is to be written; wherein, when the writing of the current batch of log data is completed, the first block pointer is moved back to the second buffer block, when the first buffer block is not the last buffer block in the buffer block queue, the second buffer block is the next buffer block of the first buffer block, and when the first buffer block is the last buffer block in the buffer block queue, the second buffer block is the first buffer block in the buffer block queue.

[0104] In an exemplary embodiment, when the current batch of log data is written to the first buffer block of the buffer block queue, the block state of the first buffer block is modified from the idle state to the pending state; the third execution unit includes: a first execution module, which is used to perform a processing operation on the current batch of log data written in the first buffer block according to the second block pointer through the data processing thread when the second block pointer of the buffer block queue points to the first buffer block and the block state of the first buffer block is the pending state, wherein the second block pointer is used to indicate the buffer block in the buffer block queue on which the processing operation is to be performed on the written log data, and when the processing operation is completed on the current batch of log data written in the first buffer block, the block state of the first buffer block is modified from the pending state to the idle state.

[0105] In an exemplary embodiment, each query thread among multiple query threads corresponds to a log data linked list; the above-mentioned device also includes: a saving unit, which is used to save the processed current batch of log data in the form of a newly added linked list node to the log data linked list corresponding to a specified query thread among the multiple query threads after the data processing thread performs a processing operation on the log data in the first buffer block according to the second block pointer, wherein the current batch of log data responds to the log data returned by a specified query statement among the multiple query statements, and the specified query thread is the query thread corresponding to the specified query statement; wherein the processed log data is summarized and sorted on the log data in the log data linked list corresponding to each query thread.

[0106] In an exemplary embodiment, each query thread among multiple query threads corresponds to a log data linked list. After processing operations are performed on log data returned in response to query statements sent by each query thread, the processed log data is saved in the form of newly added linked list nodes in the log data linked list corresponding to each query thread. The third execution unit includes: a second execution module, configured to perform multiple rounds of linked list merging operations through multiple execution threads until the log data linked lists corresponding to the multiple query threads are merged, wherein, in the current round of linked list merging operations, the log data linked lists to be merged are at least two current log data linked lists; the at least two current log data linked lists are divided into at least one current log data linked list pair with two current log data linked lists as a group, and one current log data linked list pair in the at least one current log data linked list pair is assigned to a current execution thread among the multiple execution threads; the linked list nodes of the two current log data linked lists in the current log data linked list pair assigned to the current execution thread are synchronously traversed by the current execution thread, and the traversed linked list nodes are merged into one log data linked list based on the attribute values of the specified attribute column from large to small or from small to large.

[0107] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0108] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored program, wherein the program executes the steps of any of the above method embodiments when it is run.

[0109] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0110] According to another aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the steps of any of the above-described method embodiments through the computer program. In an exemplary embodiment, the electronic device may further comprise a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0111] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0112] According to another aspect of an embodiment of the present application, a computer program product is also provided, which includes a computer program / instruction, which contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication portion 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit 701, the various functions provided by the embodiments of the present application are performed. The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0113] Figure 7 The following schematically shows a block diagram of a computer system structure of an electronic device for implementing an embodiment of the present application. Figure 7As shown, computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to programs stored in ROM 702 or programs loaded from storage 708 into RAM 703. Random access memory 703 also stores various programs and data required for system operation. CPU 701, read-only memory 702, and random access memory 703 are interconnected via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0114] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a local area network card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. Removable media 711, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is installed in the drive 710 as needed, so that computer programs read from the media can be installed in the storage section 708 as needed.

[0115] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709 and / or installed from a removable medium 711. When the computer program is executed by the central processing unit 701, the various functions defined in the system of the present application are performed.

[0116] It should be noted that Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0117] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0118] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A database log processing method, characterized in that: include: Based on the total number of logs within the specified log range requested by the current log query request and the number of query threads in the multiple query threads, the log requested by the log query request is divided into multiple log ranges, and a corresponding query statement is constructed for each log range in the multiple log ranges to obtain multiple query statements, wherein one query statement in the multiple query statements corresponds to one query thread in the multiple query threads; Executing corresponding query statements in parallel through query threads in the multiple query threads to query the log database for log data within a log range corresponding to each query statement in the multiple query statements; receiving, by a data receiving thread, at least one batch of log data returned by the log database in response to each query statement, and writing the received log data into a buffer block queue in a circular writing manner by the data receiving thread, wherein the buffer block queue is a queue formed by dividing a buffer zone allocated for the log query request into a plurality of buffer blocks; The log data written into the buffer block in the buffer block queue is processed by a data processing thread in a cyclic processing manner, and the processed log data is summarized and sorted to obtain a log query result corresponding to the log query request.

2. The method according to claim 1, characterized in that Before dividing the log requested by the log query request into multiple log ranges based on the total number of logs within the specified log range requested by the current log query request and the number of query threads in the multiple query threads, the method further includes: Based on the specified log range, sending an aggregate query statement to the log database, wherein the aggregate query statement is used to query the total number of logs within the specified log range; When receiving the aggregate query result returned by the log database in response to the aggregate query statement, the number of log entries indicated by the aggregate query result is determined as the total number of log entries in the specified log range.

3. The method according to claim 1, characterized in that A log range of the plurality of log ranges is assigned to a query thread of the plurality of query threads; The step of constructing a corresponding query statement for each of the multiple log ranges to obtain multiple query statements includes: querying the maximum column length of logs within a current log range through a current query thread, wherein the current query thread is any query thread among the multiple query threads, and the current log range is a log range allocated to the current query thread; When the maximum column length of the logs within the current log range is greater than or equal to a specified length threshold, constructing a corresponding query statement for the current log range through the current query thread; If the maximum column length of the logs within the current log range is less than a specified length threshold and the logs within the current log range include at least one large object column, each large object column in the at least one large object column is converted into a corresponding non-large object column, and a corresponding query statement is constructed for the current log range based on the converted non-large object columns.

4. The method according to any one of claims 1 to 3, characterized in that Writing the received log data into the buffer block queue in a cyclic writing manner through the data receiving thread includes: When receiving a current batch of log data through the data receiving thread, the data receiving thread writes the current batch of log data into a first buffer block of the buffer block queue according to a first block pointer of the buffer block queue, wherein the first block pointer is used to indicate an idle block in the buffer block queue to which log data is to be written; Among them, when the writing of the current batch of log data is completed, the first block pointer is moved back to the second buffer block, when the first buffer block is not the last buffer block of the buffer block queue, the second buffer block is the next buffer block of the first buffer block, and when the first buffer block is the last buffer block of the buffer block queue, the second buffer block is the first buffer block of the buffer block queue.

5. The method according to claim 4, characterized in that When the current batch of log data is written into the first buffer block of the buffer block queue, the block state of the first buffer block is modified from an idle state to a pending state; The processing operation performed on the log data written into the buffer block in the buffer block queue in a cyclic processing manner by the data processing thread includes: When the second block pointer of the buffer block queue points to the first buffer block and the block state of the first buffer block is in a pending state, the data processing thread performs the processing operation on the current batch of log data written in the first buffer block according to the second block pointer, wherein the second block pointer is used to indicate the buffer block in the buffer block queue on which the processing operation is to be performed on the written log data. When the processing operation is completed on the current batch of log data written in the first buffer block, the block state of the first buffer block is changed from the pending state to the idle state.

6. The method according to claim 5, characterized in that Each query thread in the multiple query threads corresponds to a log data linked list; After the data processing thread performs the processing operation on the log data in the first buffer block according to the second block pointer, the method further includes: Saving the processed current batch of log data in the form of a newly added linked list node to a log data linked list corresponding to a specified query thread among the multiple query threads, wherein the current batch of log data is log data returned in response to a specified query statement among the multiple query statements, and the specified query thread is the query thread corresponding to the specified query statement; The aggregating and sorting of the processed log data is performed on the log data in the log data linked list corresponding to each query thread.

7. The method according to any one of claims 1 to 3, characterized in that Each query thread among the multiple query threads corresponds to a log data linked list, and after performing the processing operation on the log data returned in response to the query statement sent by each query thread, the processed log data is saved in the form of a newly added linked list node in the log data linked list corresponding to each query thread; The processed log data is aggregated and sorted to obtain a log query result corresponding to the log query request, including: Performing multiple rounds of the following linked list merging operation through multiple execution threads until the log data linked lists corresponding to the multiple query threads are merged, wherein in the current round of the linked list merging operation, the log data linked lists to be merged are at least two current log data linked lists; Dividing the at least two current log data linked tables into at least one current log data linked table pair with two current log data linked tables as a group, and assigning one current log data linked table pair in the at least one current log data linked table pair to a current execution thread among the multiple execution threads; The current execution thread synchronously traverses the linked list nodes of the two current log data linked lists in the current log data linked list pair assigned to the current execution thread, and merges the traversed linked list nodes into one log data linked list based on the attribute values of the specified attribute column from large to small or from small to large.

8. A database log processing device, characterized in that: include: A first execution unit is configured to divide the log requested by the log query request into multiple log ranges based on the total number of logs within the specified log range requested by the current log query request and the number of query threads in the multiple query threads, and construct a corresponding query statement for each log range in the multiple log ranges to obtain multiple query statements, wherein one query statement in the multiple query statements corresponds to one query thread in the multiple query threads; A query unit, configured to execute corresponding query statements in parallel through query threads in the multiple query threads, so as to query the log database for log data within a log range corresponding to each query statement in the multiple query statements; a second execution unit, configured to receive, through a data receiving thread, at least one batch of log data returned by the log database in response to each query statement, and write the received log data into a buffer block queue in a circular writing manner through the data receiving thread, wherein the buffer block queue is a queue formed by dividing a buffer zone allocated for the log query request into a plurality of buffer blocks; The third execution unit is used to perform processing operations on the log data written into the buffer block in the buffer block queue in a loop processing manner through a data processing thread, and summarize and sort the processed log data to obtain a log query result corresponding to the log query request.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images