A cross-process behavior load collection system for a database system
The cross-process behavioral load collection system addresses the shortcomings of behavioral load collection in cross-process database systems, enabling comprehensive collection and unified management of behavioral data, and supporting autonomous management and performance optimization of the database system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 上海沄熹科技有限公司
- Filing Date
- 2023-06-08
- Publication Date
- 2026-05-22
Smart Images
Figure CN116775762B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database technology, and more specifically to a cross-process behavior load collection system for database systems. Background Technology
[0002] Existing data collected in the database primarily focuses on cluster and node detection and log monitoring, mainly targeting cluster status monitoring and execution failure monitoring. While there is a strong emphasis on database error correction, the collection of behavioral data related to application load, compilation, and execution is insufficient. This includes details such as specific statistics and optimization rules used during the SQL compilation phase, the specific time and memory consumption during the execution and planning phases, network time and wait times during execution, etc. Users lack comprehensive behavioral data about the database system, making it impossible to accurately evaluate system performance and make optimization decisions regarding SQL behavior from different perspectives. For example, the inability to comprehensively assess application load and the lack of behavioral data during the compilation and execution phases lead to inaccurate optimizer estimations, making it difficult to select the optimal execution plan. Consequently, the autonomous management system cannot comprehensively understand, manage, and adjust database application and SQL behavior to achieve performance improvements. Therefore, the application load collection system aims to collect behavioral data from the database regarding application load, compilation phase, and execution phase in a comprehensive and multi-layered manner. This behavioral data will be stored to provide data support for the autonomous data management platform in building a data warehouse and optimizing SQL engine performance.
[0003] To address the ever-emerging new scenarios and challenges, as well as the demands for massive and diverse data storage, database functionality integration has become an essential skill for every database vendor. To integrate the storage of various data types, some databases choose to merge two database systems with different data types or even different programming languages. This results in different database subsystems, while logically unified, potentially running in two different processes. To perform behavioral load sampling in such cross-process database systems, a corresponding cross-process behavioral load sampling system needs to be designed. This system must ensure that behavioral load sampling can be performed independently across database subsystems in different processes or even different languages, and that it can be managed uniformly using a single switch for unified control, while also not significantly impacting the normal execution of the entire database system or a single database subsystem.
[0004] How to collect behavioral load data from cross-process database systems is a technical problem that needs to be solved. Summary of the Invention
[0005] The technical objective of this invention is to address the above-mentioned shortcomings by providing a cross-process behavioral load collection system for database systems, thereby solving the technical problem of how to collect behavioral load from cross-process database systems.
[0006] This invention provides a cross-process behavioral load acquisition system for database systems, applicable between different database subsystem processes. The behavioral load acquisition system includes a switch synchronization system, an acquisition system, and a cross-process acquisition item aggregation system.
[0007] The switch synchronization system is used to transmit switch information and collection items between different database subsystem processes via a message queue;
[0008] There are two acquisition systems. For each of the two different database subsystems, there is a corresponding acquisition system and a switch state cache structure. The two acquisition systems work together to collect acquisition items simultaneously in their respective database subsystem processes.
[0009] The data collection aggregation system is used to collect data from the two data collection systems and manage the data in a unified manner.
[0010] Preferably, the two different database subsystems communicate via a bidirectional queue (BiMessageQueue), with different byte headers to distinguish message types. When the byte header is C, it indicates that the message is used to synchronize switch states, and when the byte header is M, it indicates that the message is used to transmit collected items.
[0011] Preferably, for two different database subsystem processes, Processor1 and Processor2,
[0012] When Processor1 changes the acquisition switch state, for a switch synchronization system, Processor1 executes the following:
[0013] Modify and lock the corresponding switch in the local switch cache. After the modification is successful, send a state synchronization command to Processor2 through the message queue. The state synchronization command includes a C byte header, the switch ID, and the switch state length.
[0014] Read the switch status length in bytes and convert the read information to a string type;
[0015] Correspondingly, for the switch synchronization system, Processor2 executes as follows:
[0016] Receive the switch ID that needs to be synchronized and its state that needs to be changed;
[0017] Lock the corresponding switch in the local switch cache, change its state, release the lock after the change, and send a response command to Processor1. The response command includes a C byte header and a response, which indicates whether the switch change was successful or failed.
[0018] When Processor1 receives a response indicating that the switch change was successful, it releases the local lock. When Processor1 receives a response indicating that the switch change failed, it rolls back the state of the local switch, releases the lock, and then returns a change error to the user.
[0019] As a preferred option, data acquisition will only be performed when the acquisition switch is turned on for each acquisition system;
[0020] For database subsystem processes without disk persistence threads, when retrieving data from the corresponding database subsystem, the corresponding data acquisition system performs the following operations:
[0021] Collect behavioral information of a behavioral load in the database system and save it to the collection item in a specified format;
[0022] The collected items are saved to a queue. After a predetermined time or when the queue reaches a predetermined length, the collected items in the queue are serialized and combined into a serialized byte stream.
[0023] The byte stream is sent to the switch synchronization system, which then sends the serialized byte stream to the database subsystem process with a disk write thread via a message queue.
[0024] For the database subsystem process with a disk write thread, its corresponding data acquisition system executes the following:
[0025] The serialized byte stream is split into individual acquisition item byte streams by a parsing thread;
[0026] The byte stream of each collected item is fed into the data write-to-disk thread one by one, and the database write-to-disk thread sends each byte stream of collected items to the processing queue.
[0027] The database disk persistence thread monitors and processes the queue, uploading the byte streams of collected items in the queue to the collected item aggregation system.
[0028] Preferably, for database subsystem processes with disk write threads, when retrieving data from the corresponding database subsystem, the corresponding data acquisition system performs the following operations:
[0029] Collect behavioral information of a behavioral load in the database system and save it to the collection item in a specified format;
[0030] The collected items are saved to a queue. After a predetermined time or when the queue reaches a predetermined length, the collected items in the queue are serialized to obtain a byte stream of collected items, and then the byte stream of collected items is sent to the processing queue.
[0031] The database disk persistence thread monitors and processes the queue, uploading the byte streams of collected items in the queue to the collected item aggregation system.
[0032] Preferably, the data collection item aggregation system is configured with a table to store the data collection item byte stream, with each row in the table representing a data collection item byte stream.
[0033] Preferably, the data collection item aggregation system is used to read the data collection item byte stream in the table and deserialize the data collection item byte stream one by one to obtain the behavioral information of the behavioral load in the database system.
[0034] Preferably, the serialized byte stream format is byte length + message + byte length + message ... + M-byte header;
[0035] The format of the byte stream of the collected item is byte length + message + M-word header.
[0036] The present invention provides a cross-process behavior load collection system for database systems, which has the following advantages: the switch synchronization system transmits switch information and collection item information between different processes through a message queue, ensuring the synchronization of switch states between processes in terms of process flow, ensuring the uniformity of collection behavior, and two independent collection systems collect data simultaneously in two different processes without affecting each other. The cross-process collection item aggregation system can asynchronously aggregate collection items together, enabling unified management of collection information and reducing the impact of load collection on the normal execution process. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0038] The invention will be further described below with reference to the accompanying drawings.
[0039] Figure 1 This is a schematic diagram of the structure of a cross-process behavior load collection system for a database system, as shown in the embodiment.
[0040] Figure 2 This is a structural diagram of a switch synchronization system in a cross-process behavior load acquisition system for a database system, as illustrated in this embodiment.
[0041] Figure 3 This is a structural diagram of a data collection item aggregation system in a cross-process behavior load collection system for a database system, as shown in the embodiment. Detailed Implementation
[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0043] This invention provides a cross-process behavioral load collection system for database systems, which solves the technical problem of how to collect behavioral load from cross-process database systems.
[0044] Example:
[0045] This invention provides a cross-process behavioral load acquisition system for database systems, applicable between different database subsystem processes. The behavioral load acquisition system includes a switch synchronization system, an acquisition system, and a cross-process acquisition item aggregation system.
[0046] The switch synchronization system is used to transmit switch information and data collection items between different database subsystem processes via message queues. There are two data collection systems; for each of the two different database subsystems, there is one corresponding data collection system and a switch state cache structure. The two data collection systems work together to simultaneously collect data items within their respective database subsystem processes. The data collection item aggregation system is used to collect data items from the two systems and manage them uniformly.
[0047] In this embodiment, the two different database subsystems communicate with each other by using a bidirectional queue (BiMessageQueue), and the message type is distinguished by different byte headers. When the byte header is C, it indicates that the message is used to synchronize the switch state, and when the byte header is M, it indicates that the message is used to transmit the collected items.
[0048] For each data acquisition system, data acquisition will only occur when the data acquisition switch is turned on.
[0049] For database subsystem processes without disk persistence threads, when retrieving data from the corresponding database subsystem, the corresponding data acquisition system performs the following operations:
[0050] (1) Collect behavioral information of a behavioral load in the database system and save it to the collection item in the specified format;
[0051] (2) Save the collected items to a queue. After the predetermined time or the queue reaches the predetermined length, serialize the collected items in the queue and form a serialized byte stream.
[0052] (3) The byte stream is sent to the switch synchronization system, which sends the serialized byte stream to the database subsystem process with a disk write thread through a message queue.
[0053] Correspondingly, for the database subsystem process with a disk write thread, after receiving the serialized byte stream, its corresponding acquisition system performs the following:
[0054] (1) The serialized byte stream is split into individual acquisition item byte streams by the parsing thread;
[0055] (2) The byte stream of the collected items is fed into the data disk writing thread one by one, and the database disk writing thread sends each byte stream of the collected items to the processing queue;
[0056] (3) The database disk write thread monitors and processes the queue, and uploads the byte stream of the collected items in the processing queue to the collected item summary system.
[0057] For database subsystem processes with disk write threads, when retrieving collected items from the corresponding database subsystem, the corresponding collection system performs the following operations:
[0058] (1) Collect behavioral information of a behavioral load in the database system and save it to the collection item in the specified format;
[0059] (2) Save the collected items to a queue. After the predetermined time or the queue reaches the predetermined length, serialize the collected items in the queue to obtain the byte stream of the collected items, and send the byte stream of the collected items to the processing queue.
[0060] (3) The database disk write thread monitors and processes the queue, and uploads the byte stream of the collected items in the processing queue to the collected item summary system.
[0061] For two different database subsystem processes, Processor1 and Processor2, when Processor1 changes the acquisition switch state, for the switch synchronization system, Processor1 executes as follows:
[0062] (1) Modify and lock the corresponding switch in the local switch cache. After the modification is successful, send a state synchronization command to Processor2 through the message queue. The state synchronization command includes a C byte header, the ID of the switch, and the length of the switch state.
[0063] (2) Read the information of the switch status length bytes and convert the read information into string type.
[0064] Correspondingly, for the switch synchronization system, Processor2 executes as follows:
[0065] (1) Receive the switch ID that needs to be synchronized and the state that needs to be changed;
[0066] (2) Lock the corresponding switch in the local switch cache, change its state, release the lock after the change, and send a response command to Processor1. The response command includes a C byte header and a response, which indicates whether the switch change was successful or failed.
[0067] When Processor1 receives a response indicating that the switch change was successful, it releases the local lock. When Processor1 receives a response indicating that the switch change failed, it rolls back the state of the local switch, releases the lock, and then returns a change error to the user.
[0068] In this embodiment, the data collection item aggregation system is configured with a table to store the data collection item byte streams. Each row in the table represents one data collection item byte stream. During data collection item analysis, the data collection item aggregation system reads the data collection item byte streams from the table and deserializes each data collection item byte stream to obtain the behavioral information of the behavioral load in the database system.
[0069] As a specific implementation of the load acquisition system, two different database subsystem processes, Processor1 and Processor2, are given.
[0070] Overall load acquisition system such as Figure 1As shown, Processor1 and Processor2 represent different database subsystem processes. Each subsystem runs an independent database system, a corresponding acquisition system, and an acquisition switch state buffer structure. Processes communicate using a bidirectional queue (BiMessageQueue), employing different byte headers to distinguish different message types. If the message header is C (control), it indicates the message is used to synchronize the switch state; if the header is M (message), it indicates the message is used to transmit acquisition information. Processor2's receiving end processes messages differently based on the different byte headers. Each database subsystem has an independent acquisition system that only acquires data when the acquisition switch is on. This acquisition system collects the behavioral information of a behavioral payload (a SQL statement) within the database system and saves it to the acquisition item according to a specified format. Process1 then serializes the acquisition items into a byte stream in batches to transmit as many messages as possible at a time. In Processor2, the parsing thread splits the data into individual byte streams, which are then fed into the data fallout thread. The data fallout thread stores each byte stream in a table according to its byte format for later analysis.
[0071] Switching synchronization system such as Figure 2 As shown, when Processor1 changes the state of the acquisition switch, it first modifies and locks the corresponding switch in the local switch cache. After the change is successful, Processor1 sends a state synchronization command to Processor2 through the message queue. This command has a C header, followed by a 4-byte switch ID, and then a 4-byte switch state length. Then, it reads the switch state length bytes and converts them to a string type. Processor2 then obtains the switch ID that needs to be synchronized and the state that needs to be changed. Processor2 locks the corresponding switch in its local switch cache, changes the state, releases the lock, and sends a response command to Processor1. The response command has a C header, followed by a 1-byte response. If Processor2's switch change is successful, it returns ACK (0x00); otherwise, it returns DCK (0x01). Only when Processor1 receives an ACK response will it release the local lock. If it receives a DCK response, it will roll back the local switch state, release the lock, and then return a change error to the user. This process ensures that the on / off states of the two processes remain consistent.
[0072] Collection Item Summary System, such as Figure 3 As shown, the acquisition system in Processor1 saves the acquired items to a queue. After a certain time or when the queue reaches a certain length, it serializes all the acquired items in the queue, assembling them into a byte stream. The byte stream format is 4 bytes long + N bytes of message + 4 bytes long + M bytes of message, and then adds an M-byte header. This stream is then sent to Processor2 via a message queue. The parsing thread in Processor2 splits the entire byte stream according to the corresponding message format and places each acquired item's byte stream into a processing queue. The acquisition system in Processor2 also saves the acquired items, similar to Processor1, serializing them periodically or in fixed quantities. However, unlike Processor1, the serialized acquired items are directly placed into the processing queue without concatenation. The data persistence thread in Processor2 monitors the processing queue and feeds the acquired item byte streams from the queue into the storage engine to save them to a specified table. Each row in the table represents one acquired item's byte stream. During subsequent analysis, these rows are read and deserialized one by one to obtain the specific content of the behavioral load acquisition.
[0073] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the means in the different embodiments described above, and these embodiments are also within the protection scope of the present invention.
Claims
1. A cross-process behavioral load acquisition system for database systems, characterized in that, The behavioral load acquisition system, which is applied between different database subsystem processes, includes a switch synchronization system, an acquisition system, and a cross-process acquisition item aggregation system. The switch synchronization system is used to transmit switch information and collection items between different database subsystem processes via a message queue; There are two acquisition systems. For each of the two different database subsystems, there is a corresponding acquisition system and a switch state cache structure. The two acquisition systems work together to collect acquisition items simultaneously in their respective database subsystem processes. The data collection aggregation system is used to collect data from the two data collection systems and manage the data in a unified manner. For the two different database subsystem processes Processor1 and Processor2 When the acquisition switch state is changed in Processor1, for a switch synchronization system, Processor1 executes the following: Modify and lock the corresponding switch in the local switch cache. After the modification is successful, send a state synchronization command to Processor2 through the message queue. The state synchronization command includes a C byte header, the switch ID, and the switch state length. Read the switch status length in bytes and convert the read information to a string type; Correspondingly, for a switch synchronization system, Processor2 executes as follows: Receive the switch ID that needs to be synchronized and its state that needs to be changed; Lock the corresponding switch in the local switch cache, change its state, release the lock after the change, and send a response command to Processor1. The response command includes a C byte header and a response, which indicates whether the switch change was successful or failed. When Processor1 receives a response indicating that the switch change was successful, it releases the local lock. When Processor1 receives a response indicating that the switch change failed, it rolls back the state of the local switch, releases the lock, and then returns a change error to the user. For each data acquisition system, data acquisition will only occur when the data acquisition switch is turned on. For database subsystem processes without disk persistence threads, when retrieving data from the corresponding database subsystem, the corresponding data acquisition system performs the following operations: Collect behavioral information of a behavioral load in the database system and save it to the collection item in a specified format; The collected items are saved to a queue. After a predetermined time or when the queue reaches a predetermined length, the collected items in the queue are serialized and combined into a serialized byte stream. The byte stream is sent to the switch synchronization system, which then sends the serialized byte stream to the database subsystem process with a disk write thread via a message queue. For the database subsystem process with a disk write thread, its corresponding data acquisition system executes the following: The serialized byte stream is split into individual acquisition item byte streams by a parsing thread; The byte stream of each collected item is fed into the data write-to-disk thread one by one, and the database write-to-disk thread sends each byte stream of collected items to the processing queue. The database disk persistence thread monitors and processes the queue, and uploads the byte stream of the collected items in the processing queue to the collected item aggregation system; For database subsystem processes with disk write threads, when retrieving collected items from the corresponding database subsystem, the corresponding collection system performs the following operations: Collect behavioral information of a behavioral load in the database system and save it to the collection item in a specified format; The collected items are saved to a queue. After a predetermined time or when the queue reaches a predetermined length, the collected items in the queue are serialized to obtain a byte stream of collected items, and then the byte stream of collected items is sent to the processing queue. The database disk persistence thread monitors and processes the queue, uploading the byte streams of collected items in the queue to the collected item aggregation system.
2. The cross-process behavioral load acquisition system for a database system according to claim 1, characterized in that, The two different database subsystems communicate with each other by using a bidirectional queue (BiMessageQueue), and the message type is distinguished by different byte headers. When the byte header is C, it means that the message is used to synchronize the switch state, and when the byte header is M, it means that the message is used to transmit the collected items.
3. The cross-process behavioral load acquisition system for a database system according to claim 1, characterized in that, The data collection item aggregation system is configured with a table to store the data collection item byte stream. Each row in the table represents a data collection item byte stream.
4. The cross-process behavioral load acquisition system for a database system according to claim 3, characterized in that, The data collection item aggregation system is used to read the data collection item byte stream from the table and deserialize each data collection item byte stream to obtain the behavioral information of the behavioral load in the database system.
5. A cross-process behavioral load acquisition system for a database system according to claim 4, characterized in that, The format of a serialized byte stream is byte length + message + byte length + message + M-byte header; The format of the byte stream of the collected item is byte length + message + M-word header.