Database log storage methods, devices, equipment and media

Abstract

This application provides a database log storage method, apparatus, device, and medium, which can be applied to the field of database technology. The method includes: dividing multiple log data generated within a preset time period into multiple log data sets according to the operation fields representing the operations of each log data set; merging the multiple log data sets according to the execution time of the operations represented by each log data set to obtain merged log data, wherein the field values ​​representing the operation fields of the merged log data are updated from initial values ​​at the beginning of the preset time period to target values ​​at the end of the preset time period; determining target log storage nodes for storing each merged log data set based on the log priority of the merged log data set and the performance information of each log storage node in the log storage cluster; and sending each merged log data set to the target log storage node for storing the merged log data set.

Description

Technical Field

[0001] This application relates to the field of database technology, and more specifically to a database log storage method, apparatus, device, and medium. Background Technology

[0002] With the development of database technology, databases are increasingly widely used in various industries, and their stability and reliability are crucial for the continuous operation of business. Database logs, as key data recording the history of database operations, play an irreplaceable role in database recovery, auditing, and performance optimization.

[0003] However, in large-scale data processing scenarios, databases generate a large amount of log data in a short period of time. Traditional database log storage methods often face the problem of low processing efficiency when processing this log data. Summary of the Invention

[0004] In view of the above problems, this application provides a database log storage method, apparatus, device and medium.

[0005] According to a first aspect of this application, a database log storage method is provided, comprising: dividing multiple log data generated within a preset time period into multiple log data sets according to the operation fields represented by each of the multiple log data sets, wherein the multiple log data sets within the log data sets correspond to the same operation field; merging the multiple log data sets in the log data sets according to the execution time of the operation represented by each of the multiple log data sets in the log data sets to obtain merged log data, wherein the field value of the operation field represented by the merged log data is updated from an initial value at the start of the preset time period to a target value at the end of the preset time period; determining a target log storage node for storing each of the merged log data sets according to the log priority of the merged log data sets and the performance information of each log storage node in the log storage cluster, wherein the log priority is determined based on the data volume of the merged log data sets and the business corresponding to each of the multiple log data sets merged by the merged log data sets; and sending each of the merged log data sets to the target log storage node for storing the merged log data sets.

[0006] According to an embodiment of this application, the above-mentioned merging of multiple log data in the log data set according to the execution time of the operation represented by each of the multiple log data in the log data set to obtain merged log data includes: merging multiple log data in the log data set according to the execution time of the operation represented by each of the multiple log data in the log data set to obtain initial merged log data; determining a target compression method corresponding to the current load state based on the current processor utilization, current memory utilization, and current read / write latency; and compressing the initial merged log data using the target compression method to obtain the merged log data.

[0007] According to an embodiment of this application, the aforementioned performance information includes node network latency, log replay rate, and node processor utilization. Determining the target log storage node for storing each of the merged log data based on the log priority of the merged log data and the performance information of each log storage node includes: determining the performance score of each of the aforementioned log storage nodes based on their current node network latency, current log replay rate, and current node processor utilization; and determining the target storage node from among the plurality of aforementioned log storage nodes based on the log priority and the performance score of each of the aforementioned log storage nodes.

[0008] According to an embodiment of this application, the above-mentioned merging of multiple log data in the log data set according to the execution time of the operation represented by each of the multiple log data in the log data set to obtain initial merged log data includes: determining, according to the execution time of the operation represented by each of the multiple log data in the log data set, a first target log data corresponding to the first operation within a preset time period and a second target log data corresponding to the last operation within the preset time period; taking the field value of the operation field in the first target log data before the execution of the first operation as the initial value at the beginning of the preset time period, and taking the field value of the operation field in the second target log data after the execution of the last operation as the target value at the end of the preset time period; and determining the transaction identifier sequence of the initial merged log data according to the transaction identifier of each of the log data, wherein the transaction identifier sequence includes the transaction identifier of each of the multiple log data.

[0009] According to an embodiment of this application, the method further includes: determining the transaction dependency relationship between the initial merged log data based on the transaction identifier sequence of each of the initial merged log data, wherein the transaction identifier sequences of two initial merged log data with transaction dependency relationship have the same transaction identifier; determining the replay group identifier of each of the initial merged log data according to the transaction dependency relationship, so that multiple merged log data with transaction dependency relationship can be replayed according to the replay group identifier during log replay, wherein the replay group identifier of two initial merged log data with transaction dependency relationship is the same.

[0010] According to an embodiment of this application, the method further includes: determining the business weight of each operation based on the business represented by each log data in the log data set and the correspondence between the business and the business weight; taking the maximum value among the multiple business weights as the target business weight; and determining the log priority of the merged log data based on the target business weight and the data volume of the merged log data.

[0011] According to an embodiment of this application, the above-mentioned sending of each of the merged log data to a target log storage node for storing each of the merged log data, so as to store the merged log data in the target log storage node, includes: sequentially sending each of the merged log data to the corresponding target log storage node according to the log priority of each of the merged log data.

[0012] A second aspect of this application provides a database log storage device, comprising: a partitioning module, configured to partition multiple log data generated within a preset time period into multiple log data sets according to the operation fields represented by the operations of each of the multiple log data sets, wherein the multiple log data sets within the log data sets correspond to the same operation field; a merging module, configured to merge the multiple log data sets in the log data sets according to the execution time of the operations represented by each of the multiple log data sets in the log data sets, to obtain merged log data, wherein the field value of the operation field represented by the merged log data is updated from an initial value at the start of the preset time period to a target value at the end of the preset time period; a determining module, configured to determine a target log storage node for storing each of the merged log data sets based on the log priority of the merged log data sets and the performance information of each log storage node in the log storage cluster, wherein the log priority is determined based on the data volume of the merged log data sets and the business corresponding to the merged log data sets; and a storage module, configured to send each of the merged log data sets to the target log storage node for storing the merged log data sets.

[0013] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0015] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0016] According to the embodiments of this application, by grouping and merging log data according to the operation fields of each log data, the amount of log data to be stored is greatly reduced, storage efficiency is improved, and log storage performance is enhanced. Furthermore, considering log priority and the performance of each node in the distributed log storage cluster, the target storage node for each merged log data is selected, so that the merged log data matches the target node, thereby improving the targeting of log storage and the utilization efficiency of the log storage cluster, and further optimizing log storage performance. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0018] Figure 1 The diagram illustrates an application scenario of a database log storage method, apparatus, device, and medium according to embodiments of this application.

[0019] Figure 2 A flowchart of a database log storage method according to an embodiment of this application is shown.

[0020] Figure 3 A schematic diagram illustrating the generation of merged log data according to an embodiment of this application is shown.

[0021] Figure 4 A flowchart illustrating the storage of database logs according to an embodiment of this application is shown.

[0022] Figure 5 A structural block diagram of a database log storage device according to an embodiment of this application is shown.

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a database log storage method according to an embodiment of this application is shown. Detailed Implementation

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] In large-scale data processing scenarios, a large amount of log data will be generated in a short period of time. Traditional database log storage methods often face the problem of low processing efficiency when processing this log data.

[0029] This application provides a database log storage method. For multiple log data generated within a preset time period, the multiple log data are divided into multiple log data sets according to the operation fields represented by each log data, wherein multiple log data within a log data set correspond to the same operation field. The multiple log data in the log data set are merged according to the execution time of the operation represented by each log data in the log data set to obtain merged log data. The field value representing the operation field in the merged log data is updated from an initial value at the start of the preset time period to a target value at the end of the preset time period. Based on the log priority of the merged log data and the performance information of each log storage node in the log storage cluster, a target log storage node is determined for storing each merged log data. The log priority is determined based on the data volume of the merged log data and the business corresponding to each of the multiple log data being merged. Each merged log data is sent to the target log storage node for storage.

[0030] According to the embodiments of this application, in the case of concurrent database operations, log data is grouped and merged according to the operation fields of each log data, which greatly reduces the amount of log data to be stored, improves storage efficiency, and improves log storage performance. Furthermore, considering log priority and the performance of each node in the distributed log storage cluster, the target storage node for each merged log data is selected, so that the merged log data matches the target node, improving the targeting of log storage and the utilization efficiency of the log storage cluster, and further optimizing log storage performance.

[0031] Figure 1 The diagram illustrates an application scenario of a database log storage method, apparatus, device, and medium according to embodiments of this application.

[0032] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105, and a log storage cluster 106. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. The log storage cluster 106 may include multiple log storage nodes, such as a first log storage node 106_1 and a second log storage node 106_2.

[0033] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server 105 can be a server providing various services, such as a backend management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The backend management server can analyze and process received user requests and other data, and feed back the processing results (e.g., web pages, information, or data obtained or generated according to user requests) to the terminal devices. Log storage cluster 106 can be used to store log data generated by the database in server 105. In some embodiments, the database can be a distributed database.

[0036] It should be noted that the database log storage method provided in this application embodiment can generally be executed by server 105. Correspondingly, the database log storage device provided in this application embodiment can generally be located in server 105. The database log storage method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the database log storage device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0037] It should be understood that Figure 1 The number of first terminal devices, second terminal devices, third terminal devices, networks, servers, log storage clusters, and log storage nodes shown in the diagram is merely illustrative. Depending on implementation needs, any number of first terminal devices, second terminal devices, third terminal devices, networks, servers, log storage clusters, and log storage nodes can be included.

[0038] The following will be based on Figure 1 The described scene, through Figures 2-4 The database log storage method of the application embodiment is described in detail.

[0039] Figure 2 A flowchart of a database log storage method according to an embodiment of this application is shown.

[0040] like Figure 2 As shown, the database log storage method of this embodiment includes operations S210 to S240.

[0041] In operation S210, for multiple log data generated within a preset time period, the multiple log data are divided into multiple log data sets according to the operation fields represented by each of the multiple log data.

[0042] In operation S220, the multiple log data in the log data set are merged according to the execution time of the operation represented by each log data in the log data set, to obtain merged log data.

[0043] In operation S230, the target log storage node for storing each merged log data is determined based on the log priority of the merged log data and the performance information of each log storage node in the log storage cluster.

[0044] In operation S240, each merged log data is sent to the target log storage node used to store the merged log data, so that the merged log data is stored on the target log storage node.

[0045] In large-scale data processing scenarios, a large amount of log data is generated in a short period of time. When hardware resources are limited, the embodiments of this application divide multiple log data into multiple log data sets according to the operation fields represented by each log data, so that multiple log data in each log data set correspond to the same operation field, and the log data sets are merged, which can significantly reduce the amount of log data to be stored, improve storage efficiency, and meet the log data storage needs in resource-constrained scenarios.

[0046] In some embodiments, after generating log data, the database can write the log data to a message queue for subsequent log merging. For example, in the message queue, the log data is arranged in chronological order of its generation, providing an ordered data foundation for subsequent merging operations.

[0047] For each log dataset, the execution time of the operation represented by each log data is further analyzed. By comparing the order of execution times, multiple log datasets with the same or similar execution times are merged. During the merging process, the focus is on changes in the field values ​​of the operation fields, updating the initial values ​​at the start time of the preset time period to the target values ​​at the end time of the preset time period, thus generating merged log data.

[0048] In the process of dividing multiple log data into multiple log data sets, each log data can be identified to obtain the operation fields representing the operation. For example, log data may include the transaction identifier of the transaction to which the operation belongs, the operation table name, the operation row identifier (such as the primary key), the operation field identifier, the field value before the operation, the field value after the operation, and the operation execution time. The operation fields representing the operation of each log data can be determined based on the operation field identifier in the log data.

[0049] After dividing multiple log data sets into multiple log data collections, the log data within each collection can be merged to obtain merged log data corresponding to each collection. The field value of the merged log data representation operation field is updated from the initial value at the start of the preset time period to the target value at the end of the preset time period. It should be noted that for log data collections containing only one log data, the log data merging operation is not required.

[0050] For example, when merging multiple log data within a log dataset, the log data can be sorted according to the execution time of the operation represented by each log data in the log dataset. Only the pre-operation field value of the first operation and the post-operation field value of the last operation within a preset time period are retained, while the intermediate states of the operation fields within the preset time period are discarded to obtain merged log data, thereby reducing the amount of log data.

[0051] In the embodiments of this application, a log storage cluster independent of the database is used to store log data. After obtaining multiple merged log data, the target log storage node for storing each merged log data can be determined based on the priority of the merged log data and the performance information of each log storage node, so as to match the merged log data with the target node, thereby improving the targeting of log storage and the utilization efficiency of the log storage cluster.

[0052] Log priority is determined based on the volume of merged log data and the specific business logic associated with each merged log entry. For example, merged log data generated by more critical business processes or those with smaller volumes can have higher priority. These high-priority merged log data can be stored on high-performance log storage nodes to ensure efficient replay processing and improve business stability.

[0053] After determining the target log storage node for each merged log data, the merged log data can be sent to the target log storage node to achieve the storage of the merged log data.

[0054] According to the embodiments of this application, by grouping and merging log data according to the operation fields of each log data, the amount of log data to be stored is greatly reduced, storage efficiency is improved, and log storage performance is enhanced. Furthermore, considering log priority and the performance of each node in the distributed log storage cluster, the target storage node for each merged log data is selected, so that the merged log data matches the target node, thereby improving the targeting of log storage and the utilization efficiency of the log storage cluster, and further optimizing log storage performance.

[0055] According to an embodiment of this application, merging multiple log data in a log data set according to the execution time of the operation represented by each log data in the log data set to obtain merged log data includes: merging multiple log data in a log data set according to the execution time of the operation represented by each log data in the log data set to obtain initial merged log data; determining a target compression method corresponding to the current load state based on the current processor utilization, current memory utilization, and current read / write latency; and compressing the initial merged log data using the target compression method to obtain merged log data.

[0056] When merging multiple log data in a log dataset to obtain initial merged log data, the intermediate states of the operation fields within a preset time period can be discarded, resulting in initial merged log data that includes only the initial value at the beginning of the preset time period and the target value at the end of the preset time period.

[0057] After obtaining the initial merged log data, it can be compressed to further reduce the amount of log data to be stored. Furthermore, considering environments with limited hardware resources, embodiments of this application determine the appropriate target compression method based on the current load status.

[0058] The current load state can be determined based on the current processor utilization, current memory utilization, and current read / write latency. Specifically, the current load state can be obtained by weighted summation of the normalized current processor utilization, normalized current memory utilization, and normalized current read / write latency. Based on the load state interval into which the current load state falls, the compression method corresponding to that interval is determined as the target compression method. The current load state can be determined using the following formula (1):

[0059] (1);

[0060] in, Indicates the current load status. This represents the normalized current processor utilization. This represents the normalized current memory usage. This represents the normalized current read / write latency. , , These represent the weighting coefficients for the normalized current processor utilization, the normalized current memory utilization, and the normalized current read / write latency, respectively.

[0061] Since network and read / write operations can become bottlenecks under high load, a higher compression ratio can be used to reduce the pressure on network transmission and storage read / write operations. The correspondence between load ranges and compression methods can be as follows: When the load range is [0, T1), corresponding to a low load state, the corresponding compression method can be the first compression method with a lower compression ratio to avoid compression overhead; when the load range is [T1, T2), corresponding to a medium load state, the corresponding compression method can be the second compression method with a medium compression ratio to pursue compression efficiency; when the load range is [T2, T3), corresponding to a high load state, the corresponding compression method can be the third compression method with a higher compression ratio to balance compression efficiency and compression overhead; when the load range is [T3, MAX], corresponding to an overload state, the corresponding compression method can be the fourth compression method with the highest compression ratio to minimize the amount of log data to be stored.

[0062] According to the embodiments of this application, by dynamically selecting the appropriate target compression method based on the current load status to compress the initial merged log data, it is possible to effectively balance compression efficiency and compression overhead in an environment with limited hardware resources, further reduce the amount of log data to be stored, and improve storage efficiency.

[0063] Figure 3 A schematic diagram illustrating the generation of merged log data according to an embodiment of this application is shown.

[0064] like Figure 3 As shown, multiple log data 301 within a preset time period are divided into multiple log data sets 302, and the multiple log data within each log data set 302 are merged to obtain multiple initial merged log data 303. According to the target compression method 305 that matches the current load state 304, each initial merged log data 303 is compressed to obtain multiple merged log data 306.

[0065] According to an embodiment of this application, multiple log data in a log data set are merged according to the execution time of the operation represented by each log data in the log data set to obtain initial merged log data. This includes: determining, according to the execution time of the operation represented by each log data in the log data set, a first target log data corresponding to the first operation within a preset time period and a second target log data corresponding to the last operation within the preset time period; using the field value of the operation field in the first target log data before the execution of the first operation as the initial value at the beginning of the preset time period, and using the field value of the operation field in the second target log data after the execution of the last operation as the target value at the end of the preset time period; and determining the transaction identifier sequence of the initial merged log data according to the transaction identifier of each log data.

[0066] When merging multiple log data sets in a log dataset, the log data can be sorted according to the operation execution time. This determines the first target log data corresponding to the first operation within a preset time period and the second target log data corresponding to the last operation within the same preset time period. The field values ​​of the operation fields in the first target log data before the first operation are used as the initial values ​​at the start of the preset time period, and the field values ​​of the operation fields in the second target log data after the last operation are used as the target values ​​at the end of the preset time period. For example, the first log data after sorting can be designated as the first target log data, and the last log data as the second target log data.

[0067] For example, within a preset time period, three update operations were performed on field C of row B in table A. The log data for these three update operations were grouped into the same log data set, and these three log data need to be merged. After sorting by the operation execution time, the first log data has the following values: the field value before the operation is 1, and the field value after the operation is 2; the second log data has the same values: the field value before the operation is 2, and the field value after the operation is 3; and the third log data has the same values: the field value before the operation is 3, and the field value after the operation is 4. After merging, only the field values ​​before the operation in the first log data and the field values ​​after the operation in the third log data are retained, resulting in the initial value 1 at the beginning of the preset time period and the target value 4 at the end of the preset time period. The intermediate values ​​2 and 3 of the operation field are discarded.

[0068] Because the transaction identifiers of each log data are different, in addition to the field values, the transaction identifiers also need to be merged. When merging transaction identifiers, the transaction identifiers of each log data can be directly stored as a transaction identifier sequence, which includes the transaction identifiers of each log data. Finally, based on the initial value, the target value, and the transaction identifier sequence, the initial merged log data can be obtained. Furthermore, the initial merged log data may also include operation table identifiers, operation row identifiers, and operation field identifiers.

[0069] For example, if the transaction identifier for the first log data is S1, the transaction identifier for the second log data is S2, and the transaction identifier for the third log data is S3, the merged transaction identifier sequence is (S1, S2, S3). The final initial merged log data includes: a transaction identifier sequence of (S1, S2, S3), an operation table identifier of table A, an operation row identifier of row B, an operation field identifier of field C, a field value of 1 before the operation, and a field value of 4 after the operation.

[0070] According to the embodiments of this application, by merging multiple log data in the log data set, only the initial value at the start time of the preset time period and the target value at the end time are retained, and the transaction identifier sequence is centrally stored, which can significantly reduce the amount of log data and retain key information.

[0071] According to an embodiment of this application, the database log storage method further includes: determining the transaction dependency relationship between the initial merged log data based on the transaction identifier sequence of each initial merged log data; and determining the replay group identifier of each initial merged log data according to the transaction dependency relationship, so that multiple merged log data with transaction dependency relationships can be replayed according to the replay group identifier during log replay.

[0072] Since the initial merged log data combines log data from different transactions, log replay needs to be performed according to transactions to ensure transaction integrity and consistency. Therefore, a replay group identifier for each initial merged log data can be determined based on its transaction identifier sequence. This allows multiple merged log data sets with transaction dependencies to be replayed according to their replay group identifiers. Specifically, two initial merged log data sets with transaction dependencies share the same transaction identifier in their transaction identifier sequences.

[0073] The transaction dependencies between initial merged log data can be determined based on their transaction identifier sequences. Specifically, if two initial merged log data sets contain the same transaction identifier, then these two sets of initial merged log data have a transaction dependency. To correctly handle these transaction dependencies during log replay, initial merged log data sets with the same transaction dependency can be assigned the same replay group identifier. In this way, during log replay, multiple merged log data sets with transaction dependencies can be replayed according to their replay group identifiers, thereby ensuring that transaction integrity and consistency are maintained.

[0074] According to embodiments of this application, by determining the transaction dependencies between the initially merged log data and assigning corresponding replay group identifiers, precise transaction-level replay can be achieved during log replay. This approach not only improves the accuracy of log replay but also helps ensure data integrity and consistency during system failure recovery.

[0075] According to an embodiment of this application, the database log storage method further includes: determining the business weight of each operation based on the business corresponding to the operation represented by each log data in the log data set and the correspondence between the business and the business weight; taking the maximum value among multiple business weights as the target business weight; and determining the log priority of the merged log data based on the target business weight and the amount of merged log data.

[0076] Since the log dataset can include log data from multiple different business processes, when determining log priorities, the log data with higher importance among the multiple log data can be used as the standard to determine the log priority of the merged log data, ensuring that the more important log data is not ignored.

[0077] In embodiments of this application, the correspondence between services and service weights can be pre-stored in a mapping table. When it is necessary to determine the log priority, the mapping table is first queried based on the service corresponding to the operation represented by each log data to obtain the service weight of the operation represented by each log data. A higher service weight indicates a higher importance of the service.

[0078] After obtaining the business weights of the operations represented by each log data, the maximum value among the multiple business weights can be used as the target business weight. Based on the target business weight and the amount of merged log data, the log priority of the merged log data can be determined.

[0079] In some embodiments, when determining log priority, the service level protocol level (SLP) of the business corresponding to the operation represented by each log data can also be considered, and the log priority of the merged log data can also be determined based on the highest SLP level to ensure the reliability of the business service. The log priority can be determined by the following formula (2):

[0080] (2);

[0081] in, Indicates the normalized log priority. This represents the amount of data after normalization. This represents the normalized target business weight. This indicates the normalized service level agreement level. , , These are the weighting coefficients.

[0082] Before determining log priorities based on data volume, target business weight, and service level, these factors can be normalized. For example, they can be normalized by dividing each factor by its maximum value in the system. The maximum values ​​for data volume, target business weight, and service level can be obtained by statistically analyzing the merged log data already stored in the log storage cluster.

[0083] According to the embodiments of this application, by comprehensively considering the amount of data and the target business weight to determine the log priority, the importance and processing priority of the merged log data can be more comprehensively reflected, thereby improving the overall storage performance and resource utilization.

[0084] According to an embodiment of this application, a target log storage node for storing each merged log data is determined based on the log priority of the merged log data and the performance information of each log storage node, including: determining the performance score of each log storage node based on the current node network latency, current log playback rate, and current node processor utilization; and determining the target storage node from multiple log storage nodes based on the log priority and the performance score of each log storage node.

[0085] In the embodiments of this application, performance information includes node network latency, log replay rate, and node processor utilization. When determining the performance score of each log storage node based on the current node network latency, current log replay rate, and current node processor utilization, the current node network latency, current log replay rate, and current node processor utilization can be normalized respectively using the maximum node network latency, maximum log replay rate, and maximum node processor utilization to obtain the normalized current node network latency, normalized current log replay rate, and normalized current node processor utilization.

[0086] The process of determining the performance score can be shown in the following formula (3):

[0087] (3);

[0088] in, Indicates performance score, This represents the normalized network latency of the current node. This represents the normalized current log replay rate. This represents the normalized processor utilization of the current node. , , These represent the weighting coefficients for the normalized current node network latency, the normalized current log replay rate, and the normalized current node processor utilization, respectively.

[0089] After determining the performance score of each log storage node, the target log storage node for each merged log data can be selected from multiple log storage nodes. For example, for merged log data with higher priority, a log storage node with a higher performance score can be selected as the target log storage node. For merged log data with lower priority, a log storage node with a moderate or lower performance score that still meets basic storage requirements can be selected. This achieves load balancing and avoids situations where some nodes are overloaded while others are idle.

[0090] In addition, a performance score threshold can be set. When the performance score of a log storage node is lower than the threshold, no new merged log data will be allocated to that node to ensure the stability and reliability of the overall storage system.

[0091] According to embodiments of this application, by comprehensively considering log priority and the performance score of log storage nodes to determine the target log storage node, storage resources can be allocated more reasonably, thereby improving the overall performance and reliability of the storage system.

[0092] According to an embodiment of this application, sending each merged log data to a target log storage node for storing the merged log data, so as to store the merged log data on the target log storage node, includes: sending each merged log data to the corresponding target log storage node in sequence according to the log priority of each merged log data.

[0093] In scenarios with limited hardware resources, it is difficult to send multiple merged log data to the target log storage node at the same time. Therefore, the merged log data can be sent to the corresponding target log storage node in order of log priority from high to low according to the log priority of each merged log data.

[0094] This sequential sending method ensures that higher-priority merged log data is processed and stored first, thus meeting the real-time and reliability requirements of critical business operations for log storage. Meanwhile, lower-priority merged log data can be processed when system resources are relatively idle, avoiding excessive impact on overall system performance.

[0095] Asynchronous transmission can be used during the sending of merged log data to reduce the performance overhead of the sending operation. Specifically, after sending the merged log data, the sending database does not need to wait for confirmation from the target log storage node (the receiving node) and can continue processing other tasks. Upon receiving the merged log data, the target log storage node performs the corresponding storage operations and sends an confirmation message to the sender upon completion. This approach fully utilizes system resources and improves log storage efficiency.

[0096] In addition, to ensure the security and integrity of merged log data during transmission, encryption and verification mechanisms can be employed. Encryption mechanisms can encrypt the merged log data to prevent tampering during transmission. Verification mechanisms can perform integrity checks on the received merged log data to ensure that no data is lost or damaged during transmission.

[0097] According to embodiments of this application, by sending merged log data in order of log priority, it is possible to ensure that log data is transmitted to the target storage node efficiently, securely, and completely, thereby improving storage performance.

[0098] Figure 4 A flowchart illustrating the storage of database logs according to an embodiment of this application is shown.

[0099] like Figure 4 As shown, the architecture for storing database logs includes a database and target log storage nodes. The database includes a transaction manager, a log generation module, a log merging module, a compression and adjustment module, and a node determination module.

[0100] In operation S401, the transaction manager commits the transaction to the log generation module.

[0101] When operating S402, the log generation module generates log data.

[0102] During operation S403, the log generation module sends log data to the log merging module.

[0103] In operation S404, the log merging module queries the compression adjustment module for the target compression method.

[0104] When operating S405, the compression adjustment module determines the target compression method based on the current load status.

[0105] During operation S406, the compression adjustment module returns the target compression method to the log merging module.

[0106] In operation S407, the log merging module merges log data into initial merged log data.

[0107] In operation S408, the log merging module compresses the initial merged log data into merged log data according to the target compression method.

[0108] In operation S409, the log merging module outputs merged log data to the node determination module.

[0109] In operation S410, the node determination module determines the target log storage node for storing merged log data.

[0110] During operation S411, the node determination module will transfer the merged log data to the target log storage node in the log storage cluster.

[0111] In the embodiments of this application, after receiving the merged log data, the target log storage node can decompress the merged log data and replay the logs according to the replay group identifier.

[0112] Based on the above-described database log storage method, this application also provides a database log storage device. The following will be combined with... Figure 5 The device is described in detail.

[0113] Figure 5 A structural block diagram of a database log storage device according to an embodiment of this application is shown.

[0114] like Figure 5 As shown, the database log storage device 500 of this embodiment includes a partitioning module 510, a merging module 520, a determination module 530, and a storage module 540.

[0115] The partitioning module 510 is used to divide multiple log data generated within a preset time period into multiple log data sets according to the operation fields represented by each log data set. In one embodiment, the partitioning module 510 can be used to execute the operation S210 described above, which will not be repeated here.

[0116] The merging module 520 is used to merge multiple log data in the log data set according to the execution time of the operation represented by each log data in the log data set, to obtain merged log data. The field value of the operation field in the merged log data is updated from its initial value at the start of a preset time period to its target value at the end of the preset time period. In one embodiment, the merging module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0117] The determining module 530 is used to determine the target log storage node for storing each merged log data based on the log priority of the merged log data and the performance information of each log storage node in the log storage cluster. The log priority is determined based on the amount of merged log data and the corresponding business. In one embodiment, the determining module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0118] The storage module 540 is used to send each merged log data to a target log storage node for storing the merged log data, so as to store the merged log data on the target log storage node. In one embodiment, the storage module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0119] According to an embodiment of this application, the merging module 520 includes a merging submodule, a compression determination submodule, and a compression submodule.

[0120] The merging submodule is used to merge multiple log data in the log data set according to the execution time of the operation represented by each log data in the log data set, so as to obtain the initial merged log data.

[0121] The compression determination submodule is used to determine the target compression method corresponding to the current load state based on the current processor utilization, current memory usage, and current read / write latency.

[0122] The compression submodule is used to compress the initial merged log data using the target compression method to obtain merged log data.

[0123] According to an embodiment of this application, the performance information includes node network latency, log replay rate, and node processor utilization; the determination module 530 includes a performance determination submodule and a node determination submodule.

[0124] The performance determination submodule is used to determine the performance score of each log storage node based on the current node network latency, current log playback rate, and current node processor utilization.

[0125] The node determination submodule is used to determine the target storage node from multiple log storage nodes based on log priority and the performance score of each log storage node.

[0126] According to an embodiment of this application, the merging submodule includes a log determination unit, a merging unit, and a transaction determination unit.

[0127] The log determination unit is used to determine, according to the execution time of the operation represented by each of the multiple log data in the log data set, the first target log data corresponding to the first operation within the preset time period and the second target log data corresponding to the last operation within the preset time period.

[0128] The merging unit is used to take the field value of the operation field in the first target log data before the first operation is performed as the initial value at the beginning of the preset time period, and take the field value of the operation field in the second target log data after the last operation is performed as the target value at the end of the preset time period.

[0129] The transaction determination unit is used to determine the initial merge log data transaction identifier sequence based on the transaction identifier of each log data. The transaction identifier sequence includes the transaction identifiers of each log data.

[0130] According to embodiments of this application, the merging submodule further includes a dependency determination unit and a playback determination unit.

[0131] The dependency determination unit is used to determine the transaction dependency relationship between the initial merged log data based on the transaction identifier sequence of each initial merged log data, wherein the transaction identifier sequences of two initial merged log data with transaction dependency relationship have the same transaction identifier.

[0132] The replay determination unit is used to determine the replay group identifier of each initial merged log data according to the transaction dependency, so that multiple merged log data with transaction dependency can be replayed according to the replay group identifier during log replay. Among them, the replay group identifiers of two initial merged log data with transaction dependency are the same.

[0133] According to an embodiment of this application, the database log storage device 500 further includes a business weight determination module, a log weight determination module, and a priority determination module.

[0134] The business weight determination module is used to determine the business weight of each operation based on the business represented by each log data in the log data set and the correspondence between the business and the business weight.

[0135] The log weight determination module is used to take the maximum value among multiple business weights as the target business weight.

[0136] The priority determination module is used to determine the log priority of merged log data based on the target business weight and the amount of merged log data.

[0137] According to an embodiment of this application, storage module 540 includes a storage submodule.

[0138] The storage submodule is used to send each merged log data to the corresponding target log storage node in sequence according to the log priority of each merged log data.

[0139] According to embodiments of this application, any multiple modules among the partitioning module 510, merging module 520, determining module 530, and storage module 540 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the partitioning module 510, merging module 520, determining module 530, and storage module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the partitioning module 510, merging module 520, determining module 530, and storage module 540 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0140] Figure 6 A block diagram of an electronic device suitable for implementing a database log storage method according to an embodiment of this application is shown.

[0141] like Figure 6As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0142] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0143] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0144] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0145] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0146] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the database log storage method provided in the embodiments of this application.

[0147] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0148] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0149] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0150] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0153] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A database log storage method, characterized in that, The method includes: For multiple log data generated within a preset time period, the multiple log data are divided into multiple log data sets according to the operation fields represented by each of the multiple log data, wherein multiple log data in the log data set correspond to the same operation field; According to the execution time of the operation represented by each of the log data in the log data set, the log data in the log data set is merged to obtain merged log data, wherein the field value of the operation field of the merged log data is updated from the initial value at the start time of the preset time period to the target value at the end time of the preset time period. Based on the log priority of the merged log data and the performance information of each log storage node in the log storage cluster, a target log storage node for storing each of the merged log data is determined. The log priority is determined based on the data volume of the merged log data and the business corresponding to each of the multiple log data merged by the merged log data. Each of the merged log data is sent to a target log storage node for storing the merged log data, so as to store the merged log data on the target log storage node.

2. The method according to claim 1, characterized in that, The step of merging multiple log data in the log data set according to the execution time of the operation represented by each of the multiple log data in the log data set to obtain merged log data includes: According to the execution time of the operation represented by each of the log data in the log data set, the log data is merged to obtain the initial merged log data; Based on the current load state determined by the current processor utilization, current memory usage, and current read / write latency, determine the target compression method corresponding to the current load state; The initial merged log data is compressed using the target compression method to obtain the merged log data.

3. The method according to claim 1, characterized in that, The performance information includes node network latency, log replay rate, and node processor utilization. The step of determining the target log storage node for storing each of the merged log data based on the log priority of the merged log data and the performance information of each log storage node in the log storage cluster includes: The performance score of each log storage node is determined based on its current node network latency, current log playback rate, and current node processor utilization. The target storage node is determined from the plurality of log storage nodes based on the log priority and the performance score of each log storage node.

4. The method according to claim 2, characterized in that, The step of merging multiple log data in the log data set according to the execution time of the operation represented by each of the multiple log data in the log data set to obtain initial merged log data includes: Based on the execution time of the operation represented by each of the log data in the log data set, determine the first target log data corresponding to the first operation within the preset time period and the second target log data corresponding to the last operation within the preset time period from the multiple log data; The field value of the operation field in the first target log data before the first operation is performed is used as the initial value at the beginning of the preset time period, and the field value of the operation field in the second target log data after the last operation is used as the target value at the end of the preset time period. Based on the transaction identifiers of each log data, a sequence of transaction identifiers for the initial merged log data is determined, wherein the sequence of transaction identifiers includes the transaction identifiers of each of the log data.

5. The method according to claim 4, characterized in that, The method further includes: Based on the transaction identifier sequence of each of the initial merged log data, the transaction dependency relationship between the initial merged log data is determined, wherein the transaction identifier sequences of two initial merged log data with transaction dependency relationship have the same transaction identifier; Based on the transaction dependencies, a replay group identifier is determined for each of the initial merged log data, so that multiple merged log data with transaction dependencies can be replayed according to the replay group identifier during log replay, wherein the replay group identifiers of two initial merged log data with the transaction dependencies are the same.

6. The method according to claim 1, characterized in that, The method further includes: Based on the business corresponding to the operation represented by each log data in the log data set and the correspondence between the business and the business weight, the business weight of each operation is determined. The maximum value among the multiple business weights is taken as the target business weight; Based on the target business weight and the amount of data in the merged log data, the log priority of the merged log data is determined.

7. The method according to claim 1, characterized in that, The step of sending each of the merged log data to a target log storage node for storing the merged log data includes: According to the log priority of each merged log data, each merged log data is sent to the corresponding target log storage node in sequence.

8. A database log storage device, characterized in that, The device includes: The partitioning module is used to divide multiple log data generated within a preset time period into multiple log data sets according to the operation fields represented by each of the multiple log data sets, wherein multiple log data sets within the log data set correspond to the same operation field; The merging module is used to merge multiple log data in the log data set according to the execution time of the operation represented by each of the multiple log data in the log data set, to obtain merged log data, wherein the field value of the operation field of the merged log data is updated from the initial value at the start time of the preset time period to the target value at the end time of the preset time period. The determination module is used to determine the target log storage node for storing each of the merged log data based on the log priority of the merged log data and the performance information of each log storage node in the log storage cluster. The log priority is determined based on the data volume of the merged log data and the business corresponding to the merged log data. A storage module is used to send each of the merged log data to a target log storage node for storing the merged log data, so as to store the merged log data on the target log storage node.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images