Log space management method, device, electronic equipment and medium

By judging the occupancy rate of log files and comparing the LSN of log files in MySQL database, dynamically managing log space, solving the problem of business interruption caused by fixed log space, realizing the infinite use of log space, and avoiding interruptions of business systems .

CN113407503BActive Publication Date: 2024-12-03BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
CN202010180442.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-03-16
Publication Date
2024-12-03
Estimated Expiration
2040-03-16

AI Technical Summary

Technical Problem

The log space used by the InnoDB engine in MySQL database is fixed in size, resulting in interruption of the business system when the log space is completely occupied.

Method used

By determining whether the occupancy rate of the log file reaches the preset ratio, and based on the target LSN determined by the checkpoint event, compare the LSN of the log file, dynamically manage the log space, and add or delete the log file to avoid the space being completely occupied.

Benefits of technology

It effectively avoids the complete occupation of log space, prevents business system interruption, and realizes the infinite use of log space by dynamically managing the number of log files.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113407503B_ABST
    Figure CN113407503B_ABST
Patent Text Reader

Abstract

The embodiments of the present invention provide a log space management method, device, electronic device and medium, which relate to the field of storage technology. The scheme of the present invention includes: for any log space, judging whether the occupancy rate of the first log file reaches a preset ratio to obtain a judgment result, wherein the first log file is the log file in the log space that is currently writing log information; and obtaining the target log sequence number LSN determined by the checkpoint event, the target LSN being the minimum LSN of the log to be restored. Then the log LSN of the log file in the log space is compared with the target LSN in size to determine the second log file from all the log files in the space, and the size of the log space is managed based on the judgment result and the second log file. The use of this method can avoid the problem of business interruption caused by the full occupation of the log space.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a log space management method, device, electronic equipment and medium. Background Art

[0002] At present, the database engine used by MySQL database is InnoDB. In the related technology, log data can be stored through InnoDB. However, the log space used to store log data has a fixed size. When the log space is fully occupied, the business system will not work for a period of time, which will cause business interruption. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a log space management method, device, electronic device and medium to solve the problem of service interruption caused by fixed log space size. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a log space management method, including:

[0005] For any log space, determine whether the occupancy rate of the first log file reaches a preset ratio to obtain a determination result, wherein the first log file is a log file in the log space that is currently writing log information; and obtain a target log sequence number LSN determined by a checkpoint event, wherein the target LSN is the minimum LSN of the log to be restored;

[0006] Compare the log LSN of the log file in the log space with the target LSN in size to determine the second log file from all the log files in the log space;

[0007] The size of the log space is managed based on the determination result and the second log file.

[0008] In a possible implementation, comparing the log LSN of the log file in the log space with the target LSN to determine the second log file from all the log files in the log space includes:

[0009] When it is determined through comparison that all log LSNs in any log file in the log space are smaller than the target LSN, the log file is determined as the second log file.

[0010] In a possible implementation manner, managing the size of the log space based on the determination result and the second log file includes:

[0011] When the judgment result is that the occupancy rate of the first log file reaches a preset ratio, adding a new log file to the log space;

[0012] And, deleting the second log file from the log space.

[0013] In a possible implementation, when the judgment result is that the occupancy rate of the first log file reaches a preset ratio, adding a new log file to the log space includes:

[0014] A new log file is created in the log space, and the new log file is numbered, where the number of the new log file is the maximum number of the log files in the log space plus one.

[0015] In a possible implementation, the log files in the log space are numbered using sixteen-digit hexadecimal numbers.

[0016] In a second aspect, an embodiment of the present application provides a log space management device, including:

[0017] A judgment module is used to judge, for any log space, whether the occupancy rate of a first log file reaches a preset ratio to obtain a judgment result, wherein the first log file is a log file in the log space that is currently writing log information; and obtain a target log sequence number LSN determined by a checkpoint event, wherein the target LSN is a minimum LSN of the log to be restored;

[0018] A comparison module, used for comparing the log LSN of the log file in the log space with the target LSN, so as to determine the second log file from all the log files in the log space;

[0019] A management module is used to manage the size of the log space based on the judgment result and the second log file.

[0020] In a possible implementation, the comparison module is specifically configured to:

[0021] When it is determined through comparison that all log LSNs in any log file in the log space are smaller than the target LSN, the log file is determined as the second log file.

[0022] In a possible implementation, the management module is specifically configured to:

[0023] When the judgment result is that the occupancy rate of the first log file reaches a preset ratio, adding a new log file to the log space;

[0024] And, deleting the second log file from the log space.

[0025] In a possible implementation, the management module is specifically configured to:

[0026] A new log file is created in the log space, and the new log file is numbered, where the number of the new log file is the maximum number of the log files in the log space plus one.

[0027] In a possible implementation, the log files in the log space are numbered using sixteen-digit hexadecimal numbers.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0029] Memory, used to store computer programs;

[0030] The processor is used to implement the method steps described in the first aspect when executing the program stored in the memory.

[0031] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in the first aspect are implemented.

[0032] In a fifth aspect, an embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method steps described in the first aspect above.

[0033] By adopting the above technical solution, for any log space, it is determined whether the occupancy rate of the first log file reaches a preset ratio, and the target LSN determined by the checkpoint event can be obtained, and then the log LSN of the log file in the log space is compared with the target LSN in size, so as to determine the second log file from all the log files in the log space, and then the size of the log space is managed based on the judgment result and the second log file. The above-mentioned first log file is the log file currently being written in the log space, that is, the embodiment of the present application can timely determine whether the log file currently being written reaches a preset ratio, thereby avoiding that all log files in the log space are filled. In addition, the above-mentioned target LSN is the minimum LSN of the log to be restored. In other words, the logs whose log LSN is less than the target LSN are all logs that do not need to be restored, so the embodiment of the present application can determine which log files do not include the logs to be restored by comparing the log LSN of the log file with the target LSN, so as to avoid the deletion of the logs to be restored. In summary, the size management of the log space based on the judgment result and the second log file can avoid the log space being fully occupied and avoid the problem of business interruption of the business system.

[0034] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flow chart of a log space management method provided in an embodiment of the present application;

[0037] Figure 2 An exemplary schematic diagram of a log file allocation method provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of the structure of a log space management device provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, an embodiment of the present application provides a log space management method, which is applied to an electronic device, and the method includes:

[0042] S101. For any log space, determine whether the occupancy rate of the first log file reaches a preset ratio to obtain a determination result; and obtain a target (Log sequence number, LSN) determined by a checkpoint event.

[0043] The first log file is a log file in the log space that is currently writing log information, and the target LSN is the minimum LSN of the log to be restored.

[0044] As an example, the preset ratio may be 50%.

[0045] S102: Compare the log LSN of the log file in the log space with the target LSN to determine the second log file from all the log files in the log space.

[0046] S103: Manage the size of the log space based on the determination result and the second log file.

[0047] The log space management method provided by the embodiment of the present application is adopted. For any log space, it is determined whether the occupancy rate of the first log file reaches a preset ratio, and the target LSN determined by the checkpoint event can be obtained, and then the log LSN of the log file in the log space is compared with the target LSN to determine the second log file from all the log files in the log space, and then the size of the log space is managed based on the judgment result and the second log file. The above-mentioned first log file is the log file currently being written in the log space, that is, the embodiment of the present application can timely determine whether the log file currently being written reaches a preset ratio, thereby avoiding all log files in the log space from being filled. In addition, the above-mentioned target LSN is the minimum LSN of the log to be restored. In other words, the logs whose log LSN is less than the target LSN are all logs that do not need to be restored. Therefore, the embodiment of the present application can determine which log files do not include the logs to be restored by comparing the log LSN of the log file with the target LSN, so as to avoid the deletion of the logs to be restored. In summary, the size management of the log space based on the judgment result and the second log file can avoid the log space from being fully occupied and avoid the problem of business interruption of the business system.

[0048] With respect to the above S101, after the checkpoint event is triggered, the write process of the database will be triggered to write the dirty data in the memory to the hard disk to ensure that the data in the memory and the hard disk are consistent. Specifically, after the checkpoint event is triggered, the write process of the database can write the log after the log corresponding to the checkpoint_LSN in the memory to the hard disk. In the embodiment of the present application, the target LSN determined by the checkpoint event is the checkpoint_LSN.

[0049] In a checkpoint event, the log after the log corresponding to the checkpoint_LSN is inconsistent with the log in the hard disk, that is, the log before the log corresponding to the checkpoint_LSN is data that has been written to the hard disk. Based on this feature of the checkpoint event, the above S102, comparing the log LSN of the log file in the log space with the target LSN to determine the second log file from all the log files in the log space, can be specifically implemented as follows:

[0050] When it is determined through comparison that all log LSNs in any log file in the log space are smaller than the target LSN, the log file is determined as the second log file.

[0051] It is understandable that if all log LSNs of the log file are smaller than the target LSN, it means that the logs included in the log file have been written to the hard disk, so the second log file is a log file that can be deleted from the log space.

[0052] The above S103, managing the size of the log space based on the judgment result and the second log file, can be specifically implemented as follows:

[0053] When the judgment result is that the occupancy rate of the first log file reaches a preset ratio, a new log file is added to the log space; and a second log file is deleted from the log space.

[0054] Adding a new log file in the log space may be specifically implemented as follows: creating a new log file in the log space and numbering the new log file.

[0055] The number of the new log file is the maximum number of the log files in the log space plus one.

[0056] Optionally, the number of the log file in the log space is a sixteen-digit hexadecimal number.

[0057] In one implementation of the embodiment of the present application, the numbers of the log files in the log space are accumulated in order from early to late in terms of creation time, and log information is written in the log files in order from early to late in terms of generation time of the log information.

[0058] Log files are usually numbered starting from 0, such as Figure 2 As shown in the figure, the first log file in the log space is numbered #00000000000000000, the second log file is numbered #0000000000000001, and the third log file is numbered #0000000000000002. When writing log information, the log information is first written to the first log file in the log space. When the first log file is full, the log information is continued to be written to the second log file.

[0059] For example, assuming that the last log file in the current log space is numbered #0000000000000002, after a new log file is created in the log space, the number of the new log file is #0000000000000003.

[0060] In addition, if Figure 2 If all log LSNs of the log file numbered #0000000000000000 are smaller than the target LSN, the log file will be deleted.

[0061] Therefore, after executing the above S103, Figure 2 The first log file in is a deleted log file, the second, third and fourth log files are currently existing log files, and other log files can be generated in the future.

[0062] In one implementation of the embodiment of the present application, the electronic device may determine the second log file according to the target LSN and the size of the log file.

[0063] The size of each log file in the log space is the same. LSN can be considered as the end point of each log in the log space, expressed as a byte offset. For example, if the size of the first log in the log space is 10Kb, the LSN of the log is 10.

[0064] In the embodiment of the present application, the log file to which the target LSN belongs in the log space can be determined based on the quotient obtained by dividing the target LSN by the log file size. For example, the log file size is 1024kb, the target LSN is 1025, and the quotient obtained by dividing 1025 by 1024 is 1, then the target LSN is in the log file numbered #0000000000000001. Since the target LSN is the minimum LSN to be restored, and the log information in the log file is stored in order from early to late in terms of generation time, the log LSNs in the log files before the log file corresponding to #0000000000000001 are all smaller than the target LSN, that is, in the log space, the log files before the log file corresponding to #0000000000000001 are all the second log files, that is, Figure 2 The log file numbered #0000000000000000 is the second log file.

[0065] Using the embodiment of the present application, if the log file number is a 16-bit hexadecimal number, the maximum number of the log file can be 2 to the power of 64. Since the embodiment of the present application can determine the second log file according to the target LSN determined by the checkpoint event, and delete the second log file, thereby releasing storage resources, and when the occupancy rate of the first log file reaches a preset ratio, the remaining storage resources in the log space can be used to create a new log file. That is, the embodiment of the present application can dynamically delete old log files and create new log files. By using the above numbering method to create new log files, it can be considered that 2 to the power of 64 log files can be created, and the log space can be considered infinite, avoiding the problem of business interruption caused by insufficient log space.

[0066] Based on the same technical concept, the embodiment of the present application also provides a log space management device, such as Figure 3 As shown, the device comprises:

[0067] The judgment module 301 is used to judge whether the occupancy rate of the first log file reaches a preset ratio for any log space, so as to obtain a judgment result, wherein the first log file is a log file in the log space that is currently writing log information; and obtain a target log sequence number LSN determined by a checkpoint event, wherein the target LSN is the minimum LSN of the log to be restored;

[0068] A comparison module 302, configured to compare the log LSN of the log file in the log space with the target LSN to determine the second log file from all the log files in the log space;

[0069] The management module 303 is used to manage the size of the log space based on the judgment result and the second log file.

[0070] Optionally, the comparison module 302 is specifically configured to:

[0071] When it is determined through comparison that all log LSNs in any log file in the log space are smaller than the target LSN, the log file is determined as the second log file.

[0072] Optionally, the management module 303 is specifically configured to:

[0073] When the judgment result is that the occupancy rate of the first log file reaches a preset ratio, adding a new log file to the log space;

[0074] And, deleting the second log file from the log space.

[0075] Optionally, the management module 303 is specifically configured to:

[0076] A new log file is created in the log space, and the new log file is numbered, where the number of the new log file is the maximum number of the log files in the log space plus one.

[0077] The log files in the log space are numbered in sixteen-digit hexadecimal numbers.

[0078] The embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0079] Memory 403, used for storing computer programs;

[0080] The processor 401 is used to implement the method steps in the above method embodiment when executing the program stored in the memory 403.

[0081] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0082] The communication interface is used for communication between the above electronic device and other devices.

[0083] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0084] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0085] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above log space management methods are implemented.

[0086] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any log space management method in the above embodiments.

[0087] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0088] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0089] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A log space management method, characterized in that: include: For any log space, determine whether the occupancy rate of the first log file reaches a preset ratio to obtain a determination result, wherein the first log file is a log file in the log space that is currently writing log information; and obtain a target log sequence number LSN determined by a checkpoint event, wherein the checkpoint event is used to write the log after the log corresponding to the target LSN in the memory to the hard disk, wherein the target LSN is the minimum LSN of the log to be restored, and the log before the log corresponding to the target LSN is the data written to the hard disk; Comparing the log LSN of the log file in the log space with the target LSN in size, so as to determine a second log file from all the log files in the log space, where the second log file is a log file that has been written to the hard disk; Managing the size of the log space based on the judgment result and the second log file; The comparing the log LSN of the log file in the log space with the target LSN to determine the second log file from all the log files in the log space includes: When it is determined through comparison that all log LSNs in any log file in the log space are smaller than the target LSN, the log file is determined as the second log file.

2. The method according to claim 1, characterized in that: The managing the size of the log space based on the judgment result and the second log file includes: When the judgment result is that the occupancy rate of the first log file reaches a preset ratio, adding a new log file to the log space; And, deleting the second log file from the log space.

3. The method according to claim 2, characterized in that When the judgment result is that the occupancy rate of the first log file reaches a preset ratio, adding a new log file to the log space includes: A new log file is created in the log space, and the new log file is numbered, where the number of the new log file is the maximum number of the log files in the log space plus one.

4. The method according to claim 3, characterized in that The log files in the log space are numbered in sixteen-digit hexadecimal numbers.

5. A log space management device, characterized in that: include: A judgment module is used to judge, for any log space, whether the occupancy rate of the first log file reaches a preset ratio to obtain a judgment result, wherein the first log file is a log file in the log space that is currently writing log information; and obtain a target log sequence number LSN determined by a checkpoint event, wherein the checkpoint event is used to write the log after the log corresponding to the target LSN in the memory to the hard disk, wherein the target LSN is the minimum LSN of the log to be restored, and the log before the log corresponding to the target LSN is the data written to the hard disk; A comparison module, used for comparing the log LSN of the log file in the log space with the target LSN, so as to determine a second log file from all the log files in the log space, where the second log file is a log file that has been written to the hard disk; A management module, used for managing the size of the log space based on the judgment result and the second log file; The comparison module is specifically configured to determine the log file as the second log file when it is determined through comparison that all log LSNs in any log file in the log space are smaller than the target LSN.

6. The device according to claim 5, characterized in that The management module is specifically used for: When the judgment result is that the occupancy rate of the first log file reaches a preset ratio, adding a new log file to the log space; And, deleting the second log file from the log space.

7. The device according to claim 6, characterized in that The log files in the log space are numbered in sixteen-digit hexadecimal numbers.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 4 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

Citation Information

Patent Citations

  • Data processing method and data processing device

    CN103838521A

  • Internal writing method for file server operating multiple file systems and server

    CN104461382A

  • Method and system for managing log data

    CN105677258A

  • Log file handling method and device

    CN106815124A

  • Log management method and server

    CN107180051A