Database log analysis control method and device, electronic equipment and storage medium

Abstract

The invention discloses a database log analysis control method which comprises the following steps: when the quantity of current service data is a preset quantity, generating a data operation instruction for a queue analysis control table; generating a target data DML log record; receiving a database log analysis request; analyzing each log record in the database log according to the database log analysis request; judging whether the log record is a target data DML log record or not; if yes, obtaining a value of a numeric field in the current DML log record of the target data as a target value; when the target value is a preset numerical value, determining a target log analysis control program according to a preset mapping relation between the preset numerical value and the log analysis control program; and executing the target log analysis control program. According to the invention, automatic operation of the log analysis program and automatic control of the process are realized, and the flexibility and execution efficiency of the log analysis task are improved. The invention further discloses a device for implementing the method, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present invention relates to the technical field of database data processing, and in particular, to a method, device, electronic device, and storage medium for controlling database log analysis. Background Art

[0002] As an operator and user of a database, in database operation and maintenance and performance analysis, log analysis is an important means to promptly discover system problems. However, with the expansion of the database scale, the amount of log data has increased sharply, resulting in a significant increase in the time cost of log analysis. In the prior art, when the log analysis task takes a long time or needs to be terminated in advance, manual intervention is usually required to stop the analysis program. This operation is acceptable in a single-machine environment, but in a distributed or parallel analysis scenario, since multiple sets of log analysis programs are running simultaneously, manually shutting them down one by one is not only inefficient but also prone to omission or misoperation, affecting the overall analysis process. To improve the efficiency of log analysis, the prior art usually adopts a parallel analysis and processing method, that is, deploying multiple log analysis programs to process log data in different sections simultaneously. However, when it is necessary to terminate the analysis task in advance, this method still relies on manual stopping of all analysis processes, which is cumbersome and lacks an automated management mechanism.

[0003] In addition, the prior art has tried to make the log analysis program automatically exit by setting specific conditions (such as analysis progress, keyword matching, etc.). However, due to the complexity and uncertainty of log data, this condition may not be met, resulting in the inability of the analysis program to terminate normally, thereby causing resource waste or the analysis task to hang for a long time. Summary of the Invention

[0004] To solve the above problems existing in the prior art, the present invention provides a method, device, electronic device, and storage medium for controlling database log analysis. The technical problems to be solved by the present invention are achieved through the following technical solutions: The first aspect of the embodiment of the present invention provides a method for controlling database log analysis, including the following steps: When the quantity of current business data is a preset quantity, generate a data operation instruction for the queue analysis control table; wherein, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numerical; Execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein, the data operation instruction instructs to insert a preset value into the queue analysis control table; Receive a database log analysis request; Log analysis step: Analyze each log record in the database log according to the database log analysis request; Determine whether the current log record is a target data DML log record; If not, return to the log analysis step; If so, obtain the value of the numerical field in the current target data DML log record as the target value; When the target value is a preset value, determine the target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program; Execute the target log analysis control program.

[0005] In an embodiment of the present invention, the method further includes: Determine whether the quantity of the current business data reaches a preset quantity; When the quantity of the current business data does not reach the preset quantity, return to the business operation system and continue to generate business data.

[0006] In an embodiment of the present invention, the method further includes: Obtain the queue identifier in the current target data DML log record; The execution of the target log analysis control program includes: Execute the target log analysis control program according to the queue identifier.

[0007] In an embodiment of the present invention, the preset quantity includes a value in one of a plurality of geometric progressions or non-geometric progressions.

[0008] A second aspect of an embodiment of the present invention provides a database log analysis control device, including: A generation module, configured to generate a data operation instruction for the queue analysis control table when the quantity of the current business data is a preset quantity; wherein, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numerical; A storage module, configured to execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein, the data operation instruction instructs to insert a preset value into the queue analysis control table; A receiving module, configured to receive a database log analysis request; An analysis module, configured to perform a log analysis step: analyze each log record in the database log according to the database log analysis request; A judgment module, configured to judge whether the current log record is a target data DML log record; A return module, configured to return to the log analysis step if not; An obtaining module, configured to obtain the value of the numerical field in the current target data DML log record as the target value if so; A determination module, configured to determine a target log analysis control program according to a preset mapping relationship between a preset value and a log analysis control program when the target value is the preset value; An execution module, configured to execute the target log analysis control program.

[0009] In an embodiment of the present invention, the generation module is further configured to determine whether the quantity of current service data reaches a preset quantity; When the quantity of current service data does not reach the preset quantity, return to the service operation system to continue generating service data.

[0010] In an embodiment of the present invention, the acquisition module is further configured to acquire a queue identifier in a current target data DML log record; The execution module is further configured to execute the target log analysis control program according to the queue identifier.

[0011] In an embodiment of the present invention, the preset quantity includes a value in one of a plurality of geometric progressions or non-geometric progressions.

[0012] A third aspect of an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, a database log analysis control method provided in the first aspect of the embodiment of the present invention is implemented.

[0013] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a database log analysis control method provided in the first aspect of the embodiment of the present invention is implemented.

[0014] Advantages of the present invention: By marking the breakpoint of the log analysis program for a custom table, that is, the queue analysis control table, and using the value of the operation on the queue analysis control table to implement the control command for log analysis, the present invention realizes the automatic operation of the log analysis program and the automatic control of the process, improves the flexibility and execution efficiency of the log analysis task, can simply and quickly control the running process of log analysis, and supports the efficient management of large-scale log analysis tasks. In a parallel analysis scenario, all log analysis programs share the same control table to ensure the consistency and reliability of the global termination instruction, and solve the cumbersome problem of manual shutdown in a distributed environment.

[0015] Other features and advantages of the present invention will be set forth in the following description, and in part will be obvious from the description, or may be learned by practice of the present invention. The objectives and other advantages of the present invention may be realized and attained by the structure particularly pointed out in the written description, claims, as well as the drawings.

[0016] The technical solution of the present invention will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0017] The drawings are used to provide a further understanding of the present invention, and constitute a part of the description. They are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. In the drawings: Figure 1 It is a schematic flowchart of a database log analysis control method provided by an embodiment of the present invention; Figure 2 It is a schematic block diagram of a database log analysis control device provided by an embodiment of the present invention. Detailed Embodiments

[0018] The present invention will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present invention are not limited thereto.

[0019] As Figure 1 shown, a database log analysis control method is provided in the first aspect of the embodiment of the present invention, including the following steps: Step 11: When the quantity of the current business data is a preset quantity, generate a data operation instruction for the queue analysis control table.

[0020] Among them, the queue analysis control table is pre-created, and at least one field in the queue analysis control table is of numerical type.

[0021] Step 12: Execute the data operation instruction of the queue analysis control table to generate a target data DML log record. The full name of DML is Data Manipulation Language, which is a data manipulation language.

[0022] Among them, the data operation instruction instructs to insert a preset value into the queue analysis control table.

[0023] Step 13: Receive a database log analysis request.

[0024] Step 14: Log analysis step: Analyze each log record in the database log according to the database log analysis request.

[0025] Step 15: Determine whether the current log record is a target data DML log record.

[0026] Step 16, if not, return to the log analysis step.

[0027] Step 17, if so, obtain the value of the numeric field in the current target data DML log record as the target value.

[0028] Step 18, when the target value is a preset value, determine the target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program.

[0029] Step 19, execute the target log analysis control program.

[0030] In this embodiment, by marking the breakpoint of the log analysis program for the custom table, that is, the queue analysis control table, and using the value of the operation on the queue analysis control table to implement the control command for log analysis, the automatic operation of the log analysis program and the automatic process control are realized, the flexibility and execution efficiency of the log analysis task are improved, the operation process of the log analysis can be simply and quickly controlled, and the efficient management of large-scale log analysis tasks is supported. In the parallel analysis scenario, all log analysis programs share the same control table to ensure the consistency and reliability of the global termination instruction, and solve the cumbersome problem of manual shutdown in the distributed environment.

[0031] Based on the first aspect of the embodiments of the present invention, the second aspect of the embodiments of the present invention further elaborates on a database log analysis control method. It should be noted that the embodiments of the present invention are applied to the operator or user of the database, rather than the developer of the database. The operator or user of the database can operate multiple databases, and the log structures of each database are different. Since the log structure cannot be obtained, a certain number in the log record cannot be used as the end condition for log analysis control. The second aspect of the embodiments of the present invention provides a database log analysis control method, which is applied to a log analysis control device independent of the database system, and includes the following steps: Step 21, determine whether the quantity of the current business data reaches a preset quantity.

[0032] Step 22, when the quantity of the current business data does not reach the preset quantity, return to the business operation system, and the business operation system continues to generate business data.

[0033] Step 23, when the quantity of the current business data is the preset quantity, generate a data operation instruction for the queue analysis control table.

[0034] Among them, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numeric. The preset quantity includes a value in one of multiple geometric progressions or non-geometric progressions.

[0035] In this step, the queue analysis control table is pre-created and stored in the database as a business table of the database. The queue analysis control table can be named with a unique identifier, and DML operations on this table will only be performed when the quantity of current business data meets the preset quantity condition. Whenever the quantity of current business data reaches a certain condition, a data operation instruction for the queue analysis control table will be generated. This operation instruction is to insert a special value into the queue analysis control table.

[0036] Step 24: Execute the data operation instruction of the queue analysis control table, insert a preset value into the queue analysis control table, record the identifier of the current log analysis queue, and generate a target data DML log record. The target data DML log record is stored in the DML log.

[0037] Execute the data operation instruction of the queue analysis control table to generate a target data DML log record. After executing the data operation instruction of the queue analysis control table, a corresponding DML log record will be generated, which is the required target data DML log record. For example, when the quantity of current business data is 5000, the data operation instruction generated for the queue analysis control table is to insert the value 100 into the specified field. After inserting the value, a corresponding DML log record is generated, and then the business system continues to generate business data. Then, when the quantity of current business data is 10000, the value 200 is inserted into the specified field. After inserting the value, a corresponding DML log record is generated, and then the business system continues to generate business data, and so on. Among them, the value inserted into the queue analysis control table each time can be the same or different. The digital form of the specific value inserted and which number to insert each time are pre-set, and each value represents a control action for a log analysis process.

[0038] For example, as shown in Table 1, inserting the preset value 100 into the code field of the queue analysis control table means exiting the log analysis program, and inserting 200 means reducing the log analysis speed, etc.

[0039] Table 1 Step 25: Receive a database log analysis request.

[0040] Step 26: Analyze each log record in the database log according to the database log analysis request. The database log includes DML logs and DDL (Data Definition Language) logs.

[0041] After receiving the database log analysis request, start analyzing each log record in the database log one by one in sequence.

[0042] Step 27: Determine whether the current log record is a target data DML log record.

[0043] While analyzing each log record, determine whether the log record is an operation on the queue analysis control table, that is, whether the log record is a target data DML log record.

[0044] Step 28: If the current log record is not a target data DML log record, return to Step 26 and continue to analyze the next log record.

[0045] Step 29: If the current log record is a target data DML log record, obtain the value of the numeric field in the current target data DML log record as the target value, and obtain the queue identifier in the target data DML log record. The queue identifier is recorded in the target data DML log record and is used to distinguish each log analysis queue.

[0046] When analyzing each log record in the database log, as long as it is determined that the current log record is a target data DML log record, Step 29 can be executed.

[0047] Step 30: When the target value is a preset value, determine the target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program.

[0048] In this step, an example of the preset mapping relationship between the preset value and the log analysis control program is shown in Table 1. Different preset values represent the control actions that different log analysis programs need to complete.

[0049] Step 31: Execute the target log analysis control program according to the queue identifier, return to Step 26 until all database logs are analyzed.

[0050] After determining the target log analysis control program, execute the target log analysis control program on the corresponding log analysis queue according to the queue identifier. If the target value is 100, the current log analysis queue corresponding to the queue identifier terminates and exits the log analysis program, and returns to continue the log analysis of the next queue until the queue analysis of all logs is completed. If the target value is 200, modify the analysis speed parameter of the current log analysis queue, reduce the analysis speed, and then jump back to the breakpoint to continue the analysis in the order of log records at the current analysis speed until the queue analysis of all log records is completed. Therefore, if the target log analysis control program is an abort program, after execution, jump back to the breakpoint to continue the analysis.

[0051] The present invention realizes the automatic operation of log analysis and the automatic control of the analysis process. For multiple parallel log analysis queues (these queues can belong to the same database or multiple different databases), the target log analysis control program can accurately control multiple log analysis queues, achieving automatic and accurate control in scenarios that require a large amount of data processing, avoiding data chaos, and improving the accuracy of control and processing efficiency. In an embodiment of the present invention, database log analysis is to synchronize the source database operations to the target database. If it is analyzed that the current log record is a target data DML log record, it is equivalent to interrupting the current synchronization and executing steps 29 to 31.

[0052] Preferably, the present invention can be applied to the log analysis process of the Yashan database.

[0053] As Figure 2 shown, the third aspect of the embodiment of the present invention provides a control device for guiding database log analysis, including: A generation module 41, configured to generate a data operation instruction for the queue analysis control table when the quantity of the current business data reaches a preset quantity; wherein, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numerical; A storage module 42, configured to execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein, the data operation instruction indicates inserting a preset value into the queue analysis control table; A receiving module 43, configured to receive a database log analysis request; An analysis module 44, configured to perform log analysis steps: analyze each log record in the DML log according to the database log analysis request; A judgment module 45, configured to judge whether the current log record is a target data DML log record; A return module 46, configured to return to the log analysis steps if not; An acquisition module 47, configured to, if so, acquire the value of the numerical field in the current target data DML log record as the target value; A determination module 48, configured to determine the target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program when the target value is the preset value; An execution module 49, configured to execute the target log analysis control program.

[0054] In an embodiment of the present invention, the generation module is further configured to judge whether the quantity of the current business data reaches the preset quantity; When the quantity of the current business data does not reach the preset quantity, return to the business operation system and continue to generate business data.

[0055] In one embodiment of the present invention, the acquisition module is further configured to acquire a queue identifier in a current target data DML log record; The execution module is further configured to execute a target log analysis control program according to the queue identifier.

[0056] In one embodiment of the present invention, the preset quantity includes a value in one of a plurality of geometric progressions or non-geometric progressions.

[0057] A fourth aspect of the embodiments of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the database log analysis control method provided by the embodiments of the present invention as described above.

[0058] A fifth aspect of the embodiments of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the database log analysis control method provided by the embodiments of the present invention as described above.

[0059] Among them, the memory may include a random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0060] The aforementioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processing, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware systems.

[0061] The method provided by the embodiments of the present invention can be applied to an electronic device. Specifically, the electronic device may be: a desktop computer, a portable computer, a smart mobile terminal, a server, etc. This is not limited herein, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0062] For the apparatus / electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For related parts, please refer to the partial description of the method embodiments.

[0063] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0064] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0066] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A database log analysis control method, characterized in that: The following steps are involved: When the amount of current business data is a preset amount, a data operation instruction for a queue analysis control table is generated; wherein the queue analysis control table is created in advance, and the type of at least one field in the queue analysis control table is a numeric type; Execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein the data operation instruction indicates to insert a preset value into the queue analysis control table; Receive database log analysis request; Log analysis step: analyzing each log record in the database log according to the database log analysis request; Determine whether the current log record is the target data DML log record; If not, return to the log analysis step; If yes, obtain the value of the numeric field in the current DML log record of the target data as the target value; When the target value is a preset value, determining a target log analysis control program according to a preset mapping relationship between the preset value and the log analysis control program; Execute the target log analysis control program.

2. The method according to claim 1, characterized in that The method further comprises: Determine whether the amount of current business data reaches a preset amount; When the amount of current business data does not reach the preset amount, return to the business operation system to continue generating business data.

3. The method according to claim 2, characterized in that The method further comprises: Get the queue ID in the current target data DML log record; The executing the target log analysis control program includes: The target log analysis control program is executed according to the queue identifier.

4. The method according to claim 2, characterized in that The preset number includes a value in a plurality of geometric progressions or unequal progressions.

5. A database log analysis control device, characterized in that: include: A generating module, used for generating a data operation instruction for a queue analysis control table when the amount of current business data is a preset amount; wherein the queue analysis control table is created in advance, and the type of at least one field in the queue analysis control table is a numerical type; A storage module, used to execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein the data operation instruction indicates to insert a preset value into the queue analysis control table; A receiving module, used for receiving a database log analysis request; The analysis module is used for the log analysis step: analyzing each log record in the database log according to the database log analysis request; A judgment module is used to judge whether a current log record is a target data DML log record; A return module, used for returning to the log analysis step if no; An acquisition module is used to obtain the value of a numeric field in a current target data DML log record as a target value; A determination module, configured to determine a target log analysis control program according to a preset mapping relationship between the preset value and the log analysis control program when the target value is a preset value; An execution module is used to execute the target log analysis control program.

6. The device according to claim 5, characterized in that The generating module is further used to determine whether the amount of current business data reaches a preset amount; When the amount of current business data does not reach the preset amount, return to the business operation system to continue generating business data.

7. The device according to claim 6, characterized in that The acquisition module is further used to acquire the queue identifier in the current DML log record of the target data; The execution module is also used to execute the target log analysis control program according to the queue identifier.

8. The device according to claim 5, characterized in that The preset number includes a value in a plurality of geometric progressions or unequal progressions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the database log analysis control method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the database log analysis control method according to any one of claims 1 to 4 is implemented.

Images