Database Log Analysis Control Method, Device, Electronic Device and Storage Medium

Abstract

The present invention discloses a method for controlling database log analysis, including: when the quantity of current business data is a preset quantity, generating a data operation instruction for a queue analysis control table; generating a target data DML log record; receiving a database log analysis request; analyzing each log record in the database log according to the database log analysis request; determining whether it is a target data DML log record; if so, obtaining the value of the numeric field in the current target data DML log record as a target value; when the target value is a preset value, determining a target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program; and executing the target log analysis control program. The present invention realizes the automatic operation of the log analysis program and the automatic control of the process, and improves the flexibility and execution efficiency of the log analysis task. The present invention also discloses a device, an electronic device and a storage medium for implementing the above method.

Description

Technical Field

[0001] The present invention relates to the technical field of database data processing, and in particular, to a method, apparatus, electronic device, and storage medium for controlling database log analysis. Background Art

[0002] As an operator and user of a database, in database operation and maintenance and performance analysis, log analysis is an important means to promptly discover system problems. However, with the expansion of the database scale, the amount of log data has increased sharply, resulting in a significant increase in the time cost of log analysis. In the prior art, when the log analysis task takes a long time or needs to be terminated in advance, it is usually necessary to manually intervene to stop the analysis program. This operation is acceptable in a single-machine environment, but in a distributed or parallel analysis scenario, since multiple sets of log analysis programs are running simultaneously, manually shutting them down one by one is not only inefficient but also prone to omission or misoperation, affecting the overall analysis process. To improve the efficiency of log analysis, the prior art usually adopts a parallel analysis and processing method, that is, deploying multiple log analysis programs to process log data in different sections simultaneously. However, when it is necessary to terminate the analysis task in advance, this method still relies on manually stopping all analysis processes, which is cumbersome and lacks an automated management mechanism.

[0003] In addition, the prior art has tried to make the log analysis program automatically exit by setting specific conditions (such as analysis progress, keyword matching, etc.), but due to the complexity and uncertainty of log data, this condition may not be met, resulting in the analysis program being unable to terminate normally, and further causing resource waste or the analysis task being suspended for a long time. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a method, apparatus, electronic device, and storage medium for controlling database log analysis. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0005] The first aspect of the embodiments of the present invention provides a method for controlling database log analysis, including the following steps:

[0006] When the number of current business data is a preset number, generate a data operation instruction for the queue analysis control table; wherein, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numeric;

[0007] Execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein, the data operation instruction indicates inserting a preset value into the queue analysis control table;

[0008] Receive a database log analysis request;

[0009] Log analysis steps: Analyze each log record in the database log according to the database log analysis request;

[0010] Determine whether the current log record is a target data DML log record;

[0011] If not, return to the log analysis step;

[0012] If so, obtain the value of the numerical field in the current target data DML log record as the target value;

[0013] When the target value is a preset value, determine the target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program;

[0014] Execute the target log analysis control program.

[0015] In an embodiment of the present invention, the method further includes:

[0016] Determine whether the quantity of the current business data reaches a preset quantity;

[0017] When the quantity of the current business data does not reach the preset quantity, return to the business operation system and continue to generate business data.

[0018] In an embodiment of the present invention, the method further includes:

[0019] Obtain the queue identifier in the current target data DML log record;

[0020] The execution of the target log analysis control program includes:

[0021] Execute the target log analysis control program according to the queue identifier.

[0022] In an embodiment of the present invention, the preset quantity includes a value in one of a plurality of geometric progressions or non-geometric progressions.

[0023] A second aspect of the embodiments of the present invention provides a database log analysis control device, including:

[0024] A generation module, configured to generate a data operation instruction for the queue analysis control table when the quantity of the current business data is a preset quantity; wherein, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numerical;

[0025] A storage module, configured to execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein, the data operation instruction instructs to insert a preset value into the queue analysis control table;

[0026] A receiving module, configured to receive a database log analysis request;

[0027] An analysis module, configured to perform a log analysis step: analyze each log record in the database log according to the database log analysis request;

[0028] A judgment module, configured to judge whether the current log record is a target data DML log record;

[0029] A return module, configured to, if not, return the log analysis step;

[0030] An obtaining module, configured to, if so, obtain the value of the numerical field in the current target data DML log record as the target value;

[0031] A determination module, configured to, when the target value is a preset value, determine a target log analysis control program according to a preset mapping relationship between the preset value and the log analysis control program;

[0032] An execution module, configured to execute the target log analysis control program.

[0033] In an embodiment of the present invention, the generation module is further configured to judge whether the quantity of the current service data reaches a preset quantity;

[0034] When the quantity of the current service data does not reach the preset quantity, return to the service operation system to continue generating service data.

[0035] In an embodiment of the present invention, the obtaining module is further configured to obtain a queue identifier in the current target data DML log record;

[0036] The execution module is further configured to execute the target log analysis control program according to the queue identifier.

[0037] In an embodiment of the present invention, the preset quantity includes a value in one of a plurality of geometric progressions or non-geometric progressions.

[0038] A third aspect of the embodiments of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements a database log analysis control method provided in the first aspect of the embodiments of the present invention.

[0039] A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, on which a computer program is stored, where when the computer program is executed by a processor, it implements a database log analysis control method provided in the first aspect of the embodiments of the present invention.

[0040] Advantages of the present invention:

[0041] The present invention marks the breakpoint of the log analysis program by operating on a custom table, i.e., the queue analysis control table, and uses the value of the operation on the queue analysis control table to implement the control command for log analysis. It realizes the automatic operation and process automatic control of the log analysis program, improves the flexibility and execution efficiency of the log analysis task, can simply and quickly control the running process of log analysis, and supports the efficient management of large-scale log analysis tasks. In the parallel analysis scenario, all log analysis programs share the same control table to ensure the consistency and reliability of the global termination instruction, and solve the cumbersome problem of manual shutdown in the distributed environment.

[0042] Other features and advantages of the present invention will be described in the following specification, and in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in the written specification, claims, and drawings.

[0043] The technical solution of the present invention will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0044] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the drawings:

[0045] Figure 1 It is a schematic flowchart of a method for controlling database log analysis provided by an embodiment of the present invention;

[0046] Figure 2 It is a schematic block diagram of a device for controlling database log analysis provided by an embodiment of the present invention. Detailed Embodiments

[0047] The present invention will be further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0048] As Figure 1 shown, the first aspect of the embodiment of the present invention provides a method for controlling database log analysis, including the following steps:

[0049] Step 11: Generate a data operation instruction for the queue analysis control table when the quantity of the current service data reaches a preset quantity.

[0050] Among them, the queue analysis control table is pre-created, and at least one field in the queue analysis control table is of numerical type.

[0051] Step 12: Execute the data operation instruction for the queue analysis control table to generate a target data DML log record. The full name of DML is Data Manipulation Language, which is a data manipulation language.

[0052] Among them, the data operation instruction indicates inserting a preset value into the queue analysis control table.

[0053] Step 13: Receive a database log analysis request.

[0054] Step 14: Log analysis step: Analyze each log record in the database log according to the database log analysis request.

[0055] Step 15: Determine whether the current log record is a target data DML log record.

[0056] Step 16: If not, return to the log analysis step.

[0057] Step 17: If so, obtain the value of the numeric field in the current target data DML log record as the target value.

[0058] Step 18: When the target value is the preset value, determine the target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program.

[0059] Step 19: Execute the target log analysis control program.

[0060] In this embodiment, by using a custom table, that is, the queue analysis control table, to mark the breakpoint of the log analysis program, and using the value of the operation on the queue analysis control table to implement the control command for log analysis, the automatic operation of the log analysis program and the automatic process control are realized, the flexibility and execution efficiency of the log analysis task are improved, the log analysis operation process can be simply and quickly controlled, and the efficient management of large-scale log analysis tasks is supported. In a parallel analysis scenario, all log analysis programs share the same control table to ensure the consistency and reliability of the global termination instruction, and solve the cumbersome problem of manual shutdown in a distributed environment.

[0061] Based on the first aspect of the embodiments of the present invention, the second aspect of the embodiments of the present invention further elaborates on a database log analysis control method. It should be noted that the embodiments of the present invention are applied to the operator or user of the database, rather than the developer of the database. The operator or user of the database can operate multiple databases, and the log structures of each database are different. Since the log structure cannot be obtained, a certain number in the log record cannot be used as the end condition for log analysis control. The second aspect of the embodiments of the present invention provides a database log analysis control method, which is applied to a log analysis control device independent of the database system and includes the following steps:

[0062] Step 21, determine whether the quantity of the current business data reaches a preset quantity.

[0063] Step 22, when the quantity of the current business data does not reach the preset quantity, return to the business operation system, and the business operation system continues to generate business data.

[0064] Step 23, when the quantity of the current business data is the preset quantity, generate a data operation instruction for the queue analysis control table.

[0065] Among them, the queue analysis control table is created in advance, and the type of at least one field in the queue analysis control table is numerical. The preset quantity includes a value in one of multiple geometric progressions or non-geometric progressions.

[0066] In this step, the queue analysis control table is created in advance and stored in the database as a business table of the database. The queue analysis control table can be named with a unique identifier, and a DML operation is performed on the table only when the condition that the quantity of the current business data is the preset quantity is met. Whenever the quantity of the current business data reaches a certain condition, a data operation instruction for the queue analysis control table is generated. This operation instruction is to insert a special value into the queue analysis control table.

[0067] Step 24, execute the data operation instruction of the queue analysis control table, insert the preset value into the queue analysis control table and record the identifier of the current log analysis queue, and generate a target data DML log record together. The target data DML log record is stored in the DML log.

[0068] Execute the data operation instruction of the execution queue analysis control table to generate the target data DML log record. After executing the data operation instruction of the execution queue analysis control table, the corresponding DML log record will be generated, that is, the required target data DML log record. For example, when the number of current business data is 5000, the data operation instruction generated for the queue analysis control table is to insert the value 100 into the specified field. After inserting the value, the corresponding DML log record is generated, and then the business system continues to generate business data. Then, when the number of current business data is 10000, insert the value 200 into the specified field. After inserting the value, the corresponding DML log record is generated, and then the business system continues to generate business data, and so on. Among them, the values inserted into the queue analysis control table each time can be the same or different. The digital form of the specific inserted value and which number to insert each time are preset, and each value represents a control action for a log analysis process.

[0069] For example, as shown in Table 1, inserting the preset value 100 into the code field of the queue analysis control table means exiting the log analysis program, and inserting 200 means reducing the log analysis speed, etc.

[0070] Table 1

[0071]

[0072] Step 25, receive the database log analysis request.

[0073] Step 26, according to the database log analysis request, analyze each log record in the database log. The database log includes DML logs and DDL (Data Definition Language) logs.

[0074] After receiving the database log analysis request, start to analyze each log record in the database log one by one in sequence.

[0075] Step 27, determine whether the current log record is the target data DML log record.

[0076] While analyzing each log record, determine whether the log record is an operation on the queue analysis control table, that is, whether the log record is the target data DML log record.

[0077] Step 28, if the current log record is not the target data DML log record, return to Step 26 and continue to analyze the next log record.

[0078] Step 29: If the current log record is a target data DML log record, obtain the value of the numeric field in the current target data DML log record as the target value, and obtain the queue identifier in the target data DML log record. The target data DML log record records the queue identifier, and the queue identifier is used to distinguish each log analysis queue.

[0079] When analyzing each log record of the database log, as long as it is analyzed that the current log record is a target data DML log record, step 29 can be executed.

[0080] Step 30: When the target value is a preset value, determine the target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program.

[0081] In this step, an example of the preset mapping relationship between the preset value and the log analysis control program is shown in Table 1. Different preset values represent the control actions that different log analysis programs need to complete.

[0082] Step 31: Execute the target log analysis control program according to the queue identifier, and return to step 26 until all database logs are analyzed.

[0083] After determining the target log analysis control program, execute the target log analysis control program on the corresponding log analysis queue according to the queue identifier. If the target value is 100, the current log analysis queue corresponding to the queue identifier terminates and exits the log analysis program, and returns to continue starting the log analysis of the next queue until the queue analysis of all logs is completed. If the target value is 200, modify the analysis speed parameter of the current log analysis queue, reduce the analysis speed, and then jump back to the breakpoint to continue the analysis in the order of the log records at the current analysis speed until the queue analysis of all log records is completed. Therefore, if the target log analysis control program is an abort program, after execution, it jumps back to the breakpoint to continue the analysis.

[0084] The present invention realizes the automatic operation of log analysis and the automatic control of the analysis process. For multiple parallel log analysis queues (these queues can belong to the same database or multiple different databases), the target log analysis control program can accurately control multiple log analysis queues, and in scenarios that require a large amount of data processing, it realizes automatic and accurate control, avoids data chaos, and improves the accuracy and processing efficiency of control. In the embodiment of the present invention, database log analysis is to synchronize the source database operation to the target database. If it is analyzed that the current log record is a target data DML log record, it is equivalent to interrupting the current synchronization and executing steps 29 - 31.

[0085] Preferably, the present invention can be applied to the log analysis process of the Yashan database.

[0086] As Figure 2 shown, a third aspect of an embodiment of the present invention provides a guiding database log analysis control device, including:

[0087] A generation module 41, configured to generate a data operation instruction for a queue analysis control table when the quantity of current service data reaches a preset quantity; wherein, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numerical;

[0088] A storage module 42, configured to execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein, the data operation instruction instructs to insert a preset numerical value into the queue analysis control table;

[0089] A receiving module 43, configured to receive a database log analysis request;

[0090] An analysis module 44, configured to perform a log analysis step: analyze each log record in the DML log according to the database log analysis request;

[0091] A judgment module 45, configured to judge whether the current log record is a target data DML log record;

[0092] A return module 46, configured to return to the log analysis step if not;

[0093] An obtaining module 47, configured to obtain the value of the numerical field in the current target data DML log record as a target value if so;

[0094] A determination module 48, configured to determine a target log analysis control program according to a preset mapping relationship between the preset numerical value and the log analysis control program when the target value is the preset numerical value;

[0095] An execution module 49, configured to execute the target log analysis control program.

[0096] In an embodiment of the present invention, the generation module is further configured to judge whether the quantity of current service data reaches the preset quantity;

[0097] When the quantity of current service data does not reach the preset quantity, return to the service operation system to continue generating service data.

[0098] In an embodiment of the present invention, the obtaining module is further configured to obtain the queue identifier in the current target data DML log record;

[0099] The execution module is further configured to execute the target log analysis control program according to the queue identifier.

[0100] In one embodiment of the present invention, the preset quantity includes a numerical value in one of a plurality of geometric progressions or non-geometric progressions.

[0101] A fourth aspect of the embodiments of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the database log analysis control method provided by the embodiments of the present invention as described above.

[0102] A fifth aspect of the embodiments of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, it implements the steps of the database log analysis control method provided by the embodiments of the present invention as described above.

[0103] Among them, the memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0104] The aforementioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware systems.

[0105] The method provided by the embodiments of the present invention can be applied to an electronic device. Specifically, the electronic device may be: a desktop computer, a portable computer, a smart mobile terminal, a server, etc. This is not limited herein. Any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0106] For the device / electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For related parts, please refer to the partial description of the method embodiments.

[0107] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general purpose computers, special purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in the flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0108] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means for implementing the functions specified in the flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0110] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A database log analysis control method, characterized in that, It includes the following steps: When the quantity of current business data is a preset quantity, generate a data operation instruction for the queue analysis control table; wherein, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numeric; Execute the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein, the data operation instruction indicates inserting a preset value into the queue analysis control table; Receive a database log analysis request; Log analysis step: Analyze each log record in the database log according to the database log analysis request; Judge whether the current log record is a target data DML log record; If not, return to the log analysis step; If so, obtain the value of the numeric field in the current target data DML log record as the target value; When the target value is a preset value, determine a target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program; Execute the target log analysis control program.

2. The method according to claim 1, characterized in that The method further includes: Judge whether the quantity of current business data reaches the preset quantity; When the quantity of current business data does not reach the preset quantity, return to the business operation system and continue to generate business data.

3. The method according to claim 2, wherein The method further includes: Obtain the queue identifier in the current target data DML log record; The execution of the target log analysis control program includes: Execute the target log analysis control program according to the queue identifier.

4. The method according to claim 2, wherein The preset quantity includes a value in one of multiple geometric progressions or non-geometric progressions.

5. A database log analysis control device, characterized in that, It includes: A generation module, used for generating a data operation instruction for the queue analysis control table when the quantity of current business data is a preset quantity; wherein, the queue analysis control table is pre-created, and the type of at least one field in the queue analysis control table is numeric; A storage module, used for executing the data operation instruction of the queue analysis control table to generate a target data DML log record; wherein, the data operation instruction indicates inserting a preset value into the queue analysis control table; A receiving module, used for receiving a database log analysis request; An analysis module, used for the log analysis step: Analyze each log record in the database log according to the database log analysis request; A judgment module, used for judging whether the current log record is a target data DML log record; A return module, used for returning to the log analysis step if not; An obtaining module, used for obtaining the value of the numeric field in the current target data DML log record as the target value if so; A determination module, used for determining a target log analysis control program according to the preset mapping relationship between the preset value and the log analysis control program when the target value is a preset value; An execution module, used for executing the target log analysis control program.

6. The device according to claim 5, characterized in that, The generation module is further used for judging whether the quantity of current business data reaches the preset quantity; When the quantity of current business data does not reach the preset quantity, return to the business operation system and continue to generate business data.

7. The device according to claim 6, characterized in that, The obtaining module is further configured to obtain a queue identifier in a current target data DML log record; The execution module is further configured to execute the target log analysis control program according to the queue identifier.

8. The device according to claim 5, characterized in that The preset quantity includes a value in a plurality of geometric progressions or non-geometric progressions.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the database log analysis control method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the database log analysis control method according to any one of claims 1 to 4.

Images