Method, device, equipment and medium for concurrently writing logs into a database

Abstract

The present invention relates to the technical field of data processing in finance and medical health, and provides a method, apparatus, device and medium for concurrently writing logs into a database. The method, apparatus, equipment and medium can utilize a daemon process in a target database to create multiple parallel pipelines and log processes corresponding to each pipeline. The creation of parallel pipelines and log processes improves the log writing speed in high-concurrency scenarios and facilitates the expansion of new pipelines and log processes. Each log process is started in sequence to monitor the corresponding pipeline in parallel. When any log process monitors that a target log is sent to the corresponding pipeline by another process, the target log is written into a target database by utilizing any log process. Since each log process monitors the corresponding pipeline separately and reads the log from the corresponding pipeline and writes it into the database, there is no need to compete for the same resource lock, which effectively improves system performance and enables efficient execution of log writing operations in high-concurrency scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of financial and medical health data processing, and in particular to a method, apparatus, device and medium for concurrently writing logs into a database. Background Art

[0002] Currently, both the financial and healthcare sectors require high-concurrency log writes to databases. For example, in finance, especially for online transactions, peak trading hours generate a large number of transaction logs, which need to be written to the database simultaneously to ensure transaction traceability and system stability. In healthcare, with the widespread adoption of online consultations and electronic medical records, the medical process generates a large amount of log information. For example, during peak hospital hours, due to the high patient flow, medical staff from various departments are operating the system simultaneously, generating a large amount of log data that needs to be written to the database.

[0003] However, in existing database systems, log writing is typically a serialized operation, requiring all log data to be processed and written by a single log process. This single-process log writing approach can easily become a performance bottleneck in high-concurrency scenarios, especially when processing large volumes of transactions and log data. This can lead to degraded database performance and even cause delays and blockages.

[0004] Specifically, the existing technology has the following deficiencies:

[0005] 1. Performance bottleneck: Single-process log writing cannot fully utilize the computing power of multi-core processors, resulting in log writing speed being unable to keep up with the transaction processing speed. This problem is particularly prominent in high-concurrency scenarios.

[0006] 2. Lock contention: All log writing operations need to compete for the same resource lock (such as the pipeline read-write lock). This increases the lock contention overhead and further degrades system performance.

[0007] 3. Poor scalability: Under high-load scenarios, single-process log writing cannot be expanded to meet the needs of large-scale transaction processing, resulting in limited overall database throughput.

[0008] Therefore, existing technologies have obvious deficiencies in log writing performance and scalability in high-concurrency scenarios. Summary of the Invention

[0009] In view of the above, it is necessary to provide a method, device, equipment and medium for concurrently writing logs to a database, aiming to solve the problems of low efficiency, low performance and poor scalability when writing logs to a database in high-concurrency scenarios.

[0010] A method for concurrently writing logs into a database, the method comprising:

[0011] In response to a concurrent log write instruction for a target database, a daemon process in the target database is used to create a plurality of parallel pipelines and a log process corresponding to each pipeline;

[0012] Start each log process in turn, and use each log process to monitor the corresponding pipeline in parallel;

[0013] Utilizing other processes in the target database to generate logs, and sending the generated logs to the multiple pipelines;

[0014] When any log process monitors that a target log is sent to a corresponding pipeline, the target log is written into the target database using the any log process.

[0015] A device for concurrently writing logs into a database, the device comprising:

[0016] a creation unit, configured to, in response to a concurrent log write instruction to a target database, create a plurality of parallel pipelines and a log process corresponding to each pipeline by using a daemon process in the target database;

[0017] The startup unit is used to start each log process in sequence and use each log process to monitor the corresponding pipeline in parallel;

[0018] a sending unit, configured to generate logs by using other processes in the target database, and send the generated logs to the multiple pipelines;

[0019] The writing unit is configured to write the target log into the target database by using any log process when any log process monitors that a target log is sent to a corresponding pipeline.

[0020] A computer device, comprising:

[0021] a memory storing at least one instruction; and

[0022] The processor executes the instructions stored in the memory to implement the method of concurrently writing logs into the database.

[0023] A computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the method for concurrently writing logs into a database.

[0024] It can be seen from the above technical solution that the present invention can use the daemon process in the target database to create multiple parallel pipelines and log processes corresponding to each pipeline. The creation of parallel pipelines and log processes improves the log writing speed in high-concurrency scenarios and facilitates the expansion of new pipelines and log processes; each log process is started in sequence, and each log process is used to monitor the corresponding pipeline in parallel, and other processes in the target database are used to generate logs and send the generated logs to the corresponding pipeline. Furthermore, when any log process monitors that a target log is sent to the corresponding pipeline, any log process is used to write the target log to the target database. Since each log process monitors the corresponding pipeline respectively and reads the log from the corresponding pipeline and writes it to the database, there is no need to compete for the same resource lock, which effectively improves the system performance and can still efficiently perform log writing operations in high-concurrency scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of a preferred embodiment of the method for concurrently writing logs into a database of the present invention;

[0026] Figure 2 This is a functional module diagram of a preferred embodiment of the device for concurrently writing logs into a database according to the present invention;

[0027] Figure 3 The figure is a schematic diagram of the structure of a computer device according to a preferred embodiment of the method for concurrently writing logs into a database according to the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the method for concurrently writing logs into a database according to the present invention. The order of the steps in the flow chart can be changed according to different requirements, and some steps can be omitted.

[0030] The method for concurrently writing logs into a database is applied to one or more computer devices, wherein the computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0031] The computer device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0032] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0033] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0034] Among them, Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0035] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0036] The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0037] S10 , in response to a concurrent log write instruction to a target database, creating a plurality of parallel pipelines and a log process corresponding to each pipeline by using a daemon process in the target database.

[0038] In this embodiment, the target database may be a database with a log writing function, such as a relational database or a non-relational database.

[0039] For example, in a financial scenario, the target database may be a log writing database for an online transaction system, a payment and clearing system, and a risk management system; in a healthcare scenario, the target database may be a log writing database for a hospital information system, a medical IoT device data acquisition system, a telemedicine platform, etc.

[0040] In this embodiment, the concurrent log writing instruction can be automatically triggered after the target database is deployed, so as to achieve full-process optimization of the log writing process of the target database.

[0041] In this embodiment, the step of creating multiple parallel pipelines and a log process corresponding to each pipeline by using the daemon process in the target database includes:

[0042] Get the number of configured log processes;

[0043] Using the daemon process to create the pipeline according to the number of log processes, and configuring the pipeline number of each pipeline according to the pipeline creation order;

[0044] The daemon process is used to create a log process corresponding to each pipeline, and the pipeline number of each pipeline is sent to the corresponding log process.

[0045] The log process data may be configured according to the demand data in the actual scenario. For example, the number of log processes may be comprehensively configured according to the data scale and system performance in the actual scenario.

[0046] The daemon process may create the pipes sequentially based on the number of log processes, starting from 0 as the pipe number. For example, when the number of log processes is N (N is a positive integer), N pipes are created, and the pipe numbers are incremented from 0 to (N-1).

[0047] Through the above embodiment, multiple parallel pipelines and log processes corresponding to each pipeline can be created. Since the pipelines are in a parallel relationship, execution conditions can be provided for the subsequent parallel writing of logs.

[0048] In this embodiment, after creating multiple parallel pipelines and a log process corresponding to each pipeline using the daemon process in the target database, the method further includes:

[0049] When the number of the log processes increases, obtaining the increase amount of the log processes;

[0050] The daemon process is used to create new pipelines and log processes corresponding to each new pipeline according to the increase.

[0051] In the above embodiment, due to the parallel relationship between pipelines, when there is a demand for new pipelines, a new pipeline and a corresponding log process can be flexibly created directly according to the new demand for the pipeline, which fully utilizes the computing power of the multi-core processor and can effectively solve the problem that the log writing mode of a single process cannot be expanded in high-load scenarios, meet the needs of large-scale transaction processing, and improve the overall throughput of the database system.

[0052] In this embodiment, before using the daemon process in the target database to create multiple parallel pipelines and a log process corresponding to each pipeline, the method further includes:

[0053] Detect whether there is a log file directory in the target database;

[0054] When it is detected that the target database does not have the log file directory, creating the log file directory by using the daemon process;

[0055] The log file directory is used to store log files created by each log process.

[0056] When it is detected that the target database has the log file directory, the existing log file directory can be directly used to store the log files created by each log process.

[0057] Through the above embodiments, log files can be stored in a unified manner, thereby facilitating subsequent management and query of the log files.

[0058] S11, start each log process in sequence, and use each log process to monitor the corresponding pipeline in parallel.

[0059] In this embodiment, starting each log process in sequence includes:

[0060] For each log process, when the log process receives the pipe number sent by the daemon process, the pipe number is stored in the memory, a log file is created according to the pipe number, and the log file switching time is configured;

[0061] The log file number of the log file is the same as the pipeline number.

[0062] For example, when the pipe number is 0, the log file number of the log file corresponding to the pipe numbered 0 is also 0.

[0063] After one log process is started, the next log process is started according to the log file number.

[0064] For example, after the log process corresponding to the pipe numbered 0 is started, the log process corresponding to the pipe numbered 1 is started again until all log processes are started.

[0065] The log file switching time is used to limit the size of the log file, and the log file switching time can be comprehensively configured according to parameters such as system performance and retrieval efficiency.

[0066] For example, the log file switching time can be configured to be every hour, every day, etc.

[0067] By configuring the log file switching time, it is possible to effectively avoid the log file being too large and affecting the normal log writing and subsequent retrieval efficiency.

[0068] Through the above embodiment, the log process corresponding to each pipeline can be started in order, so that the logs in each pipeline can be effectively monitored by using each log process.

[0069] In this embodiment, each log process can be used to monitor the pipe reading end of the corresponding pipe in parallel to wait for other processes to send logs to the pipe.

[0070] Each pipe includes a pipe read end and a pipe write end.

[0071] The pipeline reading end is used for the corresponding log process to monitor whether there is a log written into the corresponding pipeline, and read the log from the pipeline reading end.

[0072] The pipe write end is used for other processes to send logs to the corresponding pipe.

[0073] S12: Generate logs using other processes in the target database, and send the generated logs to the multiple pipelines.

[0074] In this embodiment, sending the generated logs to the multiple pipelines includes:

[0075] Obtain the process ID of each other process and the number of log processes;

[0076] Calculate the quotient of each process number and the number of log processes to obtain the quotient value corresponding to each other process;

[0077] Perform a modulo operation on each quotient value to obtain the target value corresponding to each other process;

[0078] Match each target value as a pipe number from said pipeline to the target pipe corresponding to each other process;

[0079] Send the logs generated by each other process to the corresponding target pipe.

[0080] For example, when the remainder of the quotient of the process number of other process X and the number of log processes is 3, the logs generated by other process X are sent to the pipe with pipe number 3.

[0081] Through the above embodiment, the generated logs can be written into the corresponding pipeline in an orderly manner according to unified rules, avoiding data confusion and loss problems.

[0082] S13: When any log process monitors that a target log is sent to a corresponding pipeline, the target log is written into the target database using the any log process.

[0083] In this embodiment, writing the target log into the target database by using the arbitrary log process includes:

[0084] Waking up the arbitrary log process and using the arbitrary log process to detect whether the configuration file needs to be reloaded;

[0085] When it is detected that the configuration file needs to be reloaded, the updated configuration file is loaded as the target configuration file;

[0086] Detecting whether the log file switching time stored in the target configuration file changes;

[0087] When a change in the log file switching time stored in the target configuration file is detected, the log file switching time stored in the target configuration file is determined as the target log file switching time, and the original log file switching time of any log process is configured as the target log file switching time;

[0088] Get the current timestamp;

[0089] When the current timestamp is the same as or after the target log file switching time, closing the original log file corresponding to the arbitrary log process by using the arbitrary log process, and creating a new log file as the target log file; wherein the target log file and the original log file have the same log file number, and the target log file and the original log file are distinguished according to the creation time;

[0090] The target log in the corresponding pipeline is read by using the arbitrary log process, and the target log is written into the target log file.

[0091] Among them, after starting each log process, in order to save system resources, the log process is in a dormant state when no log is monitored being sent to the corresponding pipeline. It will wake up the corresponding log process after monitoring a log being sent to the corresponding pipeline.

[0092] The configuration file is used to store information such as the log file switching time, database connection information, log parameters such as log format, and communication configuration between processes.

[0093] When the log writing requirements are updated according to changes in the application environment, data scale, etc., the configuration file will also be updated synchronously.

[0094] After the original log file corresponding to the arbitrary log process is closed by the arbitrary log process, log writing to the original log file is stopped, and the current file size is kept unchanged, thereby avoiding the influence of the subsequent retrieval efficiency due to excessive file size.

[0095] At the same time, a new log file is created as the target log file to avoid affecting subsequent log file writing operations.

[0096] Among them, if it is detected that the configuration file does not need to be reloaded, or the configuration file needs to be reloaded but the log file switching time stored in the target configuration file has not changed, the original log file switching time can be directly used as a detection condition to determine whether to create a new log file.

[0097] If the current timestamp is before the target log file switching time, the target log in the corresponding pipeline is read using the arbitrary log process, and the target log is written into the original log file.

[0098] After the target log is written into the corresponding log file, the corresponding pipeline is continuously monitored by the arbitrary log process, and the process is awakened after waiting for other log processes to send log data.

[0099] Through the above-described embodiment, parallel logging of multiple pipelines by each log process can be achieved, significantly improving the database's log writing performance. This is particularly effective in high-concurrency scenarios, effectively reducing log writing delays and blocking issues. Furthermore, the independent log file and pipeline design reduces lock contention, thereby reducing system overhead and further improving database performance. Furthermore, the rational coordination and communication between processes ensures the consistency and integrity of log data, avoiding data confusion and loss.

[0100] This embodiment can be applied to high-concurrency log writing database tasks in financial and healthcare scenarios.

[0101] For example: For financial scenarios, such as large securities trading platforms, thousands or even tens of thousands of transactions may be processed per second during peak trading hours, which means that an equal amount or even more log data needs to be written to the database at the same time every second to ensure the traceability of the transaction and data integrity. For example: For a stock trading system, in the continuous bidding phase after the end of the morning call auction, due to the concentrated orders placed by a large number of investors, there will be instantaneous high-concurrency transaction requests, and correspondingly high-concurrency log writing tasks will be generated, requiring the database to be able to quickly and accurately record the details of each transaction. At this time, if the log concurrent writing database solution of this embodiment is adopted, the concurrent writing of a large number of transaction logs can be achieved by combining the interaction of pipelines, log processes and other processes, thereby improving the efficiency of log writing to the database and avoiding log writing delays and blocking problems in high-concurrency scenarios.

[0102] For example, in healthcare scenarios, telemedicine platforms allow doctors to provide remote patients with medical services over the internet, including online consultations, remote diagnosis, and remote monitoring. During the telemedicine process, the platform needs to record detailed log information, such as video call records between doctors and patients, consultation content, diagnosis results, prescriptions issued, and patient health data transmission records. These logs not only help doctors track and manage patients' conditions but also provide evidence in case of medical disputes. With the increasing popularity of telemedicine, patients may frequently use the platform for consultations during specific times, such as evenings or weekends. Consider a large telemedicine platform serving patients in multiple regions. On a given evening, between 7:00 and 9:00 PM, hundreds of patients may simultaneously have video consultations with doctors. Each video call establishment and termination, as well as any information exchanged during the consultation, generates corresponding log records, which must be promptly written to the database. Furthermore, when patients upload health data using remote monitoring devices, a large amount of data transmission logs is generated. Consequently, during peak usage periods on the telemedicine platform, a high level of concurrent log writing to the database occurs. At this time, if the log concurrent writing database solution of this embodiment is adopted, the concurrent writing of a large number of logs to the telemedicine platform can be achieved by combining the interaction of pipelines, log processes and other processes, thereby improving the efficiency of log writing to the database and avoiding log writing delays and blocking problems in high concurrency scenarios.

[0103] It can be seen from the above technical solution that the present invention can use the daemon process in the target database to create multiple parallel pipelines and log processes corresponding to each pipeline. The creation of parallel pipelines and log processes improves the log writing speed in high-concurrency scenarios and facilitates the expansion of new pipelines and log processes; each log process is started in sequence, and each log process is used to monitor the corresponding pipeline in parallel, and other processes in the target database are used to generate logs and send the generated logs to the corresponding pipeline. Furthermore, when any log process monitors that a target log is sent to the corresponding pipeline, any log process is used to write the target log to the target database. Since each log process monitors the corresponding pipeline respectively and reads the log from the corresponding pipeline and writes it to the database, there is no need to compete for the same resource lock, which effectively improves the system performance and can still efficiently perform log writing operations in high-concurrency scenarios.

[0104] like Figure 2 , which is a functional module diagram of a preferred embodiment of the device for concurrently writing logs to a database according to the present invention. The device 11 for concurrently writing logs to a database includes a creation unit 110, a startup unit 111, a sending unit 112, and a writing unit 113. The modules / units referred to in the present invention refer to a series of computer program segments that can be executed by a processor and can perform fixed functions, and are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0105] The creation unit 110 is configured to create a plurality of parallel pipelines and a log process corresponding to each pipeline by using a daemon process in the target database in response to a concurrent log write instruction to the target database.

[0106] In this embodiment, the target database may be a database with a log writing function, such as a relational database or a non-relational database.

[0107] For example, in a financial scenario, the target database may be a log writing database for an online transaction system, a payment and clearing system, and a risk management system; in a healthcare scenario, the target database may be a log writing database for a hospital information system, a medical IoT device data acquisition system, a telemedicine platform, etc.

[0108] In this embodiment, the concurrent log writing instruction can be automatically triggered after the target database is deployed, so as to achieve full-process optimization of the log writing process of the target database.

[0109] In this embodiment, the creation unit 110 uses the daemon process in the target database to create multiple parallel pipelines and a log process corresponding to each pipeline, including:

[0110] Get the number of configured log processes;

[0111] Using the daemon process to create the pipeline according to the number of log processes, and configuring the pipeline number of each pipeline according to the pipeline creation order;

[0112] The daemon process is used to create a log process corresponding to each pipeline, and the pipeline number of each pipeline is sent to the corresponding log process.

[0113] The log process data may be configured according to the demand data in the actual scenario. For example, the number of log processes may be comprehensively configured according to the data scale and system performance in the actual scenario.

[0114] The daemon process may create the pipes sequentially based on the number of log processes, starting from 0 as the pipe number. For example, when the number of log processes is N (N is a positive integer), N pipes are created, and the pipe numbers are incremented from 0 to (N-1).

[0115] Through the above embodiment, multiple parallel pipelines and log processes corresponding to each pipeline can be created. Since the pipelines are in a parallel relationship, execution conditions can be provided for the subsequent parallel writing of logs.

[0116] In this embodiment, the creation unit 110 uses the daemon process in the target database to create multiple parallel pipelines and log processes corresponding to each pipeline, and when the number of the log processes increases, obtains the increase amount of the log processes;

[0117] The daemon process is used to create new pipelines and log processes corresponding to each new pipeline according to the increase.

[0118] In the above embodiment, due to the parallel relationship between pipelines, when there is a demand for new pipelines, a new pipeline and a corresponding log process can be flexibly created directly according to the new demand for the pipeline, which fully utilizes the computing power of the multi-core processor and can effectively solve the problem that the log writing mode of a single process cannot be expanded in high-load scenarios, meet the needs of large-scale transaction processing, and improve the overall throughput of the database system.

[0119] In this embodiment, the creation unit 110 detects whether there is a log file directory in the target database before using the daemon process in the target database to create multiple parallel pipelines and the log process corresponding to each pipeline;

[0120] When it is detected that the target database does not have the log file directory, creating the log file directory by using the daemon process;

[0121] The log file directory is used to store log files created by each log process.

[0122] When it is detected that the target database has the log file directory, the existing log file directory can be directly used to store the log files created by each log process.

[0123] Through the above embodiments, log files can be stored in a unified manner, thereby facilitating subsequent management and query of the log files.

[0124] The starting unit 111 is used to start each log process in sequence and use each log process to monitor the corresponding pipeline in parallel.

[0125] In this embodiment, the starting unit 111 starts each log process in sequence, including:

[0126] For each log process, when the log process receives the pipe number sent by the daemon process, the pipe number is stored in the memory, a log file is created according to the pipe number, and the log file switching time is configured;

[0127] The log file number of the log file is the same as the pipeline number.

[0128] For example, when the pipe number is 0, the log file number of the log file corresponding to the pipe numbered 0 is also 0.

[0129] After one log process is started, the next log process is started according to the log file number.

[0130] For example, after the log process corresponding to the pipe numbered 0 is started, the log process corresponding to the pipe numbered 1 is started again until all log processes are started.

[0131] The log file switching time is used to limit the size of the log file, and the log file switching time can be comprehensively configured according to parameters such as system performance and retrieval efficiency.

[0132] For example, the log file switching time can be configured to be every hour, every day, etc.

[0133] By configuring the log file switching time, it is possible to effectively avoid the log file being too large and affecting the normal log writing and subsequent retrieval efficiency.

[0134] Through the above embodiment, the log process corresponding to each pipeline can be started in order, so that the logs in each pipeline can be effectively monitored by using each log process.

[0135] In this embodiment, each log process can be used to monitor the pipe reading end of the corresponding pipe in parallel to wait for other processes to send logs to the pipe.

[0136] Each pipe includes a pipe read end and a pipe write end.

[0137] The pipeline reading end is used for the corresponding log process to monitor whether there is a log written into the corresponding pipeline, and read the log from the pipeline reading end.

[0138] The pipe write end is used for other processes to send logs to the corresponding pipe.

[0139] The sending unit 112 is configured to generate logs using other processes in the target database and send the generated logs to the multiple pipelines.

[0140] In this embodiment, the sending unit 112 sends the generated logs to the multiple pipelines including:

[0141] Obtain the process ID of each other process and the number of log processes;

[0142] Calculate the quotient of each process number and the number of log processes to obtain the quotient value corresponding to each other process;

[0143] Perform a modulo operation on each quotient value to obtain the target value corresponding to each other process;

[0144] Match each target value as a pipe number from said pipeline to the target pipe corresponding to each other process;

[0145] Send the logs generated by each other process to the corresponding target pipe.

[0146] For example, when the remainder of the quotient of the process number of other process X and the number of log processes is 3, the logs generated by other process X are sent to the pipe with pipe number 3.

[0147] Through the above embodiment, the generated logs can be written into the corresponding pipeline in an orderly manner according to unified rules, avoiding data confusion and loss problems.

[0148] The writing unit 113 is configured to write the target log into the target database using any log process when any log process monitors that a target log is sent to a corresponding pipeline.

[0149] In this embodiment, the writing unit 113 writes the target log into the target database using the arbitrary log process, including:

[0150] Waking up the arbitrary log process and using the arbitrary log process to detect whether the configuration file needs to be reloaded;

[0151] When it is detected that the configuration file needs to be reloaded, the updated configuration file is loaded as the target configuration file;

[0152] Detecting whether the log file switching time stored in the target configuration file changes;

[0153] When a change in the log file switching time stored in the target configuration file is detected, the log file switching time stored in the target configuration file is determined as the target log file switching time, and the original log file switching time of any log process is configured as the target log file switching time;

[0154] Get the current timestamp;

[0155] When the current timestamp is the same as or after the target log file switching time, closing the original log file corresponding to the arbitrary log process by using the arbitrary log process, and creating a new log file as the target log file; wherein the target log file and the original log file have the same log file number, and the target log file and the original log file are distinguished according to the creation time;

[0156] The target log in the corresponding pipeline is read by using the arbitrary log process, and the target log is written into the target log file.

[0157] Among them, after starting each log process, in order to save system resources, the log process is in a dormant state when no log is monitored being sent to the corresponding pipeline. It will wake up the corresponding log process after monitoring a log being sent to the corresponding pipeline.

[0158] The configuration file is used to store information such as the log file switching time, database connection information, log parameters such as log format, and communication configuration between processes.

[0159] When the log writing requirements are updated according to changes in the application environment, data scale, etc., the configuration file will also be updated synchronously.

[0160] After the original log file corresponding to the arbitrary log process is closed by the arbitrary log process, log writing to the original log file is stopped, and the current file size is kept unchanged, thereby avoiding the influence of the subsequent retrieval efficiency due to excessive file size.

[0161] At the same time, a new log file is created as the target log file to avoid affecting subsequent log file writing operations.

[0162] Among them, if it is detected that the configuration file does not need to be reloaded, or the configuration file needs to be reloaded but the log file switching time stored in the target configuration file has not changed, the original log file switching time can be directly used as a detection condition to determine whether to create a new log file.

[0163] If the current timestamp is before the target log file switching time, the target log in the corresponding pipeline is read using the arbitrary log process, and the target log is written into the original log file.

[0164] After the target log is written into the corresponding log file, the corresponding pipeline is continuously monitored by the arbitrary log process, and the process is awakened after waiting for other log processes to send log data.

[0165] Through the above-described embodiment, parallel logging of multiple pipelines by each log process can be achieved, significantly improving the database's log writing performance. This is particularly effective in high-concurrency scenarios, effectively reducing log writing delays and blocking issues. Furthermore, the independent log file and pipeline design reduces lock contention, thereby reducing system overhead and further improving database performance. Furthermore, the rational coordination and communication between processes ensures the consistency and integrity of log data, avoiding data confusion and loss.

[0166] This embodiment can be applied to high-concurrency log writing database tasks in financial and healthcare scenarios.

[0167] For example: For financial scenarios, such as large securities trading platforms, thousands or even tens of thousands of transactions may be processed per second during peak trading hours, which means that an equal amount or even more log data needs to be written to the database at the same time every second to ensure the traceability of the transaction and data integrity. For example: For a stock trading system, in the continuous bidding phase after the end of the morning call auction, due to the concentrated orders placed by a large number of investors, there will be instantaneous high-concurrency transaction requests, and correspondingly high-concurrency log writing tasks will be generated, requiring the database to be able to quickly and accurately record the details of each transaction. At this time, if the log concurrent writing database solution of this embodiment is adopted, the concurrent writing of a large number of transaction logs can be achieved by combining the interaction of pipelines, log processes and other processes, thereby improving the efficiency of log writing to the database and avoiding log writing delays and blocking problems in high-concurrency scenarios.

[0168] For example, in healthcare scenarios, telemedicine platforms allow doctors to provide remote patients with medical services over the internet, including online consultations, remote diagnosis, and remote monitoring. During the telemedicine process, the platform needs to record detailed log information, such as video call records between doctors and patients, consultation content, diagnosis results, prescriptions issued, and patient health data transmission records. These logs not only help doctors track and manage patients' conditions but also provide evidence in case of medical disputes. With the increasing popularity of telemedicine, patients may frequently use the platform for consultations during specific times, such as evenings or weekends. Consider a large telemedicine platform serving patients in multiple regions. On a given evening, between 7:00 and 9:00 PM, hundreds of patients may simultaneously have video consultations with doctors. Each video call establishment and termination, as well as any information exchanged during the consultation, generates corresponding log records, which must be promptly written to the database. Furthermore, when patients upload health data using remote monitoring devices, a large amount of data transmission logs is generated. Consequently, during peak usage periods on the telemedicine platform, a high level of concurrent log writing to the database occurs. At this time, if the log concurrent writing database solution of this embodiment is adopted, the concurrent writing of a large number of logs to the telemedicine platform can be achieved by combining the interaction of pipelines, log processes and other processes, thereby improving the efficiency of log writing to the database and avoiding log writing delays and blocking problems in high concurrency scenarios.

[0169] It can be seen from the above technical solution that the present invention can use the daemon process in the target database to create multiple parallel pipelines and log processes corresponding to each pipeline. The creation of parallel pipelines and log processes improves the log writing speed in high-concurrency scenarios and facilitates the expansion of new pipelines and log processes; each log process is started in sequence, and each log process is used to monitor the corresponding pipeline in parallel, and other processes in the target database are used to generate logs and send the generated logs to the corresponding pipeline. Furthermore, when any log process monitors that a target log is sent to the corresponding pipeline, any log process is used to write the target log to the target database. Since each log process monitors the corresponding pipeline respectively and reads the log from the corresponding pipeline and writes it to the database, there is no need to compete for the same resource lock, which effectively improves the system performance and can still efficiently perform log writing operations in high-concurrency scenarios.

[0170] like Figure 3 FIG. 1 is a schematic diagram of the structure of a computer device according to a preferred embodiment of the method for concurrently writing logs into a database according to the present invention.

[0171] The computer device 1 may include a memory 12, a processor 13 and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a program for concurrently writing logs into a database.

[0172] Those skilled in the art will understand that the schematic diagram is merely an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 may have either a bus structure or a star structure. The computer device 1 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components. For example, the computer device 1 may also include input and output devices, network access devices, etc.

[0173] It should be noted that the computer device 1 is only an example. Other existing or future electronic products that are suitable for the present invention should also be included in the scope of protection of the present invention and included here by reference.

[0174] The memory 12 includes at least one type of readable storage medium, including flash memory, a removable hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 12 may be an internal storage unit of the computer device 1, such as a removable hard disk of the computer device 1. In other embodiments, the memory 12 may also be an external storage device of the computer device 1, such as a plug-in removable hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the computer device 1. Furthermore, the memory 12 may include both an internal storage unit of the computer device 1 and an external storage device. The memory 12 can be used not only to store application software installed on the computer device 1 and various types of data, such as the code of programs that log and write to databases, but also to temporarily store data that has been output or is about to be output.

[0175] In some embodiments, processor 13 may be comprised of an integrated circuit, such as a single packaged integrated circuit or multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. Processor 13 is the control core (control unit) of computer device 1, connecting the various components of computer device 1 using various interfaces and circuits. It executes programs or modules stored in memory 12 (such as a program that executes logs and concurrently writes to a database) and accesses data stored in memory 12 to perform various functions and process data.

[0176] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the embodiment of the method for concurrently writing logs into the database, for example Figure 1 Steps shown.

[0177] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a creation unit 110, a startup unit 111, a sending unit 112, and a writing unit 113.

[0178] The integrated unit implemented as a software functional module can be stored in a computer-readable storage medium. The software functional module stored in the storage medium includes instructions for causing a computer device (which can be a personal computer, computer equipment, or network device, etc.) or a processor to execute the method for concurrently writing logs to a database described in various embodiments of the present invention.

[0179] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments.

[0180] The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0181] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0182] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks linked together using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product and service layer, and the application service layer.

[0183] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The figure shows that only one straight line is used, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13.

[0184] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 via a power management device, thereby enabling functions such as charge management, discharge management, and power consumption management through the power management device. The power supply may also include one or more DC or AC power supplies, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be further described here.

[0185] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the computer device 1 and other computer devices.

[0186] Optionally, the computer device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a display screen or display unit, and is used to display information processed by the computer device 1 and to display a visual user interface.

[0187] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0188] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0189] Combine Figure 1 The memory 12 in the computer device 1 stores a plurality of instructions to implement a method for concurrently writing logs into a database, and the processor 13 can execute the plurality of instructions to implement:

[0190] In response to a concurrent log write instruction for a target database, a daemon process in the target database is used to create a plurality of parallel pipelines and a log process corresponding to each pipeline;

[0191] Start each log process in turn, and use each log process to monitor the corresponding pipeline in parallel;

[0192] Utilizing other processes in the target database to generate logs, and sending the generated logs to the multiple pipelines;

[0193] When any log process monitors that a target log is sent to a corresponding pipeline, the target log is written into the target database using the any log process.

[0194] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0195] It should be noted that the data involved in this case were all obtained legally. The software tools or components not produced by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0196] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0197] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0198] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0199] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0200] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0201] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0202] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the present invention may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for concurrently writing logs into a database, characterized in that: The method for concurrently writing logs into a database includes: In response to a concurrent log write instruction for a target database, a daemon process in the target database is used to create multiple parallel pipes and log processes corresponding to each pipe; each pipe includes a pipe reader and a pipe writer; the pipe reader is used for the corresponding log process to monitor whether there is a log written to the corresponding pipe and read the log from the pipe reader; the pipe writer is used for other processes to send logs to the corresponding pipe; Start each log process in turn, and use each log process to monitor the corresponding pipeline in parallel; Utilizing other processes in the target database to generate logs and sending the generated logs to the multiple parallel pipelines, the method includes: obtaining the process number of each other process and the number of configured log processes; calculating the quotient of each process number and the number of log processes to obtain a quotient value corresponding to each other process; performing a modulo operation on each quotient value to obtain a target value corresponding to each other process; using each target value as a pipeline number to match a target pipeline corresponding to each other process from the pipeline; and sending the log generated by each other process to the corresponding target pipeline; When any log process monitors that a target log is sent to a corresponding pipeline, the target log is written into the target database using the any log process.

2. The method for concurrently writing logs into a database according to claim 1, wherein: The method of creating multiple parallel pipelines and log processes corresponding to each pipeline by utilizing the daemon process in the target database includes: Using the daemon process to create the pipeline according to the number of log processes, and configuring the pipeline number of each pipeline according to the pipeline creation order; The daemon process is used to create a log process corresponding to each pipeline, and the pipeline number of each pipeline is sent to the corresponding log process.

3. The method for concurrently writing logs into a database according to claim 1, wherein: After creating multiple parallel pipelines and a log process corresponding to each pipeline using the daemon process in the target database, the method further includes: When the number of the log processes increases, obtaining the increase amount of the log processes; The daemon process is used to create new pipelines and log processes corresponding to each new pipeline according to the increase.

4. The method for concurrently writing logs into a database according to claim 1, wherein: Before creating a plurality of parallel pipelines and a log process corresponding to each pipeline using the daemon process in the target database, the method further comprises: Detect whether there is a log file directory in the target database; When it is detected that the target database does not have the log file directory, creating the log file directory by using the daemon process; The log file directory is used to store log files created by each log process.

5. The method for concurrently writing logs into a database according to claim 1, wherein: Starting each log process in sequence includes: For each log process, when the log process receives the pipe number sent by the daemon process, the pipe number is stored in the memory, a log file is created according to the pipe number, and the log file switching time is configured; Wherein, the log file number of the log file is the same as the pipeline number; After one log process is started, the next log process is started according to the log file number.

6. The method for concurrently writing logs into a database according to claim 1, wherein: Writing the target log into the target database by using the arbitrary log process includes: Waking up the arbitrary log process and using the arbitrary log process to detect whether the configuration file needs to be reloaded; When it is detected that the configuration file needs to be reloaded, the updated configuration file is loaded as the target configuration file; Detecting whether the log file switching time stored in the target configuration file changes; When a change in the log file switching time stored in the target configuration file is detected, the log file switching time stored in the target configuration file is determined as the target log file switching time, and the original log file switching time of any log process is configured as the target log file switching time; Get the current timestamp; When the current timestamp is the same as or after the target log file switching time, closing the original log file corresponding to the arbitrary log process by using the arbitrary log process, and creating a new log file as the target log file; wherein the target log file and the original log file have the same log file number, and the target log file and the original log file are distinguished according to the creation time; The target log in the corresponding pipeline is read by using the arbitrary log process, and the target log is written into the target log file.

7. A device for concurrently writing logs into a database, characterized in that: The device for concurrently writing logs into a database includes: A creation unit is configured to, in response to a concurrent log write instruction to a target database, create multiple parallel pipelines and log processes corresponding to each pipeline using a daemon process in the target database; wherein each pipeline includes a pipeline reading end and a pipeline writing end; the pipeline reading end is used for the corresponding log process to monitor whether there is a log written to the corresponding pipeline and read the log from the pipeline reading end; the pipeline writing end is used for other processes to send logs to the corresponding pipeline; The startup unit is used to start each log process in sequence and use each log process to monitor the corresponding pipeline in parallel; The sending unit is configured to generate logs using other processes in the target database and send the generated logs to the multiple parallel pipelines, including: obtaining the process number of each other process and the number of log processes; calculating the quotient of each process number and the number of log processes to obtain a quotient value corresponding to each other process; performing a modulo operation on each quotient value to obtain a target value corresponding to each other process; using each target value as a pipeline number to match a target pipeline corresponding to each other process from the pipeline; and sending the log generated by each other process to the corresponding target pipeline; The writing unit is configured to write the target log into the target database by using any log process when any log process monitors that a target log is sent to a corresponding pipeline.

8. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the method for concurrently writing logs into a database as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the method for concurrently writing logs into a database as claimed in any one of claims 1 to 6.

Images