Method, device and equipment for concurrently writing logs into database and medium

Abstract

The invention relates to the technical field of data processing of finance and medical health, and provides a method, a device, equipment and a medium for concurrently writing logs into a database, which can utilize a daemon process in a target database to create a plurality of parallel pipelines and log processes corresponding to each pipeline. Due to the establishment of the parallel pipeline and the log process, the write-in speed of the log in a high-concurrency scene is increased, and a new pipeline and a new log process are convenient to expand; each log process is started in sequence to monitor the corresponding pipeline in parallel, and when any log process monitors that a target log is sent to the corresponding pipeline by other processes, the target log is written into the target database by using any log process. And the log is read from the corresponding pipeline and written into the database, so that the same resource lock does not need to be competed, the system performance is effectively improved, and the log writing operation can still be efficiently executed in a high-concurrency scene.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing in finance and healthcare, and particularly to a method, device, equipment and medium for concurrently writing logs into a database. Background Art

[0002] Currently, in both the financial field and the healthcare field, there are scenarios of concurrently writing high-concurrency logs into a database. For example, in the financial field, especially for online trading scenarios, a large number of transaction logs are generated during peak trading hours, and these transaction logs need to be written into the database simultaneously to ensure the traceability of transactions and the stability of the system; in the healthcare field, with the wide promotion of online consultations and electronic medical records, a large amount of log information is generated during the medical treatment process. For example, during the peak period of hospital visits, due to the large number of patients, medical staff in each department operate in the system simultaneously, generating a large amount of log data that needs to be written into the database.

[0003] However, in existing database systems, log writing is usually a serialized operation, and all log data needs to be processed and written through a single log process. This single-process log writing method is prone to becoming a performance bottleneck in high-concurrency scenarios. Especially when dealing with a large number of transactions and log data, it will cause a decline in database performance and even problems such as latency and blocking.

[0004] Specifically, the existing technology mainly has the following deficiencies: 1. Performance bottleneck: Single-process log writing cannot fully utilize the computing power of multi-core processors, resulting in the log writing speed being unable to keep up with the transaction processing speed, and this problem is particularly obvious in high-concurrency scenarios; 2. Lock contention: All log writing operations need to compete for the same resource lock (such as a pipe read-write lock), which increases the overhead of lock contention and further reduces the system performance; 3. Poor scalability: In high-load scenarios, single-process log writing cannot be scaled and cannot meet the needs of large-scale transaction processing, resulting in limited overall throughput of the database.

[0005] Therefore, the existing technology has obvious deficiencies in terms of log writing performance and scalability in high-concurrency scenarios. Summary of the Invention

[0006] In view of the above, it is necessary to provide a method, device, equipment and medium for concurrently writing logs into a database, aiming to solve the problems of low efficiency, low performance and poor scalability existing when writing logs into the database in high-concurrency scenarios.

[0007] A method for concurrently writing logs into a database, the method for concurrently writing logs into a database includes: In response to a log concurrent writing instruction for a target database, use a daemon process in the target database to create multiple parallel pipelines and corresponding log processes for each pipeline; Start each log process in sequence and use each log process to concurrently monitor the corresponding pipeline; Use other processes in the target database to generate logs and send the generated logs to the multiple pipelines; When any log process monitors that a target log is sent to the corresponding pipeline, use the any log process to write the target log into the target database.

[0008] An apparatus for concurrently writing logs into a database, the apparatus for concurrently writing logs into a database includes: A creation unit, configured to, in response to a log concurrent writing instruction for a target database, use a daemon process in the target database to create multiple parallel pipelines and corresponding log processes for each pipeline; A start unit, configured to start each log process in sequence and use each log process to concurrently monitor the corresponding pipeline; A sending unit, configured to use other processes in the target database to generate logs and send the generated logs to the multiple pipelines; A writing unit, configured to, when any log process monitors that a target log is sent to the corresponding pipeline, use the any log process to write the target log into the target database.

[0009] A computer device, the computer device includes: A memory, storing at least one instruction; and A processor, executing the instruction stored in the memory to implement the method for concurrently writing logs into a database.

[0010] A computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in a computer device to implement the method for concurrently writing logs into a database.

[0011] As can be seen from the above technical solutions, the present invention can use the daemon process in the target database to create multiple parallel pipelines and the log processes corresponding to each pipeline. The creation of the parallel pipelines and log processes improves the writing speed of logs in a high-concurrency scenario and facilitates the expansion of new pipelines and log processes. Each log process is started in sequence, and each log process is used to listen to the corresponding pipeline in parallel. Other processes in the target database generate logs and send the generated logs to the corresponding pipeline. Further, when any log process listens that a target log is sent to the corresponding pipeline, any log process is used to write the target log into the target database. Since each log process listens to the corresponding pipeline separately and reads the logs from the corresponding pipeline and writes them into the database, there is no need to compete for the same resource lock, effectively improving the system performance, and still being able to efficiently execute the log writing operation in a high-concurrency scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flowchart of a preferred embodiment of the method for concurrently writing logs into a database according to the present invention; Figure 2 is a functional module diagram of a preferred embodiment of the device for concurrently writing logs into a database according to the present invention; Figure 3 is a schematic structural diagram of a computer device of a preferred embodiment of the method for implementing concurrent log writing into a database according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] As Figure 1 shown, it is a flowchart of a preferred embodiment of the method for concurrently writing logs into a database according to the present invention. According to different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0015] The method for concurrently writing logs into a database is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0016] The computer device can be any electronic product that can perform human-computer interaction with users. For example, personal computers, tablet computers, smart phones, personal digital assistants (PDAs), game consoles, Internet Protocol Television (IPTV), smart wearable devices, etc.

[0017] The computer device may further include a network device and / or a user device. Among them, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0018] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0019] Among them, artificial intelligence (AI) is a theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0020] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0021] The network where the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, Virtual Private Network (VPN), etc.

[0022] S10. In response to a log concurrent write instruction for a target database, use a daemon process in the target database to create multiple parallel pipelines and log processes corresponding to each pipeline.

[0023] In this embodiment, the target database can be a relational database, a non-relational database, or other databases with a log writing function.

[0024] For example, in a financial scenario, the target database may be a log writing database for an online trading system, a payment and clearing system, and a risk management system; in a medical and health scenario, the target database may be a log writing database for a hospital information system, a medical Internet of Things device data acquisition system, a remote medical platform, etc.

[0025] In this embodiment, the log concurrent writing instruction can be automatically triggered after the target database is deployed to optimize the entire process of the log writing process for the target database.

[0026] In this embodiment, the creating, by using a daemon process in the target database, of a plurality of parallel pipelines and log processes corresponding to each pipeline includes: Obtaining the configured number of log processes; Using the daemon process to create the pipelines according to the number of log processes, and configuring a pipeline number for each pipeline according to the creation order of the pipelines; Using the daemon process to create log processes corresponding to each pipeline, and sending the pipeline number of each pipeline to the corresponding log process.

[0027] Among them, the log process data can be configured according to the demand data in the actual scenario. For example, the number of log processes can be comprehensively configured according to the data scale and system performance in the actual scenario.

[0028] Among them, the daemon process can create the pipelines in sequence starting from 0 as the pipeline numbers according to the number of log processes. For example, when the number of log processes is N (N is a positive integer), the number of created pipelines is N, and the pipeline numbers increase sequentially from 0 to (N - 1).

[0029] Through the above embodiments, it is possible to create a plurality of parallel pipelines and log processes corresponding to each pipeline. Since the pipelines are in a parallel relationship, it is possible to provide execution conditions for the subsequent parallel writing of logs.

[0030] In this embodiment, after creating a plurality of parallel pipelines and log processes corresponding to each pipeline by using a daemon process in the target database, the method further includes: When the number of log processes increases, obtaining the increase amount of the log processes; Using the daemon process to create new pipelines and log processes corresponding to each new pipeline according to the increase amount.

[0031] In the above embodiments, due to the parallel relationship between pipelines, when there is a new pipeline requirement, new pipelines and corresponding log processes can be flexibly created directly according to the new pipeline demand, making full use of the computing power of multi-core processors, effectively solving the problem that the log writing mode of a single process cannot be extended in high-load scenarios, meeting the requirements of large-scale transaction processing, and improving the overall throughput of the database system.

[0032] In this embodiment, before creating multiple parallel pipelines and log processes corresponding to each pipeline by using the daemon process in the target database, the method further includes: Detecting whether there is a log file directory in the target database; When it is detected that there is no such log file directory in the target database, creating the log file directory by using the daemon process; Wherein, the log file directory is used to store the log files created by each log process.

[0033] Wherein, when it is detected that there is such a log file directory in the target database, the existing log file directory can be directly used to store the log files created by each log process.

[0034] Through the above embodiments, the log files can be stored uniformly, facilitating subsequent management and query of the log files.

[0035] S11, starting each log process in sequence and using each log process to listen to the corresponding pipeline in parallel.

[0036] In this embodiment, the starting each log process in sequence includes: For each log process, when the log process receives the pipeline number sent by the daemon process, storing the pipeline number in the memory, creating a log file according to the pipeline number, and configuring the log file switching time; Wherein, the log file number of the log file is the same as the pipeline number.

[0037] For example: when the pipeline number is 0, the log file number of the log file corresponding to the pipeline with pipeline number 0 is also 0.

[0038] Wherein, after a log process completes startup, the next log process is started according to the log file number.

[0039] For example: after the log process corresponding to the pipeline with pipeline number 0 starts up, the log process corresponding to the pipeline with pipeline number 1 is started until all log processes are started up.

[0040] Among them, the log file switching time is used to limit the size of the log file, and the log file switching time can be comprehensively configured according to parameters such as system performance and retrieval efficiency.

[0041] For example, the log file switching time can be configured to be every hour, every day, etc.

[0042] By configuring the log file switching time, it is possible to effectively avoid the problem that the volume of the log file is too large, which affects the normal writing of the log and the subsequent retrieval efficiency.

[0043] Through the above embodiments, it is possible to start the log processes corresponding to each pipeline in an orderly manner, so that the log processes can be used to effectively monitor the logs in each pipeline.

[0044] In this embodiment, each log process can be used to listen to the pipe read end of the corresponding pipeline in parallel to wait for other processes to send logs to the pipeline.

[0045] Among them, each pipeline includes a pipe read end and a pipe write end.

[0046] The pipe read end is used for the corresponding log process to monitor whether there is a log written to the corresponding pipeline and read the log from the pipe read end.

[0047] The pipe write end is used for other processes to send logs to the corresponding pipeline.

[0048] S12, use other processes in the target database to generate logs and send the generated logs to the multiple pipelines.

[0049] In this embodiment, the sending the generated logs to the multiple pipelines includes: Obtain the process number of each other process and obtain the number of log processes; Calculate the quotient of each process number and the number of log processes to obtain a quotient value corresponding to each other process; Perform a remainder operation on each quotient value to obtain a target value corresponding to each other process; Use each target value as the pipeline number to match the target pipeline corresponding to each other process from the pipelines; Send the logs generated by each other process to the corresponding target pipeline.

[0050] For example: when the remainder of the quotient of the process number of other process X and the number of log processes is 3, then send the logs generated by other process X to the pipeline with pipeline number 3.

[0051] Through the above embodiments, it is possible to write the generated logs into the corresponding pipelines in an orderly manner according to a unified rule, avoiding data chaos and loss problems.

[0052] S13. When any log process monitors that a target log is sent to the corresponding pipeline, use the any log process to write the target log into the target database.

[0053] In this embodiment, the using the any log process to write the target log into the target database includes: Wake up the any log process, and use the any log process to detect whether it is necessary to reload the configuration file; When it is detected that it is necessary to reload the configuration file, load the updated configuration file as the target configuration file; Detect whether the log file switching time stored in the target configuration file changes; When it is detected that the log file switching time stored in the target configuration file changes, determine the log file switching time stored in the target configuration file as the target log file switching time, and configure the original log file switching time of the any log process as the target log file switching time; Obtain the current timestamp; When the current timestamp is the same as or after the target log file switching time, use the any log process to close the original log file corresponding to the any log process, and create a new log file as the target log file; wherein, the target log file has the same log file number as the original log file, and the target log file and the original log file are distinguished according to the creation time; Use the any log process to read the target log in the corresponding pipeline, and write the target log into the target log file.

[0054] Among them, after starting each log process, in order to save system resources, when no log is monitored to be sent to the corresponding pipeline, the log process is in a dormant state until a log is monitored to be sent to the corresponding pipeline, and then the corresponding log process is awakened.

[0055] Among them, the configuration file is used to store log parameters such as the log file switching time, database connection information, log format, and communication configuration between processes.

[0056] When the log writing requirements are updated according to changes in the application environment, data scale, etc., the configuration file will also be updated synchronously.

[0057] Among them, after using the any log process to close the original log file corresponding to the any log process, stop writing logs to the original log file and keep the current file size unchanged, so as to avoid the file volume being too large and affecting the subsequent retrieval efficiency.

[0058] At the same time, a new log file is created as the target log file to avoid affecting subsequent log file writing operations.

[0059] If it is detected that the configuration file does not need to be reloaded, or the configuration file needs to be reloaded but the log file switching time stored in the target configuration file has not changed, the original log file switching time can be directly used as a detection condition to determine whether to create a new log file.

[0060] If the current timestamp is before the target log file switching time, the target log in the corresponding pipeline is read using the arbitrary log process, and the target log is written into the original log file.

[0061] After the target log is written into the corresponding log file, the corresponding pipeline is continuously monitored by the arbitrary log process, and the process is awakened after waiting for other log processes to send log data.

[0062] Through the above embodiments, it is possible to implement parallel writing of logs through parallel monitoring of multiple pipelines by each log process, which significantly improves the log writing performance of the database, especially in high concurrency scenarios, and can effectively reduce log writing delays and blocking problems. In addition, the independent log file and pipeline design reduces the lock competition problem, thereby reducing system overhead and further improving database performance. At the same time, the reasonable coordination and communication between each process also ensures the consistency and integrity of the log data, avoiding data confusion and loss problems.

[0063] This embodiment can be applied to high-concurrency log writing database tasks in financial scenarios and medical and health scenarios.

[0064] For example: For financial scenarios, such as large securities trading platforms, thousands or even tens of thousands of transactions may be processed per second during peak trading hours, which means that an equal amount or even more log data needs to be written to the database at the same time every second to ensure the traceability and data integrity of the transaction. For example: For a stock trading system, in the continuous bidding phase after the end of the early call auction, due to the concentrated orders of a large number of investors, there will be instantaneous high-concurrency transaction requests, and correspondingly high-concurrency log writing tasks will be generated, requiring the database to be able to quickly and accurately record the details of each transaction. At this time, if the log concurrent writing database solution of this embodiment is adopted, the concurrent writing of a large number of transaction logs can be realized in combination with the interaction of pipelines, log processes and other processes, thereby improving the efficiency of log writing to the database and avoiding log writing delays and blocking problems in high-concurrency scenarios.

[0065] For another example, in the medical and health scenario, a telemedicine platform allows doctors to provide medical services to remote patients via the network, including online consultations, remote diagnoses, remote monitoring, etc. During the telemedicine process, the platform needs to record detailed log information, such as video call records between doctors and patients, consultation content, diagnosis results, prescribed prescriptions, and health data transmission records of patients. These logs not only help doctors track and manage patients' conditions but also provide evidence for medical disputes. With the popularization of telemedicine, during some specific time periods, such as evenings or weekends, patients may concentrate on using the telemedicine platform for consultations. Suppose a large-scale telemedicine platform serves patients in multiple regions simultaneously. Between 7 pm and 9 pm on a certain evening, there may be hundreds of patients having video consultations with doctors at the same time. The establishment and end of each video call, as well as the information interaction during the consultation process, will generate corresponding log records, and all these logs need to be written into the database in a timely manner. In addition, when patients upload health data using remote monitoring devices, a large number of data transmission logs will also be generated. Therefore, during the peak usage period of the telemedicine platform, there will be high-concurrency log writing tasks to the database. At this time, if the log concurrent writing database scheme of this embodiment is adopted, it is possible to achieve the concurrent writing of a large number of logs of the telemedicine platform by combining the interaction of pipelines, log processes, and other processes, thereby improving the efficiency of writing logs into the database and avoiding log writing delays and blocking problems in high-concurrency scenarios.

[0066] As can be seen from the above technical solutions, the present invention can create multiple parallel pipelines and corresponding log processes for each pipeline by using the daemon process in the target database. The creation of parallel pipelines and log processes improves the writing speed of logs in high-concurrency scenarios and is convenient for expanding new pipelines and log processes; start each log process in sequence, and use each log process to listen to the corresponding pipeline in parallel. Use other processes in the target database to generate logs and send the generated logs to the corresponding pipeline. Further, when any log process listens that a target log is sent to the corresponding pipeline, use any log process to write the target log into the target database. Since each log process listens to the corresponding pipeline separately and reads the logs from the corresponding pipeline and writes them into the database, there is no need to compete for the same resource lock, effectively improving the system performance, and still being able to efficiently execute the log writing operation in high-concurrency scenarios.

[0067] Such as Figure 2As shown in the figure, it is a functional module diagram of a preferred embodiment of the device for concurrent writing of logs to a database according to the present invention. The device 11 for concurrent writing of logs to a database includes a creation unit 110, a start unit 111, a sending unit 112, and a writing unit 113. The modules / units referred to in the present invention refer to a series of computer program segments that can be executed by a processor and can complete fixed functions, and are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0068] The creation unit 110 is configured to, in response to a log concurrent writing instruction for a target database, create a plurality of parallel pipes and corresponding log processes for each pipe by using a daemon process in the target database.

[0069] In this embodiment, the target database may be a relational database, a non-relational database, or other databases with a log writing function.

[0070] For example: in a financial scenario, the target database may be a log writing database for an online trading system, a payment and settlement system, and a risk management system; in a medical and health scenario, the target database may be a log writing database for a hospital information system, a medical Internet of Things device data acquisition system, a remote medical platform, etc.

[0071] In this embodiment, the log concurrent writing instruction may be automatically triggered after the target database is deployed to achieve full-process optimization of the log writing process for the target database.

[0072] In this embodiment, the creation unit 110 creating a plurality of parallel pipes and corresponding log processes for each pipe by using a daemon process in the target database includes: Obtaining the configured number of log processes; Using the daemon process to create the pipes according to the number of log processes, and configuring a pipe number for each pipe according to the pipe creation order; Using the daemon process to create log processes corresponding to each pipe, and sending the pipe number of each pipe to the corresponding log process.

[0073] Among them, the log process data can be configured according to the required data in the actual scenario. For example: the number of log processes can be comprehensively configured according to the data scale and system performance in the actual scenario.

[0074] Among them, the daemon process can create the pipes in sequence starting from 0 as the pipe number according to the number of log processes. For example: when the number of log processes is N (N is a positive integer), the number of created pipes is N, and the pipe numbers increase sequentially, from 0 to (N - 1).

[0075] Through the above embodiments, multiple parallel pipelines and corresponding log processes for each pipeline can be created. Since the pipelines are in a parallel relationship, execution conditions can be provided for subsequent parallel writing of logs.

[0076] In this embodiment, after the creation unit 110 creates multiple parallel pipelines and corresponding log processes for each pipeline by using the daemon process in the target database, when the number of log processes increases, the increase amount of the log processes is obtained; The daemon process is used to create new pipelines and corresponding log processes for each new pipeline according to the increase amount.

[0077] In the above embodiments, due to the parallel relationship between the pipelines, when there is a need for new pipelines, new pipelines and corresponding log processes can be flexibly created directly according to the new demand for pipelines, making full use of the computing power of the multi-core processor, effectively solving the problem that the single-process log writing mode cannot be extended in high-load scenarios, meeting the needs of large-scale transaction processing, and improving the overall throughput of the database system.

[0078] In this embodiment, before the creation unit 110 creates multiple parallel pipelines and corresponding log processes for each pipeline by using the daemon process in the target database, it detects whether there is a log file directory in the target database; When it is detected that there is no such log file directory in the target database, the daemon process is used to create the log file directory; Wherein, the log file directory is used to store the log files created by each log process.

[0079] Wherein, when it is detected that there is such a log file directory in the target database, the existing log file directory can be directly used to store the log files created by each log process.

[0080] Through the above embodiments, the log files can be stored uniformly, facilitating subsequent management and query of the log files.

[0081] The start unit 111 is used to start each log process in sequence and use each log process to listen to the corresponding pipeline in parallel.

[0082] In this embodiment, the start unit 111 starting each log process in sequence includes: For each log process, when the log process receives the pipeline number sent by the daemon process, the pipeline number is stored in the memory, and a log file is created according to the pipeline number, and the log file switching time is configured; Wherein, the log file number of the log file is the same as the pipeline number.

[0083] For example, when the pipeline number is 0, the log file number corresponding to the pipeline with pipeline number 0 is also 0.

[0084] Among them, after a log process is started, the next log process is started according to the log file number.

[0085] For example: after the log process corresponding to the pipeline with pipeline number 0 is started, then start the log process corresponding to the pipeline with pipeline number 1 until all log processes are started.

[0086] Among them, the log file switching time is used to limit the size of the log file, and the log file switching time can be comprehensively configured according to parameters such as system performance and retrieval efficiency.

[0087] For example: the log file switching time can be configured to be every hour, every day, etc.

[0088] By configuring the log file switching time, it can effectively avoid the log file being too large and affecting the normal writing of logs and subsequent retrieval efficiency.

[0089] Through the above embodiments, the log processes corresponding to each pipeline can be started orderly, so that the logs in each pipeline can be effectively monitored by using each log process.

[0090] In this embodiment, each log process can be used to listen to the pipeline read end of the corresponding pipeline in parallel to wait for other processes to send logs to the pipeline.

[0091] Among them, each pipeline includes a pipeline read end and a pipeline write end.

[0092] The pipeline read end is used for the corresponding log process to listen to whether there is a log written to the corresponding pipeline and read the log from the pipeline read end.

[0093] The pipeline write end is used for other processes to send logs to the corresponding pipeline.

[0094] The sending unit 112 is used to generate logs by other processes in the target database and send the generated logs to the multiple pipelines.

[0095] In this embodiment, the sending unit 112 sending the generated logs to the multiple pipelines includes: Obtain the process number of each other process and obtain the number of log processes; Calculate the quotient of each process number and the number of log processes to obtain a quotient value corresponding to each other process; Perform a modulo operation on each quotient value to obtain a target value corresponding to each other process; Use each target value as a pipeline number to match the target pipeline corresponding to each other process from the pipelines; Send the logs generated by each other process to the corresponding target pipeline.

[0096] For example: when the remainder of the quotient of the process number of another process X and the number of log processes is 3, the logs generated by another process X are sent to the pipeline with the pipeline number 3.

[0097] Through the above embodiments, the generated logs can be written into the corresponding pipelines orderly according to unified rules, avoiding data chaos and loss problems.

[0098] The writing unit 113 is configured to, when any log process monitors that a target log is sent to the corresponding pipeline, use the any log process to write the target log into the target database.

[0099] In this embodiment, the writing unit 113 using the any log process to write the target log into the target database includes: Wake up the any log process, and use the any log process to detect whether it is necessary to reload the configuration file; When it is detected that it is necessary to reload the configuration file, load the updated configuration file as the target configuration file; Detect whether the log file switching time stored in the target configuration file changes; When it is detected that the log file switching time stored in the target configuration file changes, determine the log file switching time stored in the target configuration file as the target log file switching time, and configure the original log file switching time of the any log process as the target log file switching time; Obtain the current timestamp; When the current timestamp is the same as or after the target log file switching time, use the any log process to close the original log file corresponding to the any log process, and create a new log file as the target log file; wherein, the target log file has the same log file number as the original log file, and the target log file and the original log file are distinguished according to the creation time; Use the any log process to read the target log in the corresponding pipeline, and write the target log into the target log file.

[0100] Among them, after each log process is started, in order to save system resources, when no log is monitored to be sent to the corresponding pipeline, the log process is in a dormant state until a log is monitored to be sent to the corresponding pipeline, and then the corresponding log process is awakened.

[0101] Among them, the configuration file is used to store log parameters such as the log file switching time, database connection information, log format, etc., and communication configuration between processes.

[0102] When the log writing requirements are updated according to changes in the application environment, data scale, etc., the configuration file will also be updated synchronously.

[0103] Among them, after using any log process to close the original log file corresponding to the any log process, stop writing logs to the original log file and keep the current file size unchanged, so as to avoid the file volume being too large and affecting the subsequent retrieval efficiency.

[0104] At the same time, create a new log file as the target log file, so as to avoid affecting subsequent log file writing operations.

[0105] Among them, if it is detected that the configuration file does not need to be reloaded, or when the configuration file needs to be reloaded but the log file switching time stored in the target configuration file has not changed, the original log file switching time can be directly used as the detection condition to determine whether to create a new log file.

[0106] If the current timestamp is before the target log file switching time, use any log process to read the target log in the corresponding pipeline and write the target log to the original log file.

[0107] Among them, after writing the target log to the corresponding log file, continue to use any log process to monitor the corresponding pipeline and wait to be awakened after other log processes send log data.

[0108] Through the above embodiments, parallel writing of logs can be achieved through parallel monitoring of multiple pipelines by each log process, significantly improving the log writing performance of the database. Especially in high-concurrency scenarios, it can effectively reduce log writing latency and blocking problems. Moreover, the independent log file and pipeline design reduces the lock competition problem, thereby reducing system overhead and further improving database performance. At the same time, the reasonable cooperation and communication between processes also ensure the consistency and integrity of log data, avoiding data chaos and loss problems.

[0109] This embodiment can be applied to high-concurrency log writing database tasks in financial scenarios and medical and health scenarios.

[0110] For example, in the financial scenario, such as a large securities trading platform, thousands or even tens of thousands of transactions may be processed per second during the trading peak. This means that an equal amount or even more log data needs to be written to the database simultaneously per second to ensure the traceability and data integrity of transactions. For example, in the continuous auction phase after the call auction for stocks in the morning session, due to a large number of investors placing orders concentratedly, instantaneous high-concurrency trading requests will occur, and correspondingly, high-concurrency log writing tasks will be generated, requiring the database to quickly and accurately record the details of each transaction. At this time, if the log concurrent writing database solution of this embodiment is adopted, the concurrent writing of a large number of transaction logs can be achieved by combining the interaction of pipelines, log processes, and other processes, thereby improving the efficiency of writing logs to the database and avoiding log writing delays and blocking problems in high-concurrency scenarios.

[0111] Another example: In the medical and health scenario, a telemedicine platform allows doctors to provide medical services to remote patients through the network, including online consultations, remote diagnoses, remote monitoring, etc. During the telemedicine process, the platform needs to record detailed log information, such as video call records between doctors and patients, consultation content, diagnosis results, prescriptions issued, and health data transmission records of patients. These logs not only help doctors track and manage patients' conditions but also provide evidence for medical disputes. With the popularization of telemedicine, during some specific time periods, such as evenings or weekends, patients may concentrate on using the telemedicine platform for consultations. Suppose a large telemedicine platform serves patients in multiple regions simultaneously. Between 7 pm and 9 pm on a certain evening, hundreds of patients may be having video consultations with doctors simultaneously. The establishment and end of each video call, as well as the information interaction during the consultation process, will generate corresponding log records, and all these logs need to be written to the database in a timely manner. In addition, when patients upload health data using remote monitoring devices, a large amount of data transmission logs will also be generated. Therefore, during the peak usage period of the telemedicine platform, high-concurrency log writing database tasks will occur. At this time, if the log concurrent writing database solution of this embodiment is adopted, the concurrent writing of a large number of logs on the telemedicine platform can be achieved by combining the interaction of pipelines, log processes, and other processes, thereby improving the efficiency of writing logs to the database and avoiding log writing delays and blocking problems in high-concurrency scenarios.

[0112] As can be seen from the above technical solutions, the present invention can use the daemon process in the target database to create multiple parallel pipelines and the log processes corresponding to each pipeline. The creation of the parallel pipelines and log processes improves the log writing speed in high-concurrency scenarios and facilitates the expansion of new pipelines and log processes. Each log process is started in sequence, and each log process is used to listen to the corresponding pipeline in parallel. Other processes in the target database generate logs and send the generated logs to the corresponding pipeline. Further, when any log process listens and detects that a target log is sent to the corresponding pipeline, any log process is used to write the target log into the target database. Since each log process listens to the corresponding pipeline separately and reads the logs from the corresponding pipeline and writes them into the database, there is no need to compete for the same resource lock, effectively improving the system performance, and the log writing operation can still be efficiently executed in high-concurrency scenarios.

[0113] As Figure 3 shown, it is a schematic structural diagram of a computer device of a preferred embodiment of the method for realizing concurrent log writing into a database according to the present invention.

[0114] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrows in the figure are the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a program for concurrent log writing into a database.

[0115] Those skilled in the art can understand that the schematic diagram is only an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 can be either a bus structure or a star structure. The computer device 1 may also include more or fewer other hardware or software than shown in the figure, or different component arrangements. For example, the computer device 1 may also include input / output devices, network access devices, etc.

[0116] It should be noted that the computer device 1 is only an example, and other existing or future possible electronic products that can be adapted to the present invention should also be included within the protection scope of the present invention and are included herein by reference.

[0117] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as the mobile hard disk of the computer device 1. In other embodiments, the memory 12 can also be an external storage device of the computer device 1, such as a plug-in mobile hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the computer device 1. Further, the memory 12 can also include both the internal storage unit and the external storage device of the computer device 1. The memory 12 can be used not only to store application software installed on the computer device 1 and various types of data, such as the code of the program for concurrently writing logs into the database, but also to temporarily store data that has been output or will be output.

[0118] In some embodiments, the processor 13 can be composed of integrated circuits. For example, it can be composed of a single packaged integrated circuit, or can be composed of multiple integrated circuits with the same or different functions, including the combination of one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, and various control chips, etc. The processor 13 is the control core (Control Unit) of the computer device 1, connecting various components of the entire computer device 1 through various interfaces and lines. By running or executing programs or modules stored in the memory 12 (such as executing the program for concurrently writing logs into the database), and calling data stored in the memory 12, it performs various functions of the computer device 1 and processes data.

[0119] The processor 13 executes the operating system of the computer device 1 and various installed application programs. The processor 13 executes the application program to implement the steps in the method embodiments of concurrently writing each log into the database, such as Figure 1 the steps shown.

[0120] Exemplarily, the computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a creation unit 110, a startup unit 111, a sending unit 112, and a writing unit 113.

[0121] The integrated units implemented in the form of software function modules may be stored in a computer-readable storage medium. The above-mentioned software function modules stored in a storage medium include several instructions for causing a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of the method for concurrently writing logs to a database according to each embodiment of the present invention.

[0122] If the modules / units integrated in the computer device 1 are implemented in the form of software function units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned method embodiments of the present invention, it may also be completed by a computer program instructing relevant hardware devices. The computer program may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments may be implemented.

[0123] Among them, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory, etc.

[0124] Furthermore, the computer-readable storage medium mainly includes a storage program area and a storage data area. Among them, the storage program area may store an operating system, application programs required for at least one function, etc.; the storage data area may store data created according to the use of the blockchain node, etc.

[0125] The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, in essence, is a decentralized database, a series of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, an application service layer, etc.

[0126] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, in Figure 3 it is only represented by a single straight line, but it does not mean that there is only one bus or one type of bus. The bus is arranged to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0127] Although not shown, the computer device 1 may further include a power source (such as a battery) for supplying power to each component. Preferably, the power source can be logically connected to the at least one processor 13 through a power management device, so as to realize functions such as charge management, discharge management, and power consumption management through the power management device. The power source may also include any components such as one or more DC or AC power sources, a recharge device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The computer device 1 may also include a variety of sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0128] Furthermore, the computer device 1 may further include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the computer device 1 and other computer devices.

[0129] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display the information processed in the computer device 1 and to display a visual user interface.

[0130] It should be understood that the above embodiments are only for illustration purposes and are not limited by this structure in the scope of the patent application.

[0131] Those skilled in the art can understand that Figure 3 the shown structure does not constitute a limitation on the computer device 1, and it may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0132] In combination with Figure 1 , the memory 12 in the computer device 1 stores a plurality of instructions to implement a method for concurrently writing logs into a database, and the processor 13 can execute the plurality of instructions to implement: In response to a log concurrent write instruction for a target database, use the daemon process in the target database to create a plurality of parallel pipes and log processes corresponding to each pipe; Start each log process in sequence, and use each log process to concurrently listen to the corresponding pipe; Use other processes in the target database to generate logs, and send the generated logs to the plurality of pipes; When any log process listens that a target log is sent to the corresponding pipe, use the any log process to write the target log into the target database.

[0133] Specifically, for the specific implementation method of the above instructions by the processor 13, reference may be made to Figure 1 the description of the relevant steps in the corresponding embodiments, which will not be elaborated here.

[0134] It should be noted that all the data involved in this case are legally obtained. The non-company software tools or components appearing in the embodiments of this application are only for illustrative introduction and do not represent actual use.

[0135] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation.

[0136] The present invention can be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0137] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, in each embodiment of the present invention, the functional modules can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a hardware plus software functional module.

[0139] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0140] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any associated drawing marks in the claims should not be regarded as limiting the claims involved.

[0141] In addition, it is obvious that the term "comprising" does not exclude other units or steps, and the singular does not exclude the plural. A plurality of units or devices described in the present invention can also be implemented by one unit or device through software or hardware. Terms such as first, second, etc. are used to denote names and do not denote any particular order.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for concurrently writing logs into a database, characterized in that: The method for concurrently writing logs into a database includes: In response to a concurrent log write instruction to a target database, a plurality of parallel pipelines and a log process corresponding to each pipeline are created by using a daemon process in the target database; Start each log process in turn, and use each log process to listen to the corresponding pipeline in parallel; Using other processes in the target database to generate logs, and sending the generated logs to the multiple pipelines; When any log process monitors that a target log is sent to a corresponding pipeline, the target log is written into the target database using the any log process.

2. The method for concurrently writing logs into a database as claimed in claim 1, characterized in that: The method of using the daemon process in the target database to create multiple parallel pipelines and a log process corresponding to each pipeline includes: Get the number of configured log processes; Using the daemon process to create the pipeline according to the number of log processes, and configuring the pipeline number of each pipeline according to the pipeline creation order; The daemon process is used to create a log process corresponding to each pipeline, and the pipeline number of each pipeline is sent to the corresponding log process.

3. The method for concurrently writing logs into a database as claimed in claim 2, characterized in that: After creating a plurality of parallel pipelines and a log process corresponding to each pipeline by using the daemon process in the target database, the method further comprises: When the number of the log processes increases, obtaining the increase amount of the log processes; The daemon process is used to create new pipelines and log processes corresponding to each new pipeline according to the increase.

4. The method for concurrently writing logs into a database according to claim 1, characterized in that: Before creating a plurality of parallel pipelines and a log process corresponding to each pipeline by using the daemon process in the target database, the method further comprises: Detect whether there is a log file directory in the target database; When it is detected that the target database does not have the log file directory, creating the log file directory using the daemon process; The log file directory is used to store log files created by each log process.

5. The method for concurrently writing logs into a database according to claim 1, characterized in that: Starting each log process in sequence includes: For each log process, when the log process receives the pipe number sent by the daemon process, the pipe number is stored in the memory, a log file is created according to the pipe number, and the log file switching time is configured; Wherein, the log file number of the log file is the same as the pipeline number; After a log process is started, the next log process is started according to the log file number.

6. The method for concurrently writing logs into a database as claimed in claim 2, characterized in that: The sending the generated logs to the multiple pipelines includes: Obtain the process number of each other process and the number of log processes; Calculate the quotient of each process number and the number of log processes to obtain the quotient value corresponding to each other process; Perform a modulo operation on each quotient value to obtain the target value corresponding to each other process; Match each target value as a pipe number from said pipe to the target pipe corresponding to each other process; Send the logs generated by each other process to the corresponding target pipe.

7. The method for concurrently writing logs into a database according to claim 1, characterized in that: The using the arbitrary log process to write the target log into the target database comprises: Waking up the arbitrary log process, and using the arbitrary log process to detect whether the configuration file needs to be reloaded; When it is detected that the configuration file needs to be reloaded, the updated configuration file is loaded as the target configuration file; Detecting whether the log file switching time stored in the target configuration file changes; When a change in the log file switching time stored in the target configuration file is detected, the log file switching time stored in the target configuration file is determined as the target log file switching time, and the original log file switching time of any log process is configured as the target log file switching time; Get the current timestamp; When the current timestamp is the same as or after the target log file switching time, the original log file corresponding to the arbitrary log process is closed by using the arbitrary log process, and a new log file is created as the target log file; wherein the target log file and the original log file have the same log file number, and the target log file and the original log file are distinguished according to the creation time; The target log in the corresponding pipeline is read by using the arbitrary log process, and the target log is written into the target log file.

8. A device for concurrently writing logs into a database, characterized in that: The device for concurrently writing logs into a database comprises: A creation unit, configured to create a plurality of parallel pipelines and a log process corresponding to each pipeline by using a daemon process in the target database in response to a concurrent log write instruction to the target database; The startup unit is used to start each log process in sequence and use each log process to monitor the corresponding pipeline in parallel; A sending unit, configured to generate logs by using other processes in the target database, and send the generated logs to the multiple pipelines; The writing unit is used to write the target log into the target database by using any log process when any log process monitors that a target log is sent to a corresponding pipeline.

9. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the method for concurrently writing logs into a database as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the method for concurrently writing logs into a database as described in any one of claims 1 to 7.

Images