A method and device for processing log files

Abstract

The present application relates to a method and device for processing a log file, wherein the method comprises: sending a first log task obtained by parsing a first log file to a first hash bucket included in a hash bucket of an application thread, wherein the hash bucket of the application thread comprises a first hash bucket and a second hash bucket, and the application thread is used to perform an application operation on the hash bucket currently corresponding to the application thread; in the case of determining that the hash bucket currently corresponding to the application thread is the first hash bucket, notifying the application thread to apply the first log task in the first hash bucket; sending the second log task obtained by parsing a second log file to the second hash bucket. The present application solves the technical problem of low efficiency in processing log files.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method and device for processing log files. Background Art

[0002] When physically copying the database InnoDB, the coordination thread parses the log file and sends the parsed log task to the application thread. The application thread applies the log task in its hash table. In the existing technical solution, when the coordination thread parses the log file, the application thread needs to be in a blocked state, and when the application thread applies the log task, the coordination thread needs to be in a blocked state, which causes the application thread and the coordination thread to wait alternately, affecting the processing efficiency of the system.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The present application provides a method and device for processing a log file, so as to at least solve the technical problem of low efficiency in processing log files in the related art.

[0005] According to one aspect of an embodiment of the present application, a method for processing a log file is provided, comprising:

[0006] Sending a first log task obtained by parsing the first log file to a first hash bucket included in a hash bucket of an application thread, wherein the hash bucket of the application thread includes the first hash bucket and a second hash bucket, and the application thread is used to perform an application operation on the hash bucket currently corresponding to the application thread;

[0007] In a case where it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notifying the application thread to apply the first log task in the first hash bucket;

[0008] The second log task obtained by parsing the second log file is sent to the second hash bucket.

[0009] Optionally, when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notifying the application thread to apply the first log task in the first hash bucket includes:

[0010] Change the hash bucket write identifier from the first identifier to the second identifier, wherein the hash bucket write identifier is the first identifier used to indicate that the hash bucket of the current log task to be written is the first hash bucket, and the hash bucket write identifier is the second identifier used to indicate that the hash bucket of the current log task to be written is the second hash bucket;

[0011] Reading a hash bucket read identifier, wherein the hash bucket read identifier is used to indicate the hash bucket currently corresponding to the application thread;

[0012] When the hash bucket read identifier is the first identifier, it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0013] Optionally, changing the hash bucket write identifier from the first identifier to the second identifier includes: in the process of changing the hash bucket write identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the application thread is not allowed to perform an operation on the current operation object during the process of performing an operation on the current operation object, and the current operation object includes the hash bucket write identifier;

[0014] Reading the hash bucket read identifier includes: in the process of reading the hash bucket read identifier, calling the mutex lock, wherein the current operation object includes the hash bucket read identifier.

[0015] Optionally, before notifying the application thread to apply the first log task in the first hash bucket, the method further includes:

[0016] When it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket, blocking the parsing operation on the second log file;

[0017] Wait for the application thread until it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0018] Optionally, waiting for the application thread until determining that the hash bucket currently corresponding to the application thread is the first hash bucket includes:

[0019] Waiting for the application thread;

[0020] receiving a wake-up message sent by the application thread, wherein the wake-up message is used to wake up a parsing operation on the log file;

[0021] Determine, according to the wake-up message, that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0022] According to another aspect of the embodiment of the present application, another method for processing a log file is provided, including:

[0023] Receive a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket, where the first log task is obtained by the coordination thread parsing a first log file and sent to the first hash bucket;

[0024] In response to the first notification message, applying the first log task in the first hash bucket;

[0025] When the application of the first log task is completed, a second notification message sent by the coordination thread is received, wherein the second notification message is used to notify the application of the second log task in the second hash bucket, and the second log task is obtained by the coordination thread parsing the second log file and sent to the second hash bucket;

[0026] In response to the second notification message, the second log task in the second hash bucket is applied.

[0027] Optionally, when the application of the first log task is completed, receiving a second notification message sent by the coordination thread includes:

[0028] Change the hash bucket read identifier from the first identifier to the second identifier, wherein the hash bucket read identifier is the first identifier used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is the second identifier used to indicate that the hash bucket of the current log task to be read is the second hash bucket;

[0029] Reading a hash bucket write flag, wherein the hash bucket write flag is used to indicate the hash bucket currently corresponding to the coordination thread;

[0030] When the hash bucket write identifier is the first identifier, sending a wake-up message to the coordination thread, wherein the wake-up message is used to wake up the parsing operation of the log file;

[0031] Receive the second notification message sent by the coordination thread in response to the wake-up message.

[0032] Optionally, after reading the hash bucket write identifier, the method further includes:

[0033] When the hash bucket write identifier is the second identifier, wait for the coordination thread until the second notification message is received.

[0034] Optionally, changing the hash bucket read identifier from the first identifier to the second identifier includes: in the process of changing the hash bucket read identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the coordination thread is not allowed to perform an operation on the current operation object during the process of performing an operation on the current operation object, and the current operation object includes the hash bucket read identifier;

[0035] Reading the hash bucket write identifier includes: in the process of reading the hash bucket write identifier, calling the mutex lock, wherein the current operation object includes the hash bucket write identifier.

[0036] According to another aspect of the embodiment of the present application, a log file processing system is also provided, including: a coordination thread, an application thread, a first hash bucket and a second hash bucket, wherein:

[0037] The coordination thread is used to: send the first log task obtained by parsing the first log file to the first hash bucket; when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notify the application thread to apply the first log task in the first hash bucket; send the second log task obtained by parsing the second log file to the second hash bucket; when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket, notify the application thread to apply the second log task in the second hash bucket;

[0038] The application thread is used to: apply the first log task in the first hash bucket in response to a notification from the coordination thread; and apply the second log task in the second hash bucket in response to a notification from the coordination thread.

[0039] Optionally, the coordination thread is used to:

[0040] Change the hash bucket write identifier from the first identifier to the second identifier, wherein the hash bucket write identifier is the first identifier used to indicate that the hash bucket of the current log task to be written is the first hash bucket, and the hash bucket write identifier is the second identifier used to indicate that the hash bucket of the current log task to be written is the second hash bucket;

[0041] Reading a hash bucket read identifier, wherein the hash bucket read identifier is used to indicate the hash bucket currently corresponding to the application thread;

[0042] When the hash bucket read identifier is the first identifier, it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0043] Optionally, the coordination thread is used to:

[0044] In the process of changing the hash bucket write identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the application thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket write identifier;

[0045] In the process of reading the hash bucket read identifier, the mutex lock is called, wherein the current operation object includes the hash bucket read identifier.

[0046] Optionally, the application thread is used to:

[0047] Receive a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket;

[0048] In response to the first notification message, applying the first log task in the first hash bucket;

[0049] When the application of the first log task is completed, receiving a second notification message sent by the coordination thread, wherein the second notification message is used to notify the application of the second log task in the second hash bucket;

[0050] In response to the second notification message, the second log task in the second hash bucket is applied.

[0051] Optionally, the application thread is used to: change a hash bucket read identifier from a first identifier to a second identifier, wherein the hash bucket read identifier is the first identifier and is used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is the second identifier and is used to indicate that the hash bucket of the current log task to be read is the second hash bucket; read a hash bucket write identifier, wherein the hash bucket write identifier is used to indicate the hash bucket currently corresponding to the coordination thread; when the hash bucket write identifier is the first identifier, send a wake-up message to the coordination thread, wherein the wake-up message is used to wake up the parsing operation of the log file; receive the second notification message sent by the coordination thread in response to the wake-up message;

[0052] The coordination thread is used to: block the parsing operation on the second log file when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket; wait for the application thread; receive the wake-up message sent by the application thread; determine that the hash bucket currently corresponding to the application thread is the first hash bucket based on the wake-up message; and send the second notification message to the application thread in response to the wake-up message.

[0053] Optionally, the application thread is further used for:

[0054] After reading the hash bucket write identifier, if the hash bucket write identifier is the second identifier, wait for the coordination thread until the second notification message is received.

[0055] Optionally, the application thread is further configured to:

[0056] During the process of changing the hash bucket read identifier from the first identifier to the second identifier, call a mutex lock, where the mutex lock is used to ensure that the coordination thread is not allowed to perform an operation on the current operation object during the process of performing an operation on the current operation object, and the current operation object includes the hash bucket read identifier;

[0057] During the process of reading the hash bucket write identifier, call the mutex lock, where the current operation object includes the hash bucket write identifier.

[0058] According to another aspect of the embodiments of the present application, there is also provided a processing device for a log file, including:

[0059] A first distribution module, configured to distribute a first log task obtained by parsing a first log file to a first hash bucket included in the hash bucket of the application thread, where the hash bucket of the application thread includes the first hash bucket and a second hash bucket, and the application thread is configured to perform an application operation on the hash bucket currently corresponding to the application thread;

[0060] A notification module, configured to notify the application thread to apply the first log task in the first hash bucket when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket;

[0061] A second distribution module, configured to distribute a second log task obtained by parsing a second log file to the second hash bucket.

[0062] According to another aspect of the embodiments of the present application, there is also provided another processing device for a log file, including:

[0063] A first receiving module, configured to receive a first notification message sent by a coordination thread, where the first notification message is used to notify to apply a first log task in a first hash bucket, and the first log task is obtained by the coordination thread parsing the first log file and distributed to the first hash bucket;

[0064] A first application module, configured to apply the first log task in the first hash bucket in response to the first notification message;

[0065] A second receiving module, configured to receive a second notification message sent by the coordination thread when the application of the first log task is completed, where the second notification message is used to notify to apply a second log task in a second hash bucket, and the second log task is obtained by the coordination thread parsing the second log file and distributed to the second hash bucket;

[0066] The second application module is used to apply the second log task in the second hash bucket in response to the second notification message.

[0067] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above method is executed when the program is run.

[0068] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above method through the computer program.

[0069] In an embodiment of the present application, a first log task obtained by parsing a first log file is sent to a first hash bucket included in a hash bucket of an application thread, wherein the hash bucket of the application thread includes the first hash bucket and the second hash bucket, and the application thread is used to perform application operations on the hash bucket currently corresponding to the application thread; when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, the application thread is notified to apply the first log task in the first hash bucket; the second log task obtained by parsing the second log file is sent to the second hash bucket, and the hash bucket of the application thread includes the first hash bucket and the second hash bucket. After the first log task obtained by parsing the first log file is sent to the first hash bucket, the application thread is determined to be The hash bucket currently corresponding to the thread, if the application thread currently corresponds to the first hash bucket to which the first log task has been issued, then the application thread is notified to apply the first log task in the first hash bucket. While the application thread applies the first log task in the first hash bucket, the second log file continues to be parsed and the obtained second log task is issued to the second hash bucket, thereby achieving the purpose of parallel execution of the parsing and issuing of the log file and the application process of the log task, so that the parsing and issuing of the log file and the application of the log task do not need to block and wait for each other, saving processing time, thereby achieving the technical effect of improving the processing efficiency of the log file, and then solving the technical problem of low processing efficiency of the log file. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] Figure 1 is a schematic diagram of a hardware environment of a log file processing method according to an embodiment of the present application;

[0073] Figure 2 is a flowchart of an optional log file processing method according to an embodiment of the present application;

[0074] Figure 3 is a schematic diagram of another optional log file processing process according to an embodiment of the present application;

[0075] Figure 4 is a flowchart of another optional log file processing method according to an embodiment of the present application;

[0076] Figure 5 is a schematic diagram of another optional log file processing process according to an embodiment of the present application;

[0077] Figure 6 is a flowchart of another optional log file processing system according to an embodiment of the present application;

[0078] Figure 7 is a schematic diagram of a log file processing system according to an optional implementation mode of the present application;

[0079] Figure 8 is a schematic diagram of a control process of a hash bucket according to an optional implementation mode of the present application;

[0080] Fig. 9 is a schematic diagram of an optional log file processing device according to an embodiment of the present application;

[0081] Fig.10 is a schematic diagram of another optional log file processing device according to an embodiment of the present application;

[0082] Fig.11 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0084] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0085] According to one aspect of the embodiments of the present application, a method embodiment for processing a log file is provided.

[0086] Optionally, in this embodiment, the log file processing method can be applied to Figure 1 In the hardware environment shown in FIG. 1 , a first server 101, a second server 103 and a third server 105 are formed. Figure 1 As shown, the second server 103 is connected to the first server 101 and the third server 105 respectively through the network, and the above network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the first server 101, the second server 103 and the third server 105 are not limited to the server devices in the database InnoDB. The log file processing method of the embodiment of the present application can be executed by the first server 101, or by the third server 105, or by the first server 101 and the third server 105. Among them, the first server 101 can be used but not limited to run the coordination thread in the embodiment of the present application, the second server 103 is used to deploy the first hash bucket and the second hash bucket, and the third server 105 can be used but not limited to run the application thread in the embodiment of the present application.

[0087] Figure 2 is a flowchart of an optional log file processing method according to an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps:

[0088] Step S202: Send the first log task obtained by parsing the first log file to the first hash bucket included in the hash bucket of the application thread, wherein the hash bucket of the application thread includes the first hash bucket and the second hash bucket, and the application thread is used to perform an application operation on the hash bucket currently corresponding to the application thread;

[0089] Step S204: when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notify the application thread to apply the first log task in the first hash bucket;

[0090] Step S206: Send the second log task obtained by parsing the second log file to the second hash bucket.

[0091] Through the above steps S202 to S206, the hash bucket of the application thread includes a first hash bucket and a second hash bucket. After the first log task obtained by parsing the first log file is sent to the first hash bucket, the hash bucket currently corresponding to the application thread is determined. If the application thread currently corresponds to the first hash bucket to which the first log task has been sent, the application thread is notified to apply the first log task in the first hash bucket. While the application thread applies the first log task in the first hash bucket, the second log file continues to be parsed and the obtained second log task is sent to the second hash bucket, thereby achieving the purpose of parallel execution of the parsing and sending of the log file and the application process of the log task, saving processing time, thereby achieving the technical effect of improving the processing efficiency of the log file, and further solving the technical problem of low processing efficiency of the log file.

[0092] Optionally, in this embodiment, the above-mentioned log file processing method can be but is not limited to being applied to the coordination thread (i.e., coordinator thread, Apply_coordinator_thread) in the physical replication process of InnoDB, which is used to read and parse the redo log file from the transaction log group (Ib_logfile_group), and then send the obtained log task to the application thread (i.e., Log worker thread), which applies the log task.

[0093] In the technical solution provided in step S202, the above-mentioned first log file may include, but is not limited to, a redo log file in the database InnoDB, and the redo log file is used to record modifications to physical files in InnoDB. All modifications to physical files in InnoDB need to be protected by redo so that the database can recover from a crash. Optionally, in this embodiment, the hash bucket corresponding to the application thread may include, but is not limited to, two hash buckets, a first hash bucket and a second hash bucket. A set of hash tables sys_hash_X and user_hash_X may be configured for each hash bucket, wherein sys_hash_X is used to store changes to the system tablespace, and user_hash_X is used to store changes to the user tablespace, for example: the first hash bucket includes a set of hash tables sys_hash_0 and user_hash_0, and the second hash bucket includes a set of hash tables sys_hash_1 and user_hash_1.

[0094] Optionally, in this embodiment, the parsing process of the log file may include, but is not limited to, parsing information such as the file type, header file, and file length of the log file, and then generating a log task based on the information such as the file type, header file, and file length of the log file. In the technical solution provided in step S204, if it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, it can be determined that the application thread is currently not performing other application operations, and the application thread is currently in a state that allows operations on the first hash bucket, and the coordination thread can instruct the application thread to apply the first log task in the first hash bucket.

[0095] Optionally, in this embodiment, the method of notifying the application thread to apply the first log task in the first hash bucket may include, but is not limited to, sending a notification message to the application thread.

[0096] In the technical solution provided in step S206, when the application thread applies the first log task in the first hash bucket, the second log file continues to be parsed, and the second log task obtained after the parsing is sent to the second hash bucket.

[0097] Optionally, in this embodiment, after the above step S206, when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket, that is, when the application thread has completed the application of the first log task, the application thread is notified to apply the second log task in the second hash bucket.

[0098] As an optional embodiment, when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notifying the application thread to apply the first log task in the first hash bucket includes:

[0099] S11, changing the hash bucket write identifier from the first identifier to the second identifier, wherein the hash bucket write identifier is the first identifier used to indicate that the hash bucket of the current log task to be written is the first hash bucket, and the hash bucket write identifier is the second identifier used to indicate that the hash bucket of the current log task to be written is the second hash bucket;

[0100] S12, reading a hash bucket read identifier, wherein the hash bucket read identifier is used to indicate the hash bucket currently corresponding to the application thread;

[0101] S13, when the hash bucket read identifier is the first identifier, determining that the hash bucket currently corresponding to the application thread is the first hash bucket;

[0102] S14: Notify the application thread to apply the first log task in the first hash bucket.

[0103] Optionally, in this embodiment, the hash bucket write identifier is used to indicate the hash bucket of the current log task to be written, and the hash bucket write identifier can be but is not limited to being recorded as a global variable hash_to_write. Different hash buckets can correspond to different values ​​of the hash bucket write identifier. The hash bucket write identifier is a first identifier used to indicate that the hash bucket of the current log task to be written is the first hash bucket, and the hash bucket write identifier is a second identifier used to indicate that the hash bucket of the current log task to be written is the second hash bucket. The first identifier and the second identifier can be two different numbers, characters, strings, etc. For example: the first identifier can be 0, and the second identifier can be 1, that is, the hash bucket write identifier is 0 to indicate that the hash bucket of the current log task to be written is the first hash bucket, and the hash bucket write identifier is 1 to indicate that the hash bucket of the current log task to be written is the second hash bucket.

[0104] Optionally, in this embodiment, the hash bucket read identifier is used to indicate the hash bucket to which the application thread currently corresponds, and the hash bucket read identifier can be, but is not limited to, recorded as a global variable hash_to_read. Different hash buckets can correspond to different values ​​of the hash bucket read identifier. The hash bucket read identifier is a first identifier used to indicate that the hash bucket to which the application thread currently corresponds is the first hash bucket, and the hash bucket read identifier is a second identifier used to indicate that the hash bucket to which the application thread currently corresponds is the second hash bucket. For example: a hash bucket read identifier of 0 is used to indicate that the hash bucket to which the application thread currently corresponds is the first hash bucket, and a hash bucket read identifier of 1 is used to indicate that the hash bucket to which the application thread currently corresponds is the second hash bucket.

[0105] Optionally, in this embodiment, the coordination thread is allowed to perform a change operation on the hash bucket write flag and perform a read operation on the hash bucket read flag. The application thread is allowed to perform a change operation on the hash bucket read flag and perform a read operation on the hash bucket write flag.

[0106] Optionally, in this embodiment, the concurrent execution of the coordination thread and the application thread can be ensured by controlling the inequality of the hash bucket read identifier and the hash bucket write identifier. For example: after changing the hash bucket write identifier from the first identifier to the second identifier, the hash bucket read identifier is read. If the hash bucket read identifier is the first identifier, the hash bucket read identifier is unequal to the hash bucket write identifier, then it can be determined that the hash buckets currently corresponding to the coordination thread and the application thread are different hash buckets. The application thread can be notified to perform a read operation on the first hash bucket, and at the same time, the write operation can continue to be performed on the second hash bucket, thereby realizing the parallelism of the coordination thread and the application thread.

[0107] Optionally, in this embodiment, the initial values ​​of the hash bucket write identifier and the hash bucket read identifier can both be set to 0. After the coordination thread completes the operation of sending the first log task to the first hash bucket, the hash bucket write identifier is changed from 0 to 1, and the coordination thread reads the value of the hash bucket read identifier. When the value of the hash bucket read identifier is 0, the application thread is notified to apply the first log task in the first hash bucket.

[0108] Through the above steps, after completing the parsing and sending operations of the first task file, the hash bucket write identifier is changed from the first identifier to the second identifier, and the hash bucket read identifier is read to determine the working status of the application thread. When the hash bucket read identifier is the first identifier, the application thread is notified to apply the first log task in the first hash bucket, thereby realizing parallel processing of the log file parsing and sending process and the log task application process, thereby improving the processing efficiency of the log file.

[0109] As an optional embodiment, changing the hash bucket write identifier from the first identifier to the second identifier includes:

[0110] S21, in the process of changing the hash bucket write identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the application thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket write identifier;

[0111] Reading the hash bucket read identifier includes:

[0112] S22: In the process of reading the hash bucket read identifier, calling the mutex lock, wherein the current operation object includes the hash bucket read identifier.

[0113] Optionally, in this embodiment, the mutex (worker_mutex) is used to ensure that the application thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object. In the process of changing the hash bucket write identifier from the first identifier to the second identifier, the current operation object includes the hash bucket write identifier, and in the process of reading the hash bucket read identifier, the current operation object includes the hash bucket read identifier.

[0114] Optionally, in this embodiment, the coordination thread's access to the hash bucket write flag and the hash bucket read flag may be protected by, but not limited to, a mutex (worker_mutex). After the coordination thread calls the mutex, the application thread cannot access the hash bucket write flag and the hash bucket read flag.

[0115] Optionally, in this embodiment, the situation where the coordination thread and the application thread complete the operation at the same time can be handled but is not limited to the mutual exclusion lock. That is, when the coordination thread and the application thread complete the parsing and application at the same time, whoever grabs the worker_mutex (mutex lock) first will be deemed to have completed first, and the corresponding operation will be executed.

[0116] Through the above steps, when the coordination thread operates on the hash bucket write flag or the hash bucket read flag, the mutex lock is called to ensure that the application thread does not perform other operations on the object currently being operated, thereby ensuring that when the coordination thread operates on the hash bucket write flag or the hash bucket read flag, the state of the hash bucket write flag or the hash bucket read flag is fixed.

[0117] As an optional embodiment, before notifying the application thread to apply the first log task in the first hash bucket, the method further includes:

[0118] S31, when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket, blocking the parsing operation on the second log file;

[0119] S32, waiting for the application thread until it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0120] Optionally, in this embodiment, the hash bucket currently corresponding to the application thread can be determined by, but not limited to, reading the hash bucket read identifier. For example, when the hash bucket read identifier is the second identifier, the hash bucket currently corresponding to the application thread is considered to be the second hash bucket.

[0121] Optionally, in this embodiment, if the coordination thread completes the parsing of the first log file and the issuance of the first log task first, but the application thread has not completed the application operation of the log task stored in the second hash bucket in the previous round, the next round of parsing operation on the second log file is blocked and the application thread is waited for to complete the application operation.

[0122] Through the above steps, when the coordination thread determines that the application thread corresponds to the second hash bucket, the coordination thread blocks the parsing operation on the second log file, waits until the hash bucket corresponding to the application thread is the first hash bucket, and then notifies the application thread to apply the first log task in the first hash bucket, and performs the parsing operation on the second log file, thereby realizing parallel processing of the coordination thread and the application thread.

[0123] As an optional embodiment, waiting for the application thread until it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket includes:

[0124] S41, waiting for the application thread;

[0125] S42, receiving a wake-up message sent by the application thread, wherein the wake-up message is used to wake up a parsing operation on the log file;

[0126] S43: Determine, according to the wake-up message, that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0127] Optionally, in this embodiment, the wake-up message is used to wake up the parsing operation of the log file, and the wake-up message may be, but is not limited to, a message sent by the application thread to the coordination thread using a preset transmission protocol.

[0128] Optionally, in this embodiment, the thread that completes the operation later can notify the thread that completes the operation earlier and is in a blocked state to start the next round of operations. For example, if the application thread completes the application operation later, the coordination thread that is in a blocked state can be notified through a wake-up message to continue the subsequent operation. If the coordination thread completes the parsing and issuing operations later, the application thread that is in a blocked state can be notified to continue the application operation.

[0129] The present application also provides an optional embodiment, which provides a processing method when the Coordinator (coordination thread) completes the parsing first and the Log worker (application thread) has not yet completed the application. Figure 3 is a schematic diagram of an optional log file processing process according to an embodiment of the present application, such as Figure 3 As shown, the process may include but is not limited to the following steps:

[0130] In step S302, the Coordinator first completes the log file parsing and sending operation, that is, the parsing (Parse) is completed.

[0131] Step S304, the Coordinator changes the hash bucket write flag (hash_to_write).

[0132] Step S306, the Coordinator reads the hash bucket read flag (hash_to_read), and when the values ​​of hash_to_read and hash_to_write are the same, it is determined that the Log worker has not completed the application.

[0133] Step S308, the Coordinator blocks subsequent operations and waits for the Log worker to complete the application.

[0134] Step S310, the Log worker completes the application operation of the log task in the hash bucket.

[0135] Step S312, Log worker changes the hash bucket read flag (hash_to_read).

[0136] Step S314, the Log worker reads the hash bucket write flag (hash_to_write) to determine the working status of the coordination thread. When the values ​​of hash_to_read and hash_to_write are different, it can be known that the Coordinator has completed the parsing operation of the log file.

[0137] Step S316, Log worker wakes up Coordinator.

[0138] Step S318, the Log worker blocks subsequent operations and waits for the Coordinator to complete subsequent work.

[0139] Step S320 , the Coordinator processes the completed application and performs internal event processing (checkpoint).

[0140] Step S322, the Coordinator notifies the Log worker to start applying and performing the next round of log file parsing operations.

[0141] Step S324: Log worker responds to the notification from the Coordinator to perform the next round of log file application operations.

[0142] According to another aspect of the embodiments of the present application, another embodiment of a method for transmitting multimedia resources is provided. Figure 4 is a flowchart of another optional log file processing method according to an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps:

[0143] Step S402, receiving a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket, wherein the first log task is obtained by the coordination thread by parsing a first log file and sent to the first hash bucket;

[0144] Step S404, applying the first log task in the first hash bucket in response to the first notification message;

[0145] Step S406, when the application of the first log task is completed, receiving a second notification message sent by the coordination thread, wherein the second notification message is used to notify the application of the second log task in the second hash bucket, and the second log task is obtained by the coordination thread parsing the second log file and sent to the second hash bucket;

[0146] Step S408: Apply the second log task in the second hash bucket in response to the second notification message.

[0147] Through the above steps S402 to S408, the first log task in the first hash bucket is applied in response to the first notification message sent by the coordination thread. While executing the application operation, the parsing thread will continue to parse the second log file and send the obtained second log task to the second hash bucket. If the application of the first log task is completed, the second notification message sent by the coordination thread can be directly responded to for applying the second log task in the second hash bucket, thereby avoiding the blocking waiting process after the application of the log task, achieving the purpose of parallel execution of the parsing and sending of the log file and the application process of the log task, saving processing time, thereby achieving the technical effect of improving the processing efficiency of the log file, and then solving the technical problem of low processing efficiency of the log file.

[0148] Optionally, in this embodiment, the above-mentioned log file processing method can be but is not limited to being applied to the application thread (Log worker thread) in the physical replication process of InnoDB. The coordination thread (Apply_coordinator_thread) in the physical replication process of InnoDB is used to read and parse the redo log file from the transaction log group (Ib_logfile_group), and then send the obtained log task to the application thread (Log worker thread), and the application thread applies the log task.

[0149] Optionally, in this embodiment, the above-mentioned application thread can, but is not limited to, perform corresponding application operations according to the notification message sent by the received coordination thread. If the notification message is used to notify the application of the first log task in the first hash bucket, the application thread applies the first log task in the first hash bucket. If the notification message is used to notify the application of the second log task in the second hash bucket, the application thread applies the second log task in the second hash bucket. In other words, the application thread applies the log task according to the instruction of the coordination thread.

[0150] Optionally, in this embodiment, the application process of the log task may be, but is not limited to, alternately executed on the first hash bucket and the second hash bucket, thereby achieving parallel execution of the log file parsing and log task delivery process and the log task application process.

[0151] As an optional embodiment, when the application of the first log task is completed, receiving the second notification message sent by the coordination thread includes:

[0152] S51, changing the hash bucket read identifier from the first identifier to the second identifier, wherein the hash bucket read identifier is the first identifier used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is the second identifier used to indicate that the hash bucket of the current log task to be read is the second hash bucket;

[0153] S52, reading a hash bucket write flag, wherein the hash bucket write flag is used to indicate the hash bucket currently corresponding to the coordination thread;

[0154] S53: When the hash bucket write identifier is the first identifier, sending a wake-up message to the coordination thread, wherein the wake-up message is used to wake up the parsing operation of the log file;

[0155] S54: Receive the second notification message sent by the coordination thread in response to the wake-up message.

[0156] Optionally, in this embodiment, the hash bucket read identifier is used to indicate the hash bucket of the current log task to be read, and the hash bucket read identifier can be but is not limited to being recorded as a global variable hash_to_read. Different hash buckets can correspond to different values ​​of the hash bucket read identifier. The hash bucket read identifier is a first identifier used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is a second identifier used to indicate that the hash bucket of the current log task to be read is the second hash bucket. The first identifier and the second identifier can be two different numbers, characters, strings, etc. For example: the first identifier can be 0, and the second identifier can be 1, that is, the hash bucket read identifier is 0 to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is 1 to indicate that the hash bucket of the current log task to be read is the second hash bucket.

[0157] Optionally, in this embodiment, the hash bucket write flag is used to indicate the hash bucket to which the coordination thread currently corresponds, and the hash bucket write flag can be, but is not limited to, recorded as a global variable hash_to_write. Different hash buckets can correspond to different values ​​of the hash bucket write flag. The hash bucket write flag is a first flag used to indicate that the hash bucket to which the coordination thread currently corresponds is the first hash bucket, and the hash bucket write flag is a second flag used to indicate that the hash bucket to which the coordination thread currently corresponds is the second hash bucket. For example: a hash bucket write flag of 0 is used to indicate that the hash bucket to which the coordination thread currently corresponds is the first hash bucket, and a hash bucket write flag of 1 is used to indicate that the hash bucket to which the coordination thread currently corresponds is the second hash bucket.

[0158] Optionally, in this embodiment, the coordination thread is allowed to perform a change operation on the hash bucket write flag and perform a read operation on the hash bucket read flag. The application thread is allowed to perform a change operation on the hash bucket read flag and perform a read operation on the hash bucket write flag.

[0159] Optionally, in this embodiment, the concurrent execution of the coordination thread and the application thread can be ensured by controlling the inequality of the hash bucket read identifier and the hash bucket write identifier. For example, after changing the hash bucket read identifier from the first identifier to the second identifier, the hash bucket write identifier is read. If the hash bucket write identifier is the first identifier, the hash bucket read identifier is unequal to the hash bucket write identifier, then it can be determined that the hash buckets currently corresponding to the coordination thread and the application thread are different hash buckets. A wake-up message can be sent to the coordination thread to wake up the subsequent operations of the coordination thread.

[0160] Optionally, in this embodiment, the initial values ​​of the hash bucket write flag and the hash bucket read flag can both be set to 0. After the application thread completes the application operation of the first log task, the hash bucket read flag is changed from 0 to 1, and the application thread reads the value of the hash bucket write flag. When the value of the hash bucket write flag is 0, the coordination thread is awakened.

[0161] As an optional embodiment, after reading the hash bucket write mark, the method further includes:

[0162] S61: When the identifier written into the hash bucket is the second identifier, wait for the coordination thread until the second notification message is received.

[0163] Optionally, in this embodiment, when the application thread reads the hash bucket write identifier as the second identifier, the subsequent operations of the application thread can be in a blocked state until the second notification message sent by the coordination thread is received and the second log task is applied according to the instructions of the second notification message.

[0164] Optionally, in this embodiment, the situation where the hash bucket is written with the second identifier can be considered as the application thread completing the previous round of application first, but the coordination thread has not completed the subsequent parsing. Then, the thread that completes the operation first can enter a blocked state and wait for notification from the thread that completes the operation later to start the subsequent operation process.

[0165] As an optional embodiment, changing the hash bucket read identifier from the first identifier to the second identifier includes:

[0166] S71, in the process of changing the hash bucket read identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the coordination thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket read identifier;

[0167] Reading hash bucket write flags includes:

[0168] S72: In the process of reading the hash bucket write identifier, calling the mutex lock, wherein the current operation object includes the hash bucket write identifier.

[0169] Optionally, in this embodiment, a mutex (worker_mutex) is used to ensure that the coordination thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object. In the process of changing the hash bucket read identifier from the first identifier to the second identifier, the current operation object includes the hash bucket read identifier, and in the process of reading the hash bucket write identifier, the current operation object includes the hash bucket write identifier.

[0170] Optionally, in this embodiment, the application thread's access to the hash bucket write flag and the hash bucket read flag may be protected by, but not limited to, a mutex (worker_mutex). When the application thread calls the mutex, the coordination thread cannot access the hash bucket write flag and the hash bucket read flag.

[0171] Optionally, in this embodiment, the situation where the coordination thread and the application thread complete the operation at the same time can be handled but is not limited to the mutual exclusion lock. That is, when the coordination thread and the application thread complete the parsing and application at the same time, whoever grabs the worker_mutex (mutex lock) first will be deemed to have completed first, and the corresponding operation will be executed.

[0172] Through the above steps, when the application thread operates on the hash bucket write flag or the hash bucket read flag, the mutual exclusion lock is called to ensure that the coordination thread does not perform other operations on the object currently being operated, thereby ensuring that when the application thread operates on the hash bucket write flag or the hash bucket read flag, the state of the hash bucket write flag or the hash bucket read flag is fixed.

[0173] The present application also provides another optional embodiment, which provides a processing method when the Log worker (application thread) completes the application first and the Coordinator (coordinator thread) has not completed the parsing. Figure 5 is a schematic diagram of another optional log file processing process according to an embodiment of the present application, such as Figure 5As shown, the process may include but is not limited to the following steps:

[0174] Step S502, the Log worker completes the application operation of the log task in the hash bucket.

[0175] Step S504, the Log worker changes the hash bucket read flag (hash_to_read).

[0176] Step S506, the Log worker reads the hash bucket write flag (hash_to_write) to determine the working status of the coordination thread. When the values ​​of hash_to_read and hash_to_write are the same, it can be known that the Coordinator has not completed the parsing operation of the log file.

[0177] Step S508, the Log worker blocks subsequent operations and waits for the Coordinator to complete subsequent work.

[0178] Step S510, the Coordinator completes the log file parsing and sending operation, that is, Parse is completed.

[0179] Step S512, the Coordinator changes the hash bucket write flag (hash_to_write).

[0180] Step S514, the Coordinator reads the hash bucket read flag (hash_to_read), and when the values ​​of hash_to_read and hash_to_write are different, it is determined that the Log worker has completed the application.

[0181] Step S516, the Coordinator processes the completed application and performs internal event processing (checkpoint).

[0182] Step S518, the Coordinator notifies the Log worker to start applying and performing the next round of log file parsing operations.

[0183] Step S520: Log worker responds to the notification from the Coordinator to perform the next round of log file application operation.

[0184] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for an electronic device (which can be a mobile phone, computer, server, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0186] According to another aspect of the embodiments of the present application, a log file processing system is also provided. Figure 6 is a flowchart of another optional log file processing system according to an embodiment of the present application, such as Figure 6 As shown, the system may include: a coordination thread 62, an application thread 64, a first hash bucket 66 and a second hash bucket 68, wherein:

[0187] The coordination thread 62 is used to: send the first log task obtained by parsing the first log file to the first hash bucket 66; when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket 66, notify the application thread to apply the first log task in the first hash bucket 66; send the second log task obtained by parsing the second log file to the second hash bucket 68; when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket 68, notify the application thread to apply the second log task in the second hash bucket 68;

[0188] The application thread 64 is used to: apply the first log task in the first hash bucket 66 in response to the notification of the coordination thread; and apply the second log task in the second hash bucket 68 in response to the notification of the coordination thread.

[0189] Optionally, in this embodiment, the application thread may be, but is not limited to, an application thread group, the application thread group includes multiple application threads, each application thread in the application thread group corresponds to a group of hash buckets, a first hash bucket and a second hash bucket. The coordination thread distributes log tasks to each application thread in the application thread group through a preset distribution strategy, and sends the log tasks to a group of hash buckets corresponding to the application thread in turn.

[0190] In an optional embodiment, a log file processing system architecture is provided. Figure 7 is a schematic diagram of a log file processing system according to an optional implementation mode of the present application, such as Figure 7 As shown, the system includes: a transaction log group (Ib_logfile_group), a coordination thread (i.e., coordinator thread, Apply_coordinator_thread), an application thread group, and the application thread group includes multiple application threads (i.e., Log worker thread, Log workerthread). Each Log worker thread corresponds to a set of hash buckets (hash_0 and hash_1), and each hash bucket includes two hash tables sys_hash and user_hash, wherein sys_hash is used to store changes in the system table space, and user_hash is used to store changes in the user table space. Apply_coordinator_thread is used to read (Read) redo log files from Ib_logfile_group, and parse and distribute (Parse&&distribute), and Log workerthread is used to apply log tasks stored in the hash bucket, and hash buckets hash_0 and hash_1 are used alternately.

[0191] As an optional embodiment, the coordination thread is used to:

[0192] Change the hash bucket write identifier from the first identifier to the second identifier, wherein the hash bucket write identifier is the first identifier used to indicate that the hash bucket of the current log task to be written is the first hash bucket, and the hash bucket write identifier is the second identifier used to indicate that the hash bucket of the current log task to be written is the second hash bucket;

[0193] Reading a hash bucket read identifier, wherein the hash bucket read identifier is used to indicate the hash bucket currently corresponding to the application thread;

[0194] When the hash bucket read identifier is the first identifier, it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0195] Optionally, in this embodiment, a corresponding set of hash tables sys_hash_1 and user_hash_1 is added to each log worker thread in the application thread group, and a global variable hash_to_read (hash bucket read identifier) ​​is added to identify the hash bucket corresponding to the log worker thread group, and a global variable hash_to_write (hash bucket write identifier) ​​is added to identify the hash bucket corresponding to the coordinator thread. The values ​​of hash_to_read and hash_to_write can be 0 and 1, 0 represents sys_hash_0 and user_hash_0, and 1 represents sys_hash_1 and user_hash_1. The concurrent execution of the coordinator thread and the log worker thread group is ensured by controlling the inequality of hash_to_read and hash_to_write values.

[0196] In an optional embodiment, a method of controlling the hash bucket by controlling hash_to_read and hash_to_write is provided. Figure 8 is a schematic diagram of a control process of a hash bucket according to an optional implementation mode of the present application, such as Figure 8 As shown, hash_to_write is used to identify the hash bucket currently corresponding to the coordinator thread, and hash_to_read is used to identify the hash bucket currently corresponding to the log worker thread. The hash bucket can be switched by the "1-x" expression. That is, the 1-hash_to_write expression is executed when the coordinator thread switches the hash bucket, and the 1-hash_to_read expression is executed when the log worker switches the hash bucket.

[0197] As an optional embodiment, the coordination thread is used to:

[0198] In the process of changing the hash bucket write identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the application thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket write identifier;

[0199] In the process of reading the hash bucket read identifier, the mutex lock is called, wherein the current operation object includes the hash bucket read identifier.

[0200] Optionally, in this embodiment, the access of the coordinator thread and the log worker thread to hash_to_read and hash_to_write can be protected by, but not limited to, the mutex worker_mutex. Even if the coordinator thread and the log worker thread complete the parsing and application at the same time, whoever grabs the worker_mutex (mutex) first will be deemed to have completed first, and the corresponding operation will be executed if it completes first.

[0201] As an optional embodiment, the application thread is used for:

[0202] Receive a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket;

[0203] In response to the first notification message, applying the first log task in the first hash bucket;

[0204] When the application of the first log task is completed, receiving a second notification message sent by the coordination thread, wherein the second notification message is used to notify the application of the second log task in the second hash bucket;

[0205] In response to the second notification message, the second log task in the second hash bucket is applied.

[0206] Optionally, in this embodiment, the application thread may, but is not limited to, execute the application operation of the log task according to the control of the coordination thread.

[0207] As an optional embodiment, the application thread is used to: change a hash bucket read identifier from a first identifier to a second identifier, wherein the hash bucket read identifier is the first identifier and is used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is the second identifier and is used to indicate that the hash bucket of the current log task to be read is the second hash bucket; read the hash bucket write identifier, wherein the hash bucket write identifier is used to indicate the hash bucket currently corresponding to the coordination thread; when the hash bucket write identifier is the first identifier, send a wake-up message to the coordination thread, wherein the wake-up message is used to wake up the parsing operation of the log file; receive the second notification message sent by the coordination thread in response to the wake-up message;

[0208] The coordination thread is used to: block the parsing operation on the second log file when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket; wait for the application thread; receive the wake-up message sent by the application thread; determine that the hash bucket currently corresponding to the application thread is the first hash bucket based on the wake-up message; and send the second notification message to the application thread in response to the wake-up message.

[0209] Optionally, in this embodiment, the coordination thread and the application thread that completes the operation first may enter a blocking state and wait for a notification from the other thread that completes the operation later to continue to perform further operations.

[0210] As an optional embodiment, the application thread is further used for:

[0211] After reading the hash bucket write identifier, if the hash bucket write identifier is the second identifier, wait for the coordination thread until the second notification message is received.

[0212] As an optional embodiment, the application thread is further used for:

[0213] In the process of changing the hash bucket read identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the coordination thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket read identifier;

[0214] In the process of reading the hash bucket write identifier, the mutex lock is called, wherein the current operation object includes the hash bucket write identifier.

[0215] According to another aspect of an embodiment of the present application, a log file processing device for implementing the above-mentioned log file processing method is also provided. Fig. 9 is a schematic diagram of an optional log file processing device according to an embodiment of the present application, such as Fig. 9 As shown, the device may include:

[0216] A first sending module 92 is used to send the first log task obtained by parsing the first log file to a first hash bucket included in the hash bucket of the application thread, wherein the hash bucket of the application thread includes the first hash bucket and the second hash bucket, and the application thread is used to perform an application operation on the hash bucket currently corresponding to the application thread;

[0217] a notification module 94, configured to notify the application thread to apply the first log task in the first hash bucket when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket;

[0218] The second sending module 96 is used to send the second log task obtained by parsing the second log file to the second hash bucket.

[0219] It should be noted that the first sending module 92 in this embodiment can be used to execute step S202 in the embodiment of the present application, the notification module 94 in this embodiment can be used to execute step S204 in the embodiment of the present application, and the second sending module 96 in this embodiment can be used to execute step S206 in the embodiment of the present application.

[0220] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware.

[0221] Through the above modules, the technical problem of low log file processing efficiency can be solved, thereby achieving the technical effect of improving the log file processing efficiency.

[0222] As an optional embodiment, the notification module includes:

[0223] A first changing unit, configured to change a hash bucket write identifier from a first identifier to a second identifier, wherein the hash bucket write identifier is the first identifier used to indicate that the hash bucket to which the current log task is to be written is the first hash bucket, and the hash bucket write identifier is the second identifier used to indicate that the hash bucket to which the current log task is to be written is the second hash bucket;

[0224] A first reading unit, configured to read a hash bucket read identifier, wherein the hash bucket read identifier is used to indicate the hash bucket currently corresponding to the application thread;

[0225] A first determining unit, configured to determine, when the hash bucket read identifier is the first identifier, that the hash bucket currently corresponding to the application thread is the first hash bucket;

[0226] A notification unit is used to notify the application thread to apply the first log task in the first hash bucket.

[0227] As an optional embodiment, the first changing unit is used to:

[0228] In the process of changing the hash bucket write identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the application thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket write identifier;

[0229] The reading unit is used to call the mutex lock during the process of reading the hash bucket reading identifier, wherein the current operation object includes the hash bucket reading identifier.

[0230] As an optional embodiment, the device further includes:

[0231] a blocking module, configured to block a parsing operation on the second log file when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket before notifying the application thread to apply the first log task in the first hash bucket;

[0232] The first waiting module is used to wait for the application thread until it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0233] As an optional embodiment, the first waiting module includes:

[0234] A waiting unit, used for waiting for the application thread;

[0235] A first receiving unit, configured to receive a wake-up message sent by the application thread, wherein the wake-up message is used to wake up a parsing operation on a log file;

[0236] The second determining unit is configured to determine, according to the wake-up message, that the hash bucket currently corresponding to the application thread is the first hash bucket.

[0237] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0238] According to another aspect of an embodiment of the present application, another log file processing device for implementing the above-mentioned log file processing method is also provided. Fig.10 is a schematic diagram of another optional log file processing device according to an embodiment of the present application, such as Fig.10 As shown, the device may include:

[0239] A first receiving module 102 is used to receive a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket, where the first log task is obtained by the coordination thread parsing the first log file and sent to the first hash bucket;

[0240] A first application module 104, configured to apply the first log task in the first hash bucket in response to the first notification message;

[0241] A second receiving module 106 is used to receive a second notification message sent by the coordination thread when the application of the first log task is completed, wherein the second notification message is used to notify the application of the second log task in the second hash bucket, and the second log task is obtained by the coordination thread parsing the second log file and sent to the second hash bucket;

[0242] The second application module 108 is configured to apply the second log task in the second hash bucket in response to the second notification message.

[0243] It should be noted that the first receiving module 102 in this embodiment can be used to execute step S402 in the embodiment of the present application, the first application module 104 in this embodiment can be used to execute step S404 in the embodiment of the present application, the second receiving module 106 in this embodiment can be used to execute step S406 in the embodiment of the present application, and the second application module 108 in this embodiment can be used to execute step S408 in the embodiment of the present application.

[0244] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware.

[0245] Through the above module, after obtaining the play request, data segment information for indicating that the first segment of data of the multimedia resource is allowed to be started is generated, and a data request is directly sent to the database according to the data segment information, and the data returned by the database is sent to the client for starting. This makes the start-up phase of the multimedia resource independent of Metadata, saves the time of requesting Metadata files from the Metadata server and parsing the Metadata files before requesting data, achieves the purpose of quickly starting the multimedia resources, and thus realizes the technical effect of improving the start-up speed of the multimedia resources, thereby solving the technical problem of the slow start-up speed of the multimedia resources in the related technology.

[0246] As an optional embodiment, the second receiving module includes:

[0247] A second changing unit is used to change a hash bucket read identifier from a first identifier to a second identifier, wherein the hash bucket read identifier is the first identifier used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is the second identifier used to indicate that the hash bucket of the current log task to be read is the second hash bucket;

[0248] A second reading unit is used to read a hash bucket write identifier, wherein the hash bucket write identifier is used to indicate the hash bucket currently corresponding to the coordination thread;

[0249] a sending unit, configured to send a wake-up message to the coordination thread when the identifier written into the hash bucket is the first identifier, wherein the wake-up message is used to wake up a parsing operation on the log file;

[0250] The second receiving unit is used to receive the second notification message sent by the coordination thread in response to the wake-up message.

[0251] As an optional embodiment, the device further includes:

[0252] The second waiting module is used to wait for the coordination thread after reading the hash bucket write identifier and when the hash bucket write identifier is the second identifier, until the second notification message is received.

[0253] As an optional embodiment, the second changing unit is used for:

[0254] In the process of changing the hash bucket read identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the coordination thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket read identifier;

[0255] The second reading unit is used to: call the mutex lock during the process of reading the hash bucket write identifier, wherein the current operation object includes the hash bucket write identifier.

[0256] According to another aspect of the embodiments of the present application, an electronic device for implementing the above-mentioned log file processing method is also provided.

[0257] Fig.11 is a structural block diagram of an electronic device according to an embodiment of the present application, such as Fig.11As shown, the electronic device may include: one or more (only one is shown in the figure) processors 1101, a memory 1103, and a transmission device 1105, such as Fig.11 As shown, the electronic device may further include an input / output device 1107 .

[0258] Among them, the memory 1103 can be used to store software programs and modules, such as the program instructions / modules corresponding to the processing method and device of the log file in the embodiment of the present application, and the processor 1101 executes various functional applications and data processing by running the software programs and modules stored in the memory 1103, that is, realizing the above-mentioned processing method of the log file. The memory 1103 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1103 may further include a memory remotely arranged relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0259] The above-mentioned transmission device 1105 is used to receive or send data via a network, and can also be used for data transmission between a processor and a memory. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 1105 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1105 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0260] Specifically, the memory 1103 is used to store application programs.

[0261] The processor 1101 may call the application stored in the memory 1103 through the transmission device 1105 to perform the following steps:

[0262] Sending a first log task obtained by parsing the first log file to a first hash bucket included in a hash bucket of an application thread, wherein the hash bucket of the application thread includes the first hash bucket and a second hash bucket, and the application thread is used to perform an application operation on the hash bucket currently corresponding to the application thread;

[0263] In a case where it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notifying the application thread to apply the first log task in the first hash bucket;

[0264] The second log task obtained by parsing the second log file is sent to the second hash bucket.

[0265] The processor 1101 may also call the application program stored in the memory 1103 through the transmission device 1105 to perform the following steps:

[0266] Receive a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket, where the first log task is obtained by the coordination thread parsing a first log file and sent to the first hash bucket;

[0267] In response to the first notification message, applying the first log task in the first hash bucket;

[0268] When the application of the first log task is completed, a second notification message sent by the coordination thread is received, wherein the second notification message is used to notify the application of the second log task in the second hash bucket, and the second log task is obtained by the coordination thread parsing the second log file and sent to the second hash bucket;

[0269] In response to the second notification message, the second log task in the second hash bucket is applied.

[0270] By adopting the embodiment of the present application, a log file processing scheme is provided. Through the coordination thread and the application thread working in parallel, after the coordination thread sends the first log task obtained by parsing the first log file to the first hash bucket of the application thread, the application thread performs an application operation on the first log task in the first hash bucket, and while the application thread performs the application operation on the first log task in the first hash bucket, the coordination thread sends the second log task obtained by parsing the second log file to the second hash bucket of the application thread, achieving the purpose of the coordination thread and the application thread working in parallel, thereby achieving the technical effect of improving the processing efficiency of the log file, and further solving the technical problem of low processing efficiency of the log file.

[0271] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0272] It can be understood by those skilled in the art that Fig.11 The structure shown is for illustration only, and the electronic device may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other electronic devices. Fig.11 The structure of the electronic device is not limited. Fig.11 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.11 Different configurations are shown.

[0273] A person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the electronic device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0274] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to execute the program code of the log file processing method.

[0275] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0276] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0277] Sending a first log task obtained by parsing the first log file to a first hash bucket included in a hash bucket of an application thread, wherein the hash bucket of the application thread includes the first hash bucket and a second hash bucket, and the application thread is used to perform an application operation on the hash bucket currently corresponding to the application thread;

[0278] In a case where it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notifying the application thread to apply the first log task in the first hash bucket;

[0279] The second log task obtained by parsing the second log file is sent to the second hash bucket.

[0280] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps:

[0281] Change the hash bucket read identifier from a first identifier to a second identifier, wherein the first identifier is used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the second identifier is used to indicate that the hash bucket of the current log task to be read is the second hash bucket;

[0282] Reading a hash bucket write flag, wherein the hash bucket write flag is used to indicate the hash bucket currently corresponding to the coordination thread;

[0283] When the hash bucket write identifier is the first identifier, sending a wake-up message to the coordination thread, wherein the wake-up message is used to wake up the parsing operation of the log file;

[0284] Receive the second notification message sent by the coordination thread in response to the wake-up message.

[0285] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0286] Optionally, in this embodiment, the storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0287] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0288] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0289] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0290] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0291] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0292] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0293] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing log files, It is characterized in that include: Sending a first log task obtained by parsing the first log file to a first hash bucket included in a hash bucket of an application thread, wherein the hash bucket of the application thread includes the first hash bucket and a second hash bucket, and the application thread is used to perform an application operation on the hash bucket currently corresponding to the application thread; In a case where it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notifying the application thread to apply the first log task in the first hash bucket; While the application thread applies the first log task in the first hash bucket, the second log file continues to be parsed and the second log task obtained by parsing the second log file is sent to the second hash bucket.

2. The method according to claim 1, It is characterized in that When it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notifying the application thread to apply the first log task in the first hash bucket includes: Change the hash bucket write identifier from the first identifier to the second identifier, wherein the hash bucket write identifier is the first identifier used to indicate that the hash bucket of the current log task to be written is the first hash bucket, and the hash bucket write identifier is the second identifier used to indicate that the hash bucket of the current log task to be written is the second hash bucket; Reading a hash bucket read identifier, wherein the hash bucket read identifier is used to indicate the hash bucket currently corresponding to the application thread; When the hash bucket read identifier is the first identifier, determining that the hash bucket currently corresponding to the application thread is the first hash bucket; The application thread is notified to apply the first log task in the first hash bucket.

3. The method according to claim 2, It is characterized in that Changing the hash bucket write identifier from the first identifier to the second identifier includes: in the process of changing the hash bucket write identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the application thread is not allowed to perform an operation on the current operation object during the process of performing an operation on the current operation object, and the current operation object includes the hash bucket write identifier; Reading the hash bucket read identifier includes: in the process of reading the hash bucket read identifier, calling the mutex lock, wherein the current operation object includes the hash bucket read identifier.

4. The method according to claim 1, It is characterized in that Before notifying the application thread to apply the first log task in the first hash bucket, the method further includes: When it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket, blocking the parsing operation on the second log file; Wait for the application thread until it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket.

5. The method according to claim 4, It is characterized in that Waiting for the application thread until it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket includes: Waiting for the application thread; receiving a wake-up message sent by the application thread, wherein the wake-up message is used to wake up a parsing operation on the log file; Determine, according to the wake-up message, that the hash bucket currently corresponding to the application thread is the first hash bucket.

6. A method for processing log files, It is characterized in that include: Receive a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket, where the first log task is obtained by the coordination thread parsing a first log file and sent to the first hash bucket; In response to the first notification message, applying the first log task in the first hash bucket, wherein while executing the application operation, the second log file will continue to be parsed; When the application of the first log task is completed, a second notification message sent by the coordination thread is received, wherein the second notification message is used to notify the application of the second log task in the second hash bucket, and the second log task is obtained by the coordination thread parsing the second log file and sent to the second hash bucket; In response to the second notification message, the second log task in the second hash bucket is applied.

7. The method according to claim 6, It is characterized in that When the application of the first log task is completed, receiving the second notification message sent by the coordination thread includes: Change the hash bucket read identifier from the first identifier to the second identifier, wherein the hash bucket read identifier is the first identifier used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is the second identifier used to indicate that the hash bucket of the current log task to be read is the second hash bucket; Reading a hash bucket write flag, wherein the hash bucket write flag is used to indicate the hash bucket currently corresponding to the coordination thread; When the hash bucket write identifier is the first identifier, sending a wake-up message to the coordination thread, wherein the wake-up message is used to wake up the parsing operation of the log file; Receive the second notification message sent by the coordination thread in response to the wake-up message.

8. The method according to claim 7, It is characterized in that After reading the hash bucket write identifier, the method further includes: When the hash bucket write identifier is the second identifier, wait for the coordination thread until the second notification message is received.

9. The method according to claim 7, It is characterized in that Changing the hash bucket read identifier from the first identifier to the second identifier includes: in the process of changing the hash bucket read identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the coordination thread is not allowed to perform an operation on the current operation object during the process of performing an operation on the current operation object, and the current operation object includes the hash bucket read identifier; Reading the hash bucket write identifier includes: in the process of reading the hash bucket write identifier, calling the mutex lock, wherein the current operation object includes the hash bucket write identifier.

10. A log file processing system, It is characterized in that include: A coordination thread, an application thread, a first hash bucket, and a second hash bucket, wherein: The coordination thread is used to: send the first log task obtained by parsing the first log file to the first hash bucket; when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket, notify the application thread to apply the first log task in the first hash bucket; while the application thread applies the first log task in the first hash bucket, continue to parse the second log file and send the second log task obtained by parsing the second log file to the second hash bucket; when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket, notify the application thread to apply the second log task in the second hash bucket; The application thread is used to: apply the first log task in the first hash bucket in response to a notification from the coordination thread; and apply the second log task in the second hash bucket in response to a notification from the coordination thread.

11. The system according to claim 10, It is characterized in that The coordination thread is used to: Change the hash bucket write identifier from the first identifier to the second identifier, wherein the hash bucket write identifier is the first identifier used to indicate that the hash bucket of the current log task to be written is the first hash bucket, and the hash bucket write identifier is the second identifier used to indicate that the hash bucket of the current log task to be written is the second hash bucket; Reading a hash bucket read identifier, wherein the hash bucket read identifier is used to indicate the hash bucket currently corresponding to the application thread; When the hash bucket read identifier is the first identifier, it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket.

12. The system according to claim 11, It is characterized in that The coordination thread is used to: In the process of changing the hash bucket write identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the application thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket write identifier; In the process of reading the hash bucket read identifier, the mutex lock is called, wherein the current operation object includes the hash bucket read identifier.

13. The system according to claim 10, It is characterized in that The application thread is used to: Receive a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket; In response to the first notification message, applying the first log task in the first hash bucket; When the application of the first log task is completed, receiving a second notification message sent by the coordination thread, wherein the second notification message is used to notify the application of the second log task in the second hash bucket; In response to the second notification message, the second log task in the second hash bucket is applied.

14. The system according to claim 13, It is characterized in that The application thread is used to: change the hash bucket read identifier from a first identifier to a second identifier, wherein the hash bucket read identifier is the first identifier and is used to indicate that the hash bucket of the current log task to be read is the first hash bucket, and the hash bucket read identifier is the second identifier and is used to indicate that the hash bucket of the current log task to be read is the second hash bucket; read the hash bucket write identifier, wherein the hash bucket write identifier is used to indicate the hash bucket currently corresponding to the coordination thread; when the hash bucket write identifier is the first identifier, send a wake-up message to the coordination thread, wherein the wake-up message is used to wake up the parsing operation of the log file; receive the second notification message sent by the coordination thread in response to the wake-up message; The coordination thread is used to: block the parsing operation on the second log file when it is determined that the hash bucket currently corresponding to the application thread is the second hash bucket; wait for the application thread; receive the wake-up message sent by the application thread; determine that the hash bucket currently corresponding to the application thread is the first hash bucket based on the wake-up message; and send the second notification message to the application thread in response to the wake-up message.

15. The system according to claim 14, It is characterized in that The application thread is also used to: After reading the hash bucket write identifier, if the hash bucket write identifier is the second identifier, wait for the coordination thread until the second notification message is received.

16. The system according to claim 14, It is characterized in that The application thread is also used to: In the process of changing the hash bucket read identifier from the first identifier to the second identifier, calling a mutex lock, wherein the mutex lock is used to ensure that the coordination thread is not allowed to perform operations on the current operation object during the process of performing operations on the current operation object, and the current operation object includes the hash bucket read identifier; In the process of reading the hash bucket write identifier, the mutex lock is called, wherein the current operation object includes the hash bucket write identifier.

17. A log file processing device, It is characterized in that include: A first sending module, configured to send a first log task obtained by parsing a first log file to a first hash bucket included in a hash bucket of an application thread, wherein the hash bucket of the application thread includes the first hash bucket and a second hash bucket, and the application thread is configured to perform an application operation on the hash bucket currently corresponding to the application thread; a notification module, configured to notify the application thread to apply the first log task in the first hash bucket when it is determined that the hash bucket currently corresponding to the application thread is the first hash bucket; The second sending module is used to continue parsing the second log file and send the second log task obtained by parsing the second log file to the second hash bucket while the application thread applies the first log task in the first hash bucket.

18. A log file processing device, It is characterized in that include: A first receiving module is used to receive a first notification message sent by the coordination thread, wherein the first notification message is used to notify the application of a first log task in a first hash bucket, where the first log task is obtained by the coordination thread parsing the first log file and sent to the first hash bucket; a first application module, configured to apply the first log task in the first hash bucket in response to the first notification message, wherein while executing the application operation, the second log file will continue to be parsed; A second receiving module is used to receive a second notification message sent by the coordination thread when the application of the first log task is completed, wherein the second notification message is used to notify the application of the second log task in the second hash bucket, and the second log task is obtained by the coordination thread parsing the second log file and sent to the second hash bucket; The second application module is used to apply the second log task in the second hash bucket in response to the second notification message.

19. A storage medium, It is characterized in that The storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 9 when executed.

20. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that The processor executes the method described in any one of claims 1 to 9 through the computer program.

Images