A log collection and sending method, device, equipment and storage medium

Abstract

The present application relates to log data processing technical field, disclose a kind of log collection sending method, device, equipment and storage medium, the present application when listening to log file change, obtain file initial state information, query checkpoint database to judge whether there is corresponding site information.If there is not, write in cache area according to preset block size from start offset position;If there is, according to site information to construct / update log playback array, write in cache area according to array information, and the site information of each data block is stored in checkpoint database when writing in cache area.The present application is repeated until all data blocks are written in cache area, obtain the data block to be sent from cache area and send to target position.The present application introduces cache area, checkpoint database and log playback array, and each data block has corresponding site data information, collection and sending do not interfere with each other, independent of each other, solve the anti-pressure blocking problem of single thread sending in prior art, improve the overall efficiency of log data collection and sending.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of log data processing, and in particular to a log collection and sending method, device, equipment and storage medium. BACKGROUND

[0002] In the whole log infrastructure, log collection is one of the most basic components and one of the most front components. Therefore, it is crucial to realize stable, efficient and reliable log collectors to ensure the quality and real-time performance of log collection.

[0003] In order to ensure the consistency of data, the existing log collectors (such as FileBeat, Vector, FluentBit, etc.) generally adopt checkpoint mechanism, which specifically includes: by persisting the data position (collection site) successfully collected and transmitted to the local disk, when the collector restarts or abnormally exits, it can continue to collect from the last recorded position, effectively avoiding data loss and repeated collection. However, this method has a disadvantage, which is to save the collection site in an overwrite way, in order to ensure the order of the site, only single-threaded serial method can be used to send data, when the data sending fails, the collector will repeatedly retry at the failure position until it succeeds, during which the subsequent data processing is completely blocked, the sending block will be transmitted to the collection thread through the back pressure mechanism of the cache queue, causing the whole collection process to stop, which seriously reduces the efficiency of log collection and sending. SUMMARY

[0004] Therefore, the present application provides a log collection and sending method, device, equipment and storage medium to solve the problem of how to improve the efficiency of log collection and sending.

[0005] In a first aspect, the present application provides a log collection and sending method, which comprises:

[0006] Step S1, when it is detected that the log file changes, the initial state information of the changed log file to be read is obtained, wherein the initial state information includes file identification information, file starting offset position and size of a preset data block of the log file to be read, and the log file to be read includes a plurality of data blocks to be read;

[0007] Step S2, according to the file identification information, the preset checkpoint database is queried to determine whether there is site information corresponding to the log file to be read, wherein the site information represents a plurality of data state information of the data blocks to be read in the log file to be read, each data block to be read corresponds to a site information, and the site information at least includes the sending state of the data block;

[0008] Step S3, if the site information does not exist, starting from the file start offset position, each data block in the to-be-read log file is sequentially written into the cache area according to the size of the preset data block, and the site information corresponding to the current data block is stored into the checkpoint database; wherein the sending state of the data block in the site information is to-be-sent.

[0009] Step S4, if the site information exists, the sub-elements in the log playback array are constructed / updated according to the site information, each data block in the to-be-read log file is sequentially written into the cache area according to each sub-element information of the log playback array, and the site information corresponding to the current data block is stored into the checkpoint database; wherein the sending state of the data block in the site information is to-be-sent; the above steps S1 to S4 are repeated.

[0010] Step S5, all to-be-read data blocks in the to-be-read log file are written into the cache area.

[0011] Step S6, the data block with the to-be-sent sending state is obtained from the cache area, the to-be-sent data block is sent to the target position, and the sending state of the data block in the site information of the data block that has been sent is set to sent.

[0012] The embodiment of the present application obtains the initial state information of the to-be-read log file which has changed by listening to the state of the file, stores the site information using a database, and each data block corresponds to an independent site information, avoids the existing mechanism of covering reading and writing, provides a data basis for subsequent parallel processing, queries the database according to the file identification information, when the site information does not exist, starts from the starting position of the file to process the log file by block, and writes each data block into the cache area, if the site information already exists, constructs or updates the log playback array according to the information, and checks each sub-element. Each sub-element can be independently written into the cache area, and is independent with respect to the sending process, whether the data is successfully sent or not, the next log file data block can be continuously written, the global blocking problem in the existing mode is avoided, the data block which has been successfully sent is skipped, the data block which has not been sent or has failed to be sent is directly processed, the process of repeatedly reading and writing is reduced, by storing the site information of each data block into the checkpoint database, when each data block is written into the cache area, the state thereof is recorded in the checkpoint database as to-be-sent, after the data block is successfully sent, the state is updated as sent, and in the last step, the to-be-sent data block is sent, even if a data block fails to be sent, it will not affect the processing of other data blocks, that is, the reading, caching and sending of the data block can be performed in parallel, the sending failure of a single data block will not cause the overall processing to be blocked, thereby improving the overall log collection and sending efficiency.

[0013] In an alternative embodiment, the site information further comprises a data block head offset and a data block size; and step S4 specifically comprises:

[0014] determining whether the log playback array is empty;

[0015] if the log playback array is empty, mapping the corresponding data block head offset, data block size and data block sending state of each data block into corresponding each sub-element information, and storing the sub-elements into the log playback array in ascending order of data block head offset;

[0016] if the log playback array is not empty, determining whether the file start offset position is identical to the data block head offset corresponding to the first sub-element in the log playback array, if identical, sequentially writing each subsequent sub-element corresponding data block into the cache area according to the position of the data block head offset corresponding to the first sub-element in the log playback array; if not identical, emptying the sub-elements in the log playback array, and sequentially writing each data block in the log file to be read into the cache area according to the preset data block size, starting from the file start offset position.

[0017] In the embodiment of the present application, if the log playback array is empty, the head offset, size and sending state of each data block are mapped into corresponding sub-elements, and stored into the log playback array in ascending order of data block head offset, so that the data blocks can be processed in sequence according to the file; if the log playback array is not empty, it is further determined whether the start offset position of the current file is identical to the head offset of the first sub-element in the playback array, when the offsets are identical, it indicates that the file reading process is continuous, and the subsequent data blocks can be directly written into the cache area, thereby avoiding repeated reading; when the offsets are not matched, it is possible that the file has changed or the data blocks are discontinuous, by emptying the array and re-reading, it is ensured that no data is missed or misread, so that the data blocks in the data reading and writing process are continuous, and the log collection efficiency is improved.

[0018] In an alternative embodiment, the initial state information further comprises a content signature of the log file to be read, and the site information further comprises a file path; the method further comprises:

[0019] if the site information exists, determining whether the file path corresponding to the file identification information exists in the site information;

[0020] If there is no file path corresponding thereto, the checkpoint database is queried according to the content signature of the file to be read and the file identification information to determine whether there is a target log file with the same content signature and file identification information as the file to be read, if there is, it is determined that the log file to be read is a rolling file, and the file name of the target log file is obtained, a new file path is generated according to the file name of the target log file, the new file path is stored as the file path of the log file to be read in the checkpoint database, and if there is not, it is determined that the log file to be read is not a rolling file.

[0021] In the embodiment of the application, the initial state information further comprises a content signature of the file to be read, and the site information further comprises a file path, the corresponding file path is found according to the file identification information, whether there is a same target log file is queried through the content signature and the file identification information, whether it is a rolling file is determined, when the rolling file is identified, a new file path can be directly generated and updated to the database, repeated processing of the log file with the same content is avoided, consistency of data is ensured, the number of repeated collection is reduced, and thus the log collection efficiency is improved.

[0022] In an optional embodiment, if it is determined that the log file to be read is not a rolling file, the site information corresponding to the log file to be read is deleted.

[0023] In the embodiment of the application, by deleting the site information corresponding to the log file which is not a rolling file, it is determined that the log file is a new file, and the corresponding site information is deleted, data misreading and writing are avoided, and thus the log collection efficiency is improved.

[0024] In an optional embodiment, the step S6 specifically comprises:

[0025] A plurality of threads are constructed;

[0026] A data block with a first sending state of to-be-sent is obtained from the cache area based on any one of the plurality of threads, and the data block to-be-sent is sent to a target position;

[0027] If the sending is successful, a data block with a next sending state of to-be-sent is obtained, the data block to-be-sent is sent to the target position, and until all the data blocks to-be-sent are sent to the target position, the sending state of the data block in the site information of the data block which has been sent is set to sent;

[0028] If the sending fails, it is confirmed that the current thread is blocked, the remaining any one thread which is not blocked continues to send the failed data block to the target position, until all the data blocks to be sent are sent to the target position, and the sending state of the data block in the position information of the data block which has been sent is set as sent.

[0029] The embodiment of the application constructs a preset number of threads, and the multiple threads are processed in parallel to avoid the back pressure mechanism of a single thread. Any one thread in the multiple threads acquires a first data block with a sending state of to-be-sent from the cache area, and sends the data block to a target position. The multiple threads can acquire data from the cache area and send the data in turn, and the sending efficiency is improved. If the sending succeeds, a next data block with a sending state of to-be-sent is acquired, and the data block is sent to the target position until all the data blocks to be sent are sent to the target position, and the processing continuity is maintained. If the sending fails, it is confirmed that the current thread is blocked, and the remaining any one thread which is not blocked continues to send the failed data block to the target position. The thread can be immediately switched to another thread which is not blocked to continue processing, the thread is automatically transferred after a fault, the thread can be quickly switched to continue processing when a failure occurs, and thus the overall efficiency of log collection and sending is effectively improved.

[0030] In an optional embodiment, the position information further comprises timestamp information; and the method further comprises:

[0031] When the data block is written into the cache area in step S3 or step S4, or when the sub-element is stored into the log replay array in the order of the offset of the head of the data block from small to large, the current time is converted into corresponding timestamp, and the timestamp information in the position information is updated;

[0032] According to the timestamp information, it is determined whether the corresponding position information is expired. When the position information is expired, the position information is cleared.

[0033] According to the timestamp information, it is determined whether the corresponding position information is expired. When the position information is expired, the position information is cleared.

[0034] In the embodiment of the application, the timestamp of the current time is recorded every time the data block is written into the cache area or the log replay array is updated. The timestamp is added to the position information, the latest processing time of each data block is recorded, the accumulation of a large number of useless position records is effectively avoided by clearing the expired position information, the invalid data storage of the log collector in the checkpoint database is reduced, the efficiency of querying the position information which needs to be processed is directly improved, and the log collection efficiency is further improved.

[0034] In a second aspect, the application provides a log collection and sending device, and the device comprises:

[0035] The file monitoring module is configured to acquire initial state information of the to-be-read log file when a change in the log file is monitored, wherein the initial state information comprises file identification information, a file starting offset position, and a size of a preset data block of the to-be-read log file, and the to-be-read log file comprises a plurality of to-be-read data blocks.

[0036] The checkpoint querying module is configured to query a preset checkpoint database according to the file identification information, and determine whether there is site information corresponding to the to-be-read log file, wherein the site information represents a plurality of data state information of the to-be-read data blocks in the to-be-read log file, each to-be-read data block corresponds to one site information, and the site information at least comprises a sending state of the data block.

[0037] The initial data writing module is configured to, if the site information does not exist, sequentially write each data block in the to-be-read log file into a cache area from the file starting offset position according to the size of the preset data block, and store site information corresponding to a current data block into the checkpoint database, wherein the sending state of the data block in the site information is to-be-sent.

[0038] The data updating module is configured to, if the site information exists, construct / update a sub-element in a log playback array according to the site information, sequentially write each data block in the to-be-read log file into the cache area according to each sub-element information of the log playback array, and store site information corresponding to a current data block into the checkpoint database, wherein the sending state of the data block in the site information is to-be-sent.

[0039] The site recording module is configured to write all to-be-read data blocks in the to-be-read log file into the cache area, and repeat the initial data writing module and the data updating module.

[0040] The data sending module is configured to acquire a data block with a to-be-sent sending state from the cache area, send the to-be-sent data block to a target position, and set the sending state of the data block in the site information of the data block that has been sent to a sent state.

[0041] In a third aspect, the present application provides a computer device, comprising a memory and a processor, the memory and the processor are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the log collection and sending method in the first aspect or any of the corresponding embodiments thereof.

[0042] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon computer instructions for causing a computer to execute the log collection and sending method of the first aspect or any of its possible implementation forms.

[0043] In a fifth aspect, the present application provides a computer program product comprising computer instructions for causing a computer to execute the log collection and sending method of the first aspect or any of its possible implementation forms. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0045] Figure 1 is a flowchart of a log collection and sending method according to an embodiment of the present application;

[0046] Figure 2 is a format diagram of an array sub-element corresponding data block according to an embodiment of the present application;

[0047] Figure 3 is a flowchart of another log collection and sending method according to an embodiment of the present application;

[0048] Figure 4 is a flowchart of another log collection and sending method according to an embodiment of the present application;

[0049] Figure 5 is a flowchart of another log collection and sending method according to an embodiment of the present application;

[0050] Figure 6 is a flowchart of another log collection and sending method according to an embodiment of the present application;

[0051] Figure 7 is a flowchart of a related prior art log collection and sending method according to an embodiment of the present application;

[0052] Figure 8 is a data block detection site storage structure diagram of a related prior art log collection and sending method according to an embodiment of the present application;

[0053] Figure 9 is a structural block diagram of a log collection and sending device according to an embodiment of the present application;

[0054] Figure 10 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0056] The prior art solution is mainly based on a checkpoint mechanism, specifically including that a data position (that is, a collection site) collected and transmitted is persisted to a local disk; in order to ensure the order of the collection site, the collector sends data to a target position in a single-threaded manner; after the data is successfully sent, the previous collection site is directly overwritten, and a new collection site is persisted; when the sending fails, the collector is cycled to send and block at this position until the sending succeeds. The single-threaded sending manner limits the concurrency capability of data transmission, and when the sending fails, the collector is cycled to send and block at this position, which causes the whole collection process to stagnate, further blocks the collection thread, and affects the overall collection and sending efficiency.

[0057] The embodiments of the present application provide a log collection and sending method, which is applied to large-scale log collection, distributed system log monitoring and other log analysis scenes. By introducing a cache area, a checkpoint database and a log playback array, each data block has corresponding site data information, the collection and the sending do not interfere with each other and are independent of each other, the problem of back pressure blocking in the single-threaded sending in the prior art is solved, and the overall efficiency of log data collection and sending is improved.

[0058] According to the embodiments of the present application, a log collection and sending method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] In the present embodiment, a log collection and sending method is provided, which can be used in the above computer, Figure 1 is a flowchart of a log collection and sending method according to an embodiment of the present application, as Figure 1 shown, the flowchart includes the following steps:

[0060] Step S1, when a change in the log file is monitored, initial state information of the changed log file to be read is acquired, wherein the initial state information includes file identification information, a file starting offset position and a preset data block size of the log file to be read, and the log file to be read includes a plurality of data blocks to be read;

[0061] It should be noted that monitoring a change in the log file means that the collection module monitors any change of the file in the log directory in real time, such as modification, addition, deletion and the like. The initial state information includes three: file identification information, a file starting offset position and a preset data block size.

[0062] Specifically, technical means such as file monitoring mechanism, file metadata acquisition, content signature calculation and the like can be used. In an example, first, a file listener is established to monitor the log directory, when a file change is detected, the devnode (file identification information) of the file is acquired, and the signature (file signature) of the file is calculated for subsequent file renaming scenario judgment. Finally, the starting reading position of the file and the preset data block size are determined. For example, the a.log file in the xxx / xx / log / directory is monitored, when the file changes: the devnode of the file is 12345, the hash value of the first 1024 bytes is calculated to obtain the signature xxx123, the initial read_offset (file starting offset position) is set to 0, and the preset read_size (preset data block size) is set to 4096 bytes. Thus, necessary initial information is provided for subsequent cyclic processing of data blocks.

[0063] Step S2, according to the file identification information, a preset checkpoint database is queried to determine whether there is bit point information corresponding to the log file to be read, wherein the bit point information represents a plurality of data state information of the data blocks to be read in the log file to be read, each data block to be read corresponds to a bit point information, and the bit point information at least includes a sending state of the data block;

[0064] It should be noted that the bit point information refers to the data block state record stored in the checkpoint database, and in the embodiment of the present application, the bit point information is shown in the following table:

[0065] Field Name Field Type Field Explanation file_path TEXT Real file path devnode INTEGER File node identifier signature INTEGER File content signature read_offset INTEGER Read offset read_size INTEGER Read block size committed INTEGER Commit flag, 1 is committed, 0 is not committed ts INTEGER Last update timestamp

[0066] In the above table, the site information includes file_path (file path), devnode (node identifier), signature (content signature), read_offset (read offset), read_size (read size), committed (commit status), ts (timestamp), and the like. The file node identifier can be obtained from the file metadata information, and the file identifier will not change after the file is renamed. The file content signature is the hash value of the file content. In the implementation, the hash value of the content of the first 1024 bytes or other preset size of the file is calculated. The file read offset and read block size record the position and offset of each file block collected by the collector, which facilitates the data backtracking during the later log replay. The commit flag of the file content block is 0 when the file content block is initially inserted into the checkpoint table, and is updated to 1 after the sending is successful. The ts is updated to the current operation timestamp each time the file block is written or updated.

[0067] Specifically, the sqlite (embedded database) is used for site information query; the file is confirmed through the double verification of the devnode and the signature, so as to obtain the site information based on the log file.

[0068] Step S3, if there is no site information, starting from the starting offset position of the file, each data block in the log file to be read is written into the cache area according to the preset data block size, and the site information corresponding to the current data block is stored in the checkpoint database, wherein the sending state of the data block in the site information is to be sent.

[0069] Specifically, the starting offset position of the file is confirmed, the file content is read according to the preset data block size, the data block is written into the cache area, and the corresponding site record is created.

[0070] For example, if a new log file is processed, starting from offset = 0, 4096 bytes are read each time, and a new site record is created after being written into the cache area, and the committed is set to 0.

[0071] Step S4, if there is site information, the sub-elements in the log replay array are constructed / updated according to the site information, each data block in the log file to be read is written into the cache area according to the information of each sub-element in the log replay array, and the site information corresponding to the current data block is stored in the checkpoint database, wherein the sending state of the data block in the site information is to be sent.

[0072] Specifically, the uncommitted (committed=0) site information is put into the playback array, the playback array is sorted according to the read_offset, and the data blocks are processed according to the order of the information in the playback array. The breakpoint continuous acquisition is realized through the ordered playback array, and the data processing efficiency is improved.

[0073] For example, if there is a playback array REArray, each sub-element in the REArray corresponds to the site information of a data block, and the playback array can be referred to as Figure 2 The array contains three sub-elements S1, S2, and S3 as shown in Figure 2 The stored site information includes: the first end offset 1, the data block size 1, and the sending state of S1; the first end offset 2, the data block size 2, and the sending state of S2; and the first end offset 3, the data block size 3, and the sending state of S3.

[0074] Specifically, in step S3 or step S4, the site information is updated immediately after the data block is written into the cache area, and the playback array state is checked to determine the next processing position. The next data block is collected in a loop until all data blocks are processed, and continuous data block acquisition is realized. It can be understood that if any data block to be read in the log file to be read is not written into the cache area, the above steps S1 to S4 are repeated.

[0075] Step S5, write all data blocks to be read in the log file to be read into the cache area;

[0076] In this embodiment, the committed=0 indicates that the data block has been written into the cache but has not been successfully sent. Through the query and processing of real-time site information, continuous data block acquisition is realized, and the purpose is to ensure parallel processing of subsequent data blocks, thereby improving the acquisition efficiency.

[0077] Step S6, acquire the data block with the sending state of committed=0 from the cache area, send the data block to be sent to the target position, and set the sending state of the data block in the site information of the data block that has been sent to the sent state.

[0078] It should be noted that the target position refers to the destination to which the data block needs to be finally sent, which can be a remote server, a message queue, etc. The committed=1 indicates that the data block has been successfully sent to the target position.

[0079] Specifically, the data block with committed=0 is acquired from the cache area, and the data block is sent to the target position through network transmission. When it is confirmed that the data block is successfully sent, the committed field of the corresponding site information in the checkpoint database is updated to 1.

[0080] An example is that a 4096-byte data block with committed=0 is obtained from the cache area, the data block is sent to the target server through HTTP / TCP or the like, after receiving the confirmation response of the server, the bit point record in the checkpoint database is updated, committed=1, and ts is the current timestamp, parallel processing of the data block is realized, even if a certain data block fails to be sent, it will not affect the processing of other data blocks, global blocking is avoided, and the overall log collection and sending efficiency is improved.

[0081] The embodiment of the present application obtains the initial state information of the changed log file to be read by listening to the state of the file, stores the bit point information in the database, and each data block corresponds to an independent bit point information, avoids the existing cover read-write mechanism, provides data basis for subsequent parallel processing, queries the database according to the file identification information, when the bit point information does not exist, starts from the starting position of the file to process the log file by block, and writes each data block into the cache area, if the bit point information already exists, constructs or updates the log playback array according to the information, and checks each sub-element. Each sub-element can be independently written into the cache area, and is independent with respect to the sending process, whether the data is successfully sent or not, the next log file data block can be continuously written, the global blocking problem in the existing mode is avoided, the data block that has been successfully sent is skipped, the data block that has not been sent or has failed to be sent is directly processed, the process of repeated reading and writing is reduced, by storing the bit point information of each data block into the checkpoint database, when each data block is written into the cache area, its state is recorded in the checkpoint database as to be sent, after the data block is successfully sent, the state is updated as having been sent, and in the last step, the data block to be sent is sent, even if a certain data block fails to be sent, it will not affect the processing of other data blocks, that is, the reading, caching and sending of the data block can be performed in parallel, the sending failure of a single data block will not cause the overall processing to be blocked, thereby improving the overall log collection and sending efficiency.

[0082] In an alternative embodiment, step S4 specifically comprises: judging whether the log replay array is empty; if the log replay array is empty, mapping the corresponding data block head offset of each data block, the size of the data block, and the sending state of the data block into corresponding each sub-element information, and storing the sub-element into the log replay array in the order of the data block head offset from small to large. If the log replay array is not empty, judging whether the file starting offset position is the same as the data block head offset corresponding to the first sub-element in the log replay array. If they are the same, writing each subsequent sub-element corresponding data block into the cache area in the order of the data block head offset corresponding to the first sub-element in the log replay array. If they are not the same, emptying the sub-element in the log replay array, and writing each data block in the log file to be read into the cache area in the order of the preset data block size from the file starting offset position.

[0083] Specifically, the log replay array is an ordered set of data block meta information. The empty set refers to the state that the array does not contain any element, and the judgment of the empty set can be realized by checking whether the array length is 0 or whether there is a valid element.

[0084] For example, refer to Figure 3 When the log replay array is empty, the information of the data block needs to be mapped into a sub-element and stored. In specific implementation, a data structure containing the three attributes can be created as a sub-element, and stored in the array in the order of read_offset from small to large, to ensure that the data blocks are processed in the order of the file and the data continuity. When the log replay array is not empty, it is necessary to judge whether the current file starting offset position matches the offset of the first sub-element in the array. The purpose of this judgment is to ensure the continuity of the data block. In specific implementation, the starting offset position of the current file is compared with the read_offset of the first element in the array. If the offsets are the same, it indicates that the data blocks are continuous, and the data blocks can be directly written into the cache area in the order recorded in the log replay array. At this time, the data block processing is continuous and ordered, and there is no problem of data loss or duplication. If the offsets are different, it indicates that there may be a gap between the data blocks or the file has changed, and the log replay array needs to be emptied and the reading needs to be restarted. Then, the data blocks are read and written into the cache area in the order of the preset data block size from the current file starting offset position, to ensure that the correct data block sequence is re-established in the case of discontinuous data.

[0085] If the log playback array is empty, the initial state information of each data block is mapped to a corresponding sub-element, and the initial state information is stored in the log playback array in ascending order of the initial offset of the data block. The sequential storage of the offset ensures that the data blocks are processed in the order of the file. If the log playback array is not empty, it is further determined whether the initial offset of the current file is the same as the initial offset of the first sub-element in the playback array. When the offsets are the same, the file reading process is continuous, and the subsequent data blocks can be directly written into the cache area, thereby avoiding repeated reading. When the offsets are not matched, the file may have changed or the data blocks are discontinuous. By emptying the array and re-reading, it is ensured that no data is missed or misread, so that the data blocks in the data reading and writing process are continuous, and the log collection efficiency is improved.

[0086] In an optional embodiment, the initial state information further includes a content signature of the file to be read, and the site information further includes a file path. After step S4, the embodiment further includes:

[0087] If the site information exists, it is determined whether the file path corresponding to the file identification information exists in the site information. If the file path corresponding to the file identification information does not exist, the checkpoint database is queried according to the content signature of the file to be read and the file identification information to determine whether there is a target log file with the same content signature and file identification information as the file to be read. If the target log file exists, it is determined that the log file to be read is a rolling file, and the file name of the target log file is obtained. A new file path is generated according to the file name of the target log file, the new file path is stored in the checkpoint database as the file path of the log file to be read, and it is determined that the log file to be read is not a rolling file if the target log file does not exist.

[0088] It should be noted that the content signature of the file to be read is added to the initial state information in the embodiment. The signature is the hash value of the file content. In specific implementation, the hash value of the first 1024 bytes or other preset size of the file content can be calculated. The content signature is used to identify the unique file content. Even if the file name changes, as long as the file content does not change, the content signature remains unchanged. The file path (file_path) field is added to the site information to record the complete path information of the file.

[0089] Specifically, the corresponding file path is found in the site information according to the file identification information (devnode). The file identification information is a unique identifier of the file, which can be obtained from the metadata of the file. Even if the file is renamed, the identifier will not change. The database query statement is used to achieve this.

[0090] For example, refer to Figure 4, the line is checked whether there is site information, and it is further judged whether there is a corresponding file path, when it is found that the file path does not exist, it is necessary to judge whether it is a rolling file scene. Rolling file refers to that when the log file reaches a certain size, it will be renamed and a new log file will be created. The file content signature and file identification information (devnode) are queried in the checkpoint database. If the query result exists, it means that this is a rolling file, because the rolling file has the same file identification information and content signature (that is, the beginning part of the file content does not change), for the case of confirming that it is a rolling file, the file path information needs to be updated, the file name of the target log file is obtained, and then a new complete file path is generated based on the file name. For example, if the original file a.log is renamed as a.log.1, the file path needs to be updated from / xx / xx / a.log to / xx / xx / a.log.1. If no matching record is found in the checkpoint database (that is, the content signature and file identification information do not match), it means that this file is not a rolling file, and it may be a new file or a deleted file, which needs further processing.

[0091] Further, if it is determined that the log file to be read is not a rolling file, the site information corresponding to the log file to be read is deleted.

[0092] The embodiment of the application determines that the log file which is not a rolling file is a new file by deleting the site information corresponding to the log file, and deletes the corresponding site information, thereby avoiding misreading and writing of data, and improving the log collection efficiency.

[0093] If the log playback array is an empty set, the head offset, size and sending state of each data block are mapped to corresponding sub-elements, and are stored in the log playback array in ascending order of the head offset of the data block. Through the sequential storage of the offset, it can be ensured that the data blocks are processed in sequence according to the file; if the log playback array is not an empty set, it is further judged whether the starting offset position of the current file is the same as the head offset of the first sub-element in the playback array. When the offsets are the same, it means that the file reading process is continuous, and the subsequent data blocks can be directly written into the cache area in sequence, thereby avoiding repeated reading. When the offsets do not match, it may be due to the change of the file or the discontinuity of the data block. By emptying the array and re-reading, it is ensured that there is no omission or misreading and writing, thereby ensuring that the data blocks in the data reading and writing process are continuous, and improving the log collection efficiency.

[0094] In an alternative embodiment, step S6 specifically comprises: constructing a preset number of threads; obtaining a data block with a first sending state of to-be-sent from the cache area based on any one of the threads, sending the data block to the target location; if the sending is successful, obtaining a next data block with a sending state of to-be-sent, sending the data block to the target location, until all the data blocks with the sending state of to-be-sent are sent to the target location, and meanwhile, setting the sending state of the data block in the position information of the data block that has been sent to be sent; if the sending fails, confirming that the current thread is blocked, and based on any one of the remaining threads that is not blocked, continuing to send the data block with the sending state of to-be-sent to the target location, until all the data blocks with the sending state of to-be-sent are sent to the target location, and meanwhile, setting the sending state of the data block in the position information of the data block that has been sent to be sent.

[0095] It should be noted that the construction of the preset number of threads in the embodiment refers to the creation of a thread pool, and the number of threads can be flexibly adjusted according to the configuration. The threads are working threads of the sending module and are used to process the sending task of the data block.

[0096] Specifically, reference can be made to Figure 5 After the preset number of threads are constructed, the process of obtaining the data block to be sent from the cache area adopts a preemptive polling mechanism. When a thread successfully obtains a data block, the data block is immediately sent to the target location. After the sending is successful, the committed state of the corresponding data block in the checkpoint database is updated to 1, and the ts field is updated to the current timestamp. After the sending fails, the failed data block is returned to the cache area, and the current thread is marked as a blocked state. Other threads that are not blocked can continue to obtain the data block (the data block that previously failed to be sent) from the cache area for sending. Through the multi-thread parallel processing and the automatic transfer mechanism for failure, the overall processing flow will not be affected due to the sending failure of a single data block, and thus the efficiency of data sending is improved.

[0097] The embodiment of the present application constructs a preset number of threads, and the multiple threads are processed in parallel to avoid the back pressure mechanism of single thread. A first data block with a sending state of to-be-sent is obtained from the cache area based on any one of the multiple threads, and the data block with the to-be-sent sending state is sent to a target position. The multiple threads can obtain data from the cache area and send the data in sequence, so that the sending efficiency is improved. If the sending is successful, a next data block with a to-be-sent sending state is obtained, and the data block with the to-be-sent sending state is sent to the target position until all the data blocks with the to-be-sent sending state are sent to the target position, so that the processing continuity is maintained. If the sending fails, it is confirmed that the current thread is blocked, and the data block with the sending failure is continuously sent to the target position based on any one of the remaining threads which are not blocked. The thread can be immediately switched to another thread which is not blocked to continue processing, so that the automatic transfer of the thread after failure is realized. When the failure occurs, the thread can be quickly switched to continue processing, so that the overall efficiency of log collection and sending is effectively improved.

[0098] In an optional embodiment, the method further comprises: when the data block is written into the cache area in step S3 or step S4, or when the sub-element is stored into the log playback array in the order of the offset of the head of the data block from small to large, the current time is converted into a corresponding timestamp, and the timestamp information in the site information is updated; whether the corresponding site information is expired is determined according to the timestamp information, and when the site information has expired, the site information is cleared.

[0099] Specifically, when the data block is written into the memory buffer in step S3 or step S4, a pre-write checkpoint operation is required to insert or update the corresponding site information in the checkpoint table. The current time is converted into a timestamp format, and the ts field of the site information is updated to a new timestamp. Alternatively, when the sub-element is stored into the log playback array in the order of the offset of the head of the data block from small to large, the ts field of the site information of the sub-element is updated to the current timestamp. The expired site information is regularly checked and cleaned up. The current system time is obtained and converted into a corresponding timestamp format. All site information in the checkpoint table is traversed. The ts field of each record is read. The current timestamp is compared with the ts of each record. The time difference is calculated. If the time difference exceeds a preset expiration threshold (for example, 7 days), the site information is considered to have expired. The expired site information is deleted from the checkpoint table. By automatically cleaning up the expired data, the accumulation of too many invalid records in the checkpoint database is avoided.

[0100] The embodiment of the present application records the time stamp of the current time whenever the data block is written into the cache area or the log playback array is updated. The time stamp is added to the site information, so that the latest processing time of each data block can be recorded, and a large number of useless site records can be effectively avoided by clearing the expired site information, the invalid data storage of the log collector in the checkpoint database is reduced, the efficiency of querying the site information currently needing to be processed is directly improved, and the log collection efficiency is further improved.

[0101] In an actual application scenario, the overall implementation process of the log collection and sending method provided by the embodiment of the present application can refer to Figure 6 , which is divided into a collection module and a sending module group, the collection module loads and verifies a checkpoint, arranges a playback segment, reads a file block along a playback site if there is a playback segment, reads along the latest site otherwise, writes a buffer area, prewrites a checkpoint, and the state is uncommitted, the sending module group reads a file block from the buffer area, sends data, continues to read and send data if sending fails, updates the checkpoint state to committed if sending succeeds, and clears the checkpoint file.

[0102] In the prior art, the collection and sending process of log data can refer to Figure 7 and Figure 8 If a file has three file blocks S1, S2 and S3 to be collected, the collector saves the end position of the S1 file block to the checkpoint after the S1 is successfully sent, and the collector continues to collect the file block S2 along the end position of the S1 if the collector restarts or an exception occurs. Only one collection site of each file is saved in the checkpoint, and the latest site covers the old site, and the data before the collection site in the checkpoint file has been successfully collected and sent to the target position. That is, the next file is read only after the file block is read and the file is successfully sent. If the sending fails once, the collector will be blocked at this position and will not be sent until the sending succeeds. In the process of sending blocking, the collection thread is also blocked due to the back pressure mechanism of the cache queue.

[0103] Therefore, compared with the log collection method in the prior art, the embodiment of the present application has the following beneficial technical effects:

[0104] (1) By introducing the start address of the file block, the file block size, the submission state and the like, the states of the collector sites are independent of each other. The design of the data format is the most core part of the checkpoint mechanism, decouples the sending module and the collection module, so that the sending module can be parallelly expanded, the overall collection performance is improved, and the state of the collection site in the prior art is avoided, so that the overall efficiency of the collector is greatly improved.

[0105] (2) The checkpoint data format is used for log playback and checking rollback log files, ensuring breakpoint continuous collection and rolling scene collection after log interruption, and ensuring log integrity.

[0106] (3) The independence of the collection module and the sending module. Unlike the prior art, the collection module also waits for the sending module to update the collection site. In the embodiment of the application, the sending module group can be linearly expanded according to a preset value, improving the sending efficiency.

[0107] In the embodiment, a log collection and sending device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and has been described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0108] The embodiment provides a log collection and sending device, as shown in Figure 9 , comprising:

[0109] A file monitoring module is configured to acquire initial state information of a to-be-read log file that has changed when it is monitored that the log file has changed, wherein the initial state information includes file identification information, a file starting offset position and a size of a preset data block of the to-be-read log file, and the to-be-read log file includes a plurality of to-be-read data blocks.

[0110] A checkpoint query module is configured to query a preset checkpoint database according to the file identification information, and determine whether there is site information corresponding to the to-be-read log file, wherein the site information represents a plurality of data state information of the to-be-read data blocks in the to-be-read log file, each to-be-read data block corresponds to one site information, and the site information at least includes a sending state of the data block.

[0111] An initial data writing module is configured to, if the site information does not exist, write each data block in the to-be-read log file into a cache area in sequence from the file starting offset position according to the size of the preset data block, and store site information corresponding to the current data block to the checkpoint database, wherein the sending state of the data block in the site information is to-be-sent.

[0112] a data updating module, configured to, if the position information exists, construct / update a sub-element in a log replay array according to the position information, write each data block in the to-be-read log file into the cache area according to each sub-element information in the log replay array in sequence, and store position information corresponding to a current data block into the checkpoint database; wherein a sending state of the data block in the position information is to-be-sent;

[0113] a position recording module, configured to write all to-be-read data blocks in the to-be-read log file into the cache area; and repeat the initial data writing module and the data updating module;

[0114] a data sending module, configured to acquire a data block with a to-be-sent sending state from the cache area, send the to-be-sent data block to a target position, and set the sending state of the data block in the position information to sent.

[0115] In some optional embodiments, the position information further includes a data block head offset and a size of the data block; and the data updating module specifically includes:

[0116] a log replay array empty set judgment unit, configured to judge whether the log replay array is an empty set;

[0117] a data block mapping storage unit, configured to, if the log replay array is an empty set, map the corresponding data block head offset, the size of the data block, and the sending state of the data block into corresponding each sub-element information, and store the sub-element into the log replay array in an ascending order of the data block head offset;

[0118] an offset matching processing unit, configured to, if the log replay array is not an empty set, judge whether a file starting offset position is the same as a data block head offset corresponding to a first sub-element in the log replay array, if yes, write each subsequent sub-element corresponding data block into the cache area in sequence according to a position of the data block head offset corresponding to the first sub-element in the log replay array, and if not, empty the sub-element in the log replay array, and write each data block in the to-be-read log file into the cache area in sequence according to a preset data block size, starting from the file starting offset position.

[0119] In some optional embodiments, the initial state information further includes a content signature of the to-be-read file, and the position information further includes a file path; and the apparatus further includes:

[0120] a path detection module, configured to, if the position information exists, find whether a file path corresponding to the file identification information exists in the position information according to the file identification information;

[0121] The rolling file detection module is configured to: if there is no file path corresponding thereto, query the checkpoint database according to the content signature and the file identification information of the file to be read, determine whether there is a target log file with the same content signature and file identification information as the file to be read, if there is, determine that the log file to be read is a rolling file, acquire the file name of the target log file, generate a new file path according to the file name of the target log file, store the new file path as the file path of the log file to be read in the checkpoint database, and if there is not, determine that the log file to be read is not a rolling file.

[0122] In some optional embodiments, if it is determined that the log file to be read is not a rolling file, the site information corresponding to the log file to be read is deleted.

[0123] In some optional embodiments, the data sending module specifically comprises:

[0124] The thread pool construction unit is configured to construct a plurality of threads in a preset number.

[0125] The data block sending unit is configured to acquire, based on any one thread of the plurality of threads, a data block with a first sending state to be sent from the cache area, and send the data block to be sent to the target position.

[0126] The sending success processing unit is configured to, if the sending is successful, acquire a data block with a next sending state to be sent, send the data block to be sent to the target position, and continue to send the data block to be sent to the target position until all the data blocks to be sent are sent to the target position, and set the sending state of the data block in the site information of the data block that has been sent to be sent.

[0127] The sending failure processing unit is configured to, if the sending fails, confirm that the current thread is blocked, continue to send the data block that fails to send to the target position based on any one thread of the remaining threads that is not blocked, until all the data blocks to be sent are sent to the target position, and set the sending state of the data block in the site information of the data block that has been sent to be sent.

[0128] In some optional embodiments, the site information further comprises timestamp information, and the method further comprises:

[0129] The timestamp updating module is configured to, when the data block is written into the cache area in the initial data writing module or the data updating module, or when the sub-elements are stored into the log playback array in the order of the data block head end offset from small to large, convert the current time into a corresponding timestamp, and update the timestamp information in the site information.

[0130] The expired site cleaning module is configured to determine whether the corresponding site information is expired according to the timestamp information, and clean up the site information when the site information has been expired.

[0131] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0132] In this embodiment, the log collection and transmission device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0133] This invention also provides a computer device having the above-described features. Figure 9 The log collection and transmission device shown.

[0134] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 10 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 10 Take a processor 10 as an example.

[0135] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0136] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0137] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required for at least one function, etc. The data storage area can store data created by the computer device, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state memory device. In some alternative embodiments, the memory 20 can optionally include memory that is remotely located with respect to the processor 10, and which can be connected to the computer device through a network. Examples of such networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communications network, and combinations thereof.

[0138] The memory 20 can include a volatile memory, such as a random access memory, and / or can include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid state memory device. The memory 20 can also include an array of multi-state flash memory cells, which can be used to store data and / or instructions in multiple states.

[0139] The computer device also includes a communications interface 30 for communicating with other devices or communication networks.

[0140] The embodiments of the present application also provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or implemented as computer code to be originally stored in a remote storage medium or a non-transitory machine readable storage medium downloaded through a network and stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned kinds of storage. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0141] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file and the like, and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0142] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the defined scope.

Claims

1. A log collection and transmission method, characterized by, The method comprises: Step S1, when a change in a log file is detected, initial state information of the changed log file to be read is acquired, wherein the initial state information comprises file identification information, a file starting offset position and a size of a preset data block of the log file to be read, and the log file to be read comprises a plurality of data blocks to be read; Step S2, the file identification information is used to query a preset checkpoint database to determine whether there is site information corresponding to the log file to be read, wherein the site information represents a plurality of data state information of the data blocks to be read in the log file to be read, each data block to be read corresponds to one site information, and the site information at least comprises a sending state of the data block; Step S3, if the site information does not exist, each data block in the log file to be read is sequentially written into a cache area from the file starting offset position according to the size of the preset data block, and site information corresponding to the current data block is stored into the checkpoint database; wherein the sending state of the data block in the site information is to be sent; Step S4, if the site information exists, sub-elements in a log playback array are constructed / updated according to the site information, each data block in the log file to be read is sequentially written into the cache area according to each sub-element information of the log playback array, and site information corresponding to the current data block is stored into the checkpoint database; wherein the sending state of the data block in the site information is to be sent; the above steps S1 to S4 are repeated; Step S5, all the data blocks to be read in the log file to be read are written into the cache area; Step S6, a data block with a sending state to be sent is acquired from the cache area, the data block to be sent is sent to a target position, and the sending state of the data block in the site information of the data block that has been sent is set to be sent.

2. The method of claim 1, wherein, The site information further comprises a data block head offset and a size of the data block; and step S4 specifically comprises: determining whether the log playback array is an empty set; if the log playback array is an empty set, corresponding data block head offsets, sizes of data blocks and sending states of data blocks of each data block are mapped into corresponding each sub-element information, and the sub-elements are stored into the log playback array in ascending order of the data block head offsets; if the log playback array is not an empty set, it is determined whether the file starting offset position is the same as a data block head offset corresponding to a first sub-element in the log playback array, if yes, data blocks corresponding to subsequent each sub-element are sequentially written into the cache area according to a position of the data block head offset corresponding to the first sub-element in the log playback array, and if not, the sub-elements in the log playback array are emptied, each data block in the log file to be read is sequentially written into the cache area from the file starting offset position according to the size of the preset data block.

3. The method of claim 2, wherein, The initial state information further comprises a content signature of the to-be-read file, and the site information further comprises a file path; the method further comprises: If the site information exists, whether a file path corresponding to the file identification information exists in the site information is searched according to the file identification information; If the file path corresponding to the file identification information does not exist, whether a target log file with the same content signature and file identification information as the to-be-read file exists is determined according to the content signature of the to-be-read file and the file identification information; if the target log file exists, it is determined that the to-be-read log file is a rolling file, a file name of the target log file is obtained, a new file path is generated according to the file name of the target log file, the new file path is stored as the file path of the to-be-read log file in the checkpoint database; if the target log file does not exist, it is determined that the to-be-read log file is not a rolling file.

4. The method of claim 3, wherein, If it is determined that the to-be-read log file is not a rolling file, the site information corresponding to the to-be-read log file is deleted.

5. The method of claim 4, wherein, Step S6 specifically comprises: A plurality of threads are constructed; A first data block with a to-be-sent sending state is obtained from the cache area based on any one of the plurality of threads, and the to-be-sent data block is sent to a target position; If the sending is successful, a next data block with a to-be-sent sending state is obtained, and the to-be-sent data block is sent to the target position until all the to-be-sent data blocks are sent to the target position, and the sending state of the data block in the site information of the data block that has been sent is set to sent; If the sending fails, it is confirmed that the current thread is blocked, and the data block that fails to be sent is continuously sent to the target position based on any one of the remaining threads that are not blocked until all the to-be-sent data blocks are sent to the target position, and the sending state of the data block in the site information of the data block that has been sent is set to sent.

6. The method according to any one of claims 1 to 5, characterized in that, The site information further comprises timestamp information; the method further comprises: When the data block is written into the cache area in step S3 or step S4, or when the sub-element is stored in the log playback array in the order of the data block head end offset from small to large, the current time is converted into a corresponding timestamp, and the timestamp information in the site information is updated; Whether the corresponding site information is expired is determined according to the timestamp information, and the site information is cleared when the site information has expired.

7. A log collection and sending apparatus characterized by comprising: The device comprises: A file monitoring module configured to, when a change in a log file is monitored, obtain initial state information of the to-be-read log file that has changed, wherein the initial state information comprises file identification information, a file start offset position, and a size of a preset data block of the to-be-read log file, and the to-be-read log file comprises a plurality of to-be-read data blocks; The checkpoint query module is configured to query a preset checkpoint database according to the file identification information, and determine whether there is site information corresponding to the log file to be read. The initial data writing module is configured to, if the site information does not exist, write each data block in the log file to be read into a cache area in sequence from the file start offset position according to the size of the preset data block, and store site information corresponding to a current data block into the checkpoint database, wherein the sending state of the data block in the site information is to be sent. The data updating module is configured to, if the site information exists, construct / update a sub-element in a log playback array according to the site information, write each data block in the log file to be read into the cache area in sequence according to each sub-element information of the log playback array, and store site information corresponding to a current data block into the checkpoint database, wherein the sending state of the data block in the site information is to be sent. The site recording module is configured to write all the data blocks to be read in the log file to be read into the cache area. The data sending module is configured to acquire a data block with a sending state of to be sent from the cache area, send the data block to a target position, and set the sending state of the data block in the site information of the data block that has been sent to sent.

8. A computer device, comprising: The memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the log collection and sending method in any one of claims 1-6. The computer readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the log collection and sending method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer instructions are used to make a computer execute the log collection and sending method in any one of claims 1-6.

10. A computer program product, characterised in that, ​

Images