Log collecting and sending method and device, equipment and storage medium

Abstract

The invention relates to the technical field of log data processing, and discloses a log collecting and sending method, device and equipment and a storage medium. When log file changes are monitored, file initial state information is obtained, and a check point database is inquired to judge whether corresponding site information exists or not; if not, writing into the cache region from the initial offset position according to the preset block size; if the data block exists, a log playback array is constructed / updated according to the site information, the data block is written into the cache region according to the array information, and the site information of the data block is stored into the check point database every time the data block is written into the cache region. And repeating until all the data blocks are written into the cache region, acquiring the to-be-sent data blocks from the cache region, and sending the to-be-sent data blocks to the target position. According to the method, the cache region, the check point database and the log playback array are introduced, each data block has corresponding site data information, collection and sending do not interfere with each other and are independent from each other, the problem of back pressure blockage of single-thread sending in the prior art is solved, and the overall efficiency of log data collection and sending is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of log data processing, and in particular to a log collection and sending method, device, equipment and storage medium. Background Art

[0002] In the entire log infrastructure, log collection is one of the most basic components and one of the most advanced components. Therefore, implementing a stable, efficient, and reliable log collector is crucial to ensuring the quality and real-time performance of log collection.

[0003] To ensure data consistency, existing log collectors (such as FileBeat, Vector, FluentBit, etc.) generally adopt a checkpoint mechanism, which specifically includes: by persisting the successfully collected and transmitted data location (collection site) to the local disk, when the collector is restarted or abnormally exits, it can continue to collect from the last recorded location, effectively avoiding data loss and repeated collection. However, this method has a disadvantage. The collection site is saved by overwriting. To ensure the orderliness of the site, data can only be sent in a single-threaded serial manner. When data transmission fails, the collector will repeatedly retry at the failed location until it succeeds, completely blocking the processing of subsequent data during this period. The transmission blockage will be transmitted to the collection thread through the back pressure mechanism of the cache queue, causing the entire collection process to stagnate, seriously reducing the efficiency of log collection and transmission. Summary of the invention

[0004] In view of this, the present invention provides a log collection and sending method, device, equipment and storage medium to solve the problem of how to improve the efficiency of log collection and sending.

[0005] In a first aspect, the present invention provides a log collection and sending method, the method comprising:

[0006] Step S1, when a change in a log file is detected, initial state information of the changed log file to be read is obtained, wherein the initial state information includes file identification information of the log file to be read, a starting offset position of the file, and a size of a preset data block, and the log file to be read includes multiple data blocks to be read;

[0007] Step S2, querying a preset checkpoint database according to the file identification information to determine whether there is location information corresponding to the log file to be read, wherein the location information represents a variety of data status information of a data block to be read in the log file to be read, each data block to be read corresponds to a location information, and the location information at least includes a sending status of the data block;

[0008] Step S3, if the location information does not exist, starting from the starting offset position of the file, each data block in the to-be-read log file is sequentially written into the cache area according to the size of the preset data block, and the location information corresponding to the current data block is stored in the checkpoint database; wherein the sending status of the data block in the location information is to be sent;

[0009] Step S4, if the location information exists, construct / update the sub-element in the log playback array according to the location information, write each data block in the log file to be read into the buffer area in turn according to each sub-element information of the log playback array, and store the location information corresponding to the current data block into the checkpoint database; wherein the sending status of the data block in the location information is to be sent; repeat the above steps S1 to S4;

[0010] Step S5, writing all the data blocks to be read in the log file to be read into the buffer area;

[0011] Step S6, obtaining a data block with a sending status of "to be sent" from the buffer area, sending the data block to be sent to the target location, and at the same time, setting the sending status of the data block in the location information of the data block that has been sent to "sent".

[0012] The embodiment of the present invention obtains the initial status information of the changed log file to be read by monitoring the status of the file, uses the database to store the location information, and each data block corresponds to an independent location information, avoiding the existing overwriting reading and writing mechanism, providing a data basis for subsequent parallel processing, and querying the database according to the file identification information. When the location information does not exist, the log file is processed block by block starting from the starting position of the file, and each data block is written into the cache area. If the location information already exists, the log playback array is constructed or updated according to this information, and each sub-element is checked. Each sub-element can be written to the cache independently and is independent of the sending process. Regardless of whether the data is sent successfully, the data block of the next log file can continue to be written, avoiding the global blocking problem under the existing method, skipping the data blocks that have been sent successfully, and directly processing the data blocks that have not been sent or failed to send, reducing the repeated re-reading and re-writing process. By storing the location information of each data block in the checkpoint database, when each data block is written to the cache, its status will be recorded in the checkpoint database as waiting to be sent. After the data block is sent successfully, the status is updated to sent. In the last step, the data block to be sent is sent. Even if a data block fails to send, it will not affect the processing of other data blocks. In other words, the reading, caching and sending of data blocks can be carried out in parallel, and the failure to send a single data block will not cause overall processing blocking, thereby improving the overall log collection and sending efficiency.

[0013] In an optional implementation, the location information further includes a data block head end offset and a data block size; step S4 specifically includes:

[0014] Determine whether the log playback array is an empty set;

[0015] If the log playback array is an empty set, the corresponding data block head end offset, data block size, and data block sending status of each data block are mapped to each corresponding sub-element information, and the sub-elements are stored in the log playback array in the order of the data block head end offset from small to large;

[0016] If the log playback array is not an empty set, determine whether the starting offset position of the file is the same as the data block head offset corresponding to the first sub-element in the log playback array; if they are the same, write the data blocks corresponding to each subsequent sub-element into the cache area in sequence according to the position of the data block head offset corresponding to the first sub-element in the log playback array; if they are not the same, clear the sub-elements in the log playback array, and start from the starting offset position of the file, write each data block in the log file to be read into the cache area in sequence according to the size of the preset data block.

[0017] In the embodiment of the present invention, if the log playback array is an empty set, the head end offset, size, and sending status of each data block are mapped to corresponding sub-elements, and are stored in the log playback array in the order of the data block head end offset from small to large. By storing the offset in sequence, it can be ensured that the data blocks are processed in sequence according to the order of the files; if the log playback array is not an empty set, it is further determined whether the starting offset position of the current file is the same as the head end offset of the first sub-element in the playback array. When the offsets are the same, it means that the file reading process is continuous, and the subsequent data blocks can be directly written into the cache area in sequence to avoid repeated reading. When the offsets do not match, it may be due to the change of the file and the discontinuity of the data blocks. By clearing the array and re-reading, it is ensured that no omissions or misreading and writing are made, thereby ensuring that the data blocks in the data reading and writing process are continuous, thereby improving the collection efficiency of the log.

[0018] In an optional implementation, the initial state information further includes a content signature of the file to be read, and the location information further includes a file path; the method further includes:

[0019] If the location information exists, searching, according to the file identification information, whether there is a file path corresponding to the location information;

[0020] If there is no corresponding file path, the checkpoint database is queried according to the content signature of the file to be read and the file identification information to determine whether there is a target log file with the same content signature and file identification information as the file to be read. If so, it is determined that the log file to be read is a rolling file, and the file name of the target log file is obtained. A new file path is generated according to the file name of the target log file, and the new file path is used as the file path of the log file to be read and stored in the checkpoint database. If it does not exist, it is determined that the log file to be read is not a rolling file.

[0021] In an embodiment of the present invention, the initial status information also includes a content signature of the file to be read, and the location information also includes a file path. The corresponding file path is searched according to the file identification information, and whether the same target log file exists is queried through the content signature and the file identification information to determine whether it is a rolling file. When a rolling file is identified, a new file path can be directly generated and updated to the database, avoiding repeated processing of log files with the same content, ensuring data consistency, reducing the number of repeated collections, and thus improving log collection efficiency.

[0022] In an optional implementation, if it is determined that the log file to be read is not a rolling file, the location information corresponding to the log file to be read is deleted.

[0023] The embodiment of the present invention deletes the location information corresponding to the log file that is not a rolling file, determines that the log file is a new file, and deletes the corresponding location information, thereby avoiding erroneous reading and writing of data, thereby improving log collection efficiency.

[0024] In an optional implementation, step S6 specifically includes:

[0025] Build a preset number of multiple threads;

[0026] Based on any one of the multiple threads obtaining a first data block in a sending state to be sent from the buffer area, sending the data block to be sent to a target location;

[0027] If the sending is successful, the next data block with a sending status of to-be-sent is obtained, and the data block to-be-sent is sent to the target location until all the data blocks to be sent are sent to the target location. At the same time, the sending status of the data block in the location information of the data block that has been sent is set to sent;

[0028] If the sending fails, it is confirmed that the current thread is blocked, and any other unblocked thread will continue to send the data block that failed to be sent to the target location until all the data blocks to be sent are sent to the target location. At the same time, the sending status of the data block in the location information of the data block that has been sent is set to sent.

[0029] The embodiment of the present invention constructs a preset number of multiple threads, and multi-threaded parallel processing avoids the back pressure mechanism of a single thread. Based on any one of the multiple threads, the first data block with a sending status of to-be-sent is obtained from the cache area, and the data block to-be-sent is sent to the target location. The multiple threads can sequentially obtain data from the cache area and send it, thereby improving the sending efficiency. If the sending is successful, the next data block with a sending status of to-be-sent is obtained, and the data block to-be-sent is sent to the target location until all the data blocks to be sent are sent to the target location, thereby maintaining processing continuity. If the sending fails, it is confirmed that the current thread is blocked, and based on any other unblocked thread, the data block that failed to be sent continues to be sent to the target location, and the processing can be switched to other unblocked threads immediately for continued processing, thereby realizing automatic thread transfer after a failure. In case of failure, the thread can be quickly switched to continue processing, thereby effectively improving the overall efficiency of log collection and sending.

[0030] In an optional implementation manner, the location information further includes timestamp information; and the method further includes:

[0031] When the data block is written into the cache in step S3 or step S4, or when the sub-elements are stored in the log playback array in the order of the data block head end offset from small to large, the current moment is converted into a corresponding timestamp, and the timestamp information in the location information is updated;

[0032] Determine whether the corresponding location information is expired according to the timestamp information, and clear the location information when the location information is expired.

[0033] In the embodiment of the present invention, whenever a data block is written into the cache area or the log playback array is updated, the timestamp of the current moment is recorded. By adding the timestamp to the location information, the latest processing time of each data block can be recorded. By clearing the expired location information, a large number of useless location records can be effectively avoided from being accumulated, and the invalid data storage of the log collector in the checkpoint database can be reduced, which directly improves the efficiency of querying the location information that needs to be processed currently, and further improves the efficiency of log collection.

[0034] In a second aspect, the present invention provides a log collection and sending device, the device comprising:

[0035] A file monitoring module, configured to obtain initial status information of the changed log file to be read when a change in the log file is detected, wherein the initial status information includes file identification information of the log file to be read, a starting offset position of the file, and a size of a preset data block, and the log file to be read includes multiple data blocks to be read;

[0036] A checkpoint query module is used to query a preset checkpoint database according to the file identification information to determine whether there is location information corresponding to the log file to be read, wherein the location information represents a variety of data status information of the data block to be read in the log file to be read, each data block to be read corresponds to a location information, and the location information at least includes a sending status of the data block;

[0037] An initial data writing module is used to write each data block in the log file to be read into the cache area in sequence according to the size of the preset data block, starting from the starting offset position of the file if the location information does not exist, and store the location information corresponding to the current data block into the checkpoint database; wherein the sending status of the data block in the location information is to be sent;

[0038] A data update module, for constructing / updating sub-elements in a log playback array according to the location information if the location information exists, writing each data block in the log file to be read into the buffer area in turn according to each sub-element information of the log playback array, and storing the location information corresponding to the current data block into the checkpoint database; wherein the sending status of the data block in the location information is to be sent;

[0039] A location recording module is used to write all the data blocks to be read in the log file to be read into the buffer area; repeat the above initial data writing module and data updating module;

[0040] The data sending module is used to obtain the data block with the sending status of waiting to be sent from the buffer area, send the data block to be sent to the target location, and at the same time, set the sending status of the data block in the location information of the data block that has been sent to sent.

[0041] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the log collection and sending method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the log collection and sending method of the first aspect or any corresponding embodiment thereof.

[0043] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the log collection and sending method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of a log collection and sending method according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the data block format corresponding to the array sub-elements of the log collection and sending method according to an embodiment of the present invention;

[0047] Figure 3 is a flow chart of another log collection and sending method according to an embodiment of the present invention;

[0048] Figure 4 is a flow chart of another log collection and sending method according to an embodiment of the present invention;

[0049] Figure 5 is a flow chart of another log collection and sending method according to an embodiment of the present invention;

[0050] Figure 6 is a flow chart of another log collection and sending method according to an embodiment of the present invention;

[0051] Figure 7 It is a flowchart of a method for collecting and sending logs in the related prior art according to an embodiment of the present invention;

[0052] Figure 8 It is a schematic diagram of a data block detection site storage structure in a log collection and sending method in the related prior art according to an embodiment of the present invention;

[0053] Fig. 9 is a structural block diagram of a log collection and sending device according to an embodiment of the present invention;

[0054] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] The existing technology is mainly based on a checkpoint mechanism, which specifically includes that the location of the collected and transmitted data (that is, the collection site) is persisted to the local disk; in order to ensure the orderliness of the collection site, the collector uses a single-threaded method to send data to the target location; after the data is successfully sent, it directly overwrites the previous collection site and persists the new collection site; when the sending fails, the collector sends and blocks here in a loop until the sending succeeds. The single-threaded sending method limits the concurrent capability of data transmission. When the sending fails, the collector will send and block here in a loop, causing the entire collection process to stagnate, further blocking the collection thread and affecting the overall collection and sending efficiency.

[0057] The embodiment of the present invention provides a log collection and sending method, which is applied to log analysis scenarios such as large-scale log collection and distributed system log monitoring. By introducing a cache area, a checkpoint database and a log playback array, each data block has corresponding site data information, and collection and sending do not interfere with each other and are independent of each other, which solves the back pressure blocking problem of single-threaded sending in the prior art and improves the overall efficiency of log data collection and sending.

[0058] According to an embodiment of the present invention, an embodiment of a log collection and sending method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] In this embodiment, a log collection and sending method is provided, which can be used for the above-mentioned computer. Figure 1 is a flow chart of a log collection and sending method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0060] Step S1, when a change in a log file is detected, initial state information of the changed log file to be read is obtained, wherein the initial state information includes file identification information, a starting offset position of the file, and a size of a preset data block of the log file to be read, and the log file to be read includes multiple data blocks to be read;

[0061] It should be noted that monitoring log file changes means that the acquisition module monitors any modification, addition, deletion, etc. of files in the log directory in real time. The initial state information includes three: file identification information, file starting offset position, and preset data block size.

[0062] Specifically, technical means such as file monitoring mechanism, file metadata acquisition, and content signature calculation can be used. In one example, a file listener is first established to monitor the log directory. When a file change is detected, the devnode (file identification information) of the file is obtained, and the signature (file signature) of the file is calculated for subsequent file renaming scenario judgment. Finally, the starting reading position of the file and the preset data block size are determined. For example, monitor the a.log file in the xxx / xx / log / directory. When the file changes: obtain the file devnode as 12345, calculate the hash value of the first 1024 bytes to obtain the signature as xxx123, set the initial read_offset (file starting offset position) to 0, and preset read_size (preset data block size) to 4096 bytes. This provides the necessary initial information for the subsequent loop processing of collecting data blocks.

[0063] Step S2, querying a preset checkpoint database according to the file identification information to determine whether there is location information corresponding to the log file to be read, wherein the location information represents a variety of data status information of a data block to be read in the log file to be read, each data block to be read corresponds to a location information, and the location information at least includes a sending status of the data block;

[0064] It should be noted that the location information refers to the data block status record stored in the checkpoint database. The location information in the embodiment of the present invention is shown in the following table:

[0065] Field Name Field Type Field Explanation file_path TEXT The actual file path devnode INTEGER File node identifier signature INTEGER File content signature read_offset INTEGER Read offset read_size INTEGER Read block size committed INTEGER Commit flag, 1 means submitted, 0 means not submitted ts INTEGER Last updated timestamp

[0066] In the above table, the site information includes fields such as file_path (file path), devnode (node ​​identifier), signature (content signature), read_offset (read offset), read_size (read size), committed (committed status), and ts (timestamp). The file node identifier can be obtained from the file metadata information. After the file is renamed, the file identifier will not change. The file content signature is the hash value of the file content. In the specific implementation, the hash value of the first 1024 bytes of the file or other preset value size is calculated. The file read offset and read block size record the position and offset of the file block collected by the collector each time, so that the data can be traced back when the log is replayed later. When the file content block is first inserted into the checkpoint table, the commit flag is 0 until it is sent successfully, then it will be updated to 1. Each time a file block is written or updated, ts will be updated to the timestamp of the current operation.

[0067] Specifically, sqlite (embedded database) is used to query the location information; the file itself is confirmed through double verification of devnode and signature, so as to obtain the location information based on the log file.

[0068] Step S3, if the location information does not exist, starting from the file start offset position, each data block in the log file to be read is written into the cache area in turn according to the preset data block size, and the location information corresponding to the current data block is stored in the checkpoint database, wherein the sending status of the data block in the location information is to be sent;

[0069] Specifically, the starting offset position of the file is confirmed, the file content is read according to the preset data block size, the data block is written into the buffer area and the corresponding site record is created.

[0070] For example, if a new log file is processed, starting from offset=0, 4096 bytes are read each time, and a new location record is created after writing to the buffer area, and committed is set to 0.

[0071] Step S4, if there is location information, construct / update the sub-element in the log playback array according to the location information, write each data block in the log file to be read into the cache area in turn according to each sub-element information of the log playback array, and store the location information corresponding to the current data block into the checkpoint database, wherein the sending status of the data block in the location information is to be sent;

[0072] Specifically, the uncommitted (committed = 0) site information is put into the playback array, the playback array is sorted according to read_offset, and the data blocks are processed according to the information sequence in the playback array. The orderly playback array is used to achieve breakpoint continuation, ensuring improved data processing efficiency.

[0073] For example, if there is a playback array REArray, each sub-element in REArray corresponds to the location information of a data block, which can be referred to Figure 2 , the array contains Figure 2 The three sub-elements S1, S2, and S3 shown in the figure respectively store the following location information: S1's head end offset 1, data block size 1, and sending status; S2's head end offset 2, data block size 2, and sending status; S3's head segment offset 3, data block size 3, and sending status.

[0074] Specifically, in step S3 or step S4, the location information is updated immediately after the data block is written into the cache area, and the playback array status is checked to determine the next processing position, and the next data block is collected in a loop until all data blocks are processed, thereby realizing continuous collection of data blocks. It can be understood that if any data block to be read in the log file to be read is not written into the cache area, the above steps S1 to S4 are repeated.

[0075] Step S5, writing all the data blocks to be read in the log file to be read into the cache area;

[0076] In this embodiment, the pending state (committed=0) indicates that the data block has been written into the cache but has not been successfully sent. By querying and processing the real-time location information, continuous data block collection is achieved, aiming to ensure parallel processing of subsequent data blocks, thereby improving collection efficiency.

[0077] Step S6, obtaining the data block with the sending status of "to be sent" from the buffer area, sending the data block to be sent to the target location, and at the same time, setting the sending status of the data block in the location information of the data block that has been sent to "sent".

[0078] It should be noted that the target location refers to the destination to which the data block needs to be sent eventually, which may be a remote server, a message queue, etc. The sent status (committed=1) indicates that the data block has been successfully sent to the target location.

[0079] Specifically, a data block with committed=0 is obtained from the cache area, and the data block is sent to the target location through network transmission. When it is confirmed that the data block is sent successfully, the committed field of the corresponding location information in the checkpoint database is updated to 1.

[0080] Exemplarily, a 4096-byte data block with committed=0 is obtained from the cache area, and the data block is sent to the target server through HTTP / TCP and other protocols. After receiving the confirmation response from the server, the site record in the checkpoint database is updated, committed=1, ts is the current timestamp, to achieve parallel processing of data blocks. Even if a data block fails to be sent, it will not affect the processing of other data blocks, avoiding global blocking and improving the overall log collection and sending efficiency.

[0081] The embodiment of the present invention obtains the initial status information of the changed log file to be read by monitoring the status of the file, uses the database to store the location information, and each data block corresponds to an independent location information, avoiding the existing overwriting reading and writing mechanism, providing a data basis for subsequent parallel processing, and querying the database according to the file identification information. When the location information does not exist, the log file is processed block by block starting from the starting position of the file, and each data block is written into the cache area. If the location information already exists, the log playback array is constructed or updated according to this information, and each sub-element is checked. Each sub-element can be written to the cache independently and is independent of the sending process. Regardless of whether the data is sent successfully, the data block of the next log file can continue to be written, avoiding the global blocking problem under the existing method, skipping the data blocks that have been sent successfully, and directly processing the data blocks that have not been sent or failed to send, reducing the repeated re-reading and re-writing process. By storing the location information of each data block in the checkpoint database, when each data block is written to the cache, its status will be recorded in the checkpoint database as waiting to be sent. After the data block is sent successfully, the status is updated to sent. In the last step, the data block to be sent is sent. Even if a data block fails to send, it will not affect the processing of other data blocks. In other words, the reading, caching and sending of data blocks can be carried out in parallel, and the failure to send a single data block will not cause overall processing blocking, thereby improving the overall log collection and sending efficiency.

[0082] In an optional embodiment, step S4 specifically includes: determining whether the log playback array is an empty set; if the log playback array is an empty set, mapping the corresponding data block head end offset, data block size, and data block sending status of each data block to each corresponding sub-element information, and storing the sub-elements in the log playback array in the order of the data block head end offset from small to large. If the log playback array is not an empty set, determining whether the file starting offset position is the same as the data block head end offset corresponding to the first sub-element in the log playback array. If the same, then according to the position of the data block head end offset corresponding to the first sub-element in the log playback array, the data blocks corresponding to each subsequent sub-element are written into the cache area in sequence. If they are not the same, clear the sub-elements in the log playback array, and starting from the file starting offset position, write each data block in the log file to be read into the cache area in sequence according to the preset data block size.

[0083] Specifically, the log playback array is an ordered set that stores metadata of data blocks. An empty set refers to a state where the array does not contain any elements. The empty set can be determined by checking whether the length of the array is 0 or whether there are valid elements.

[0084] For example, see Figure 3 , when the log playback array is empty, the information of the data block needs to be mapped as a sub-element and stored. In specific implementation, a data structure containing these three attributes can be created as a sub-element, sorted from small to large according to read_offset and stored in the array to ensure that the data blocks are processed in the order in which they appear in the file and ensure the continuity of the data. When the log playback array is not empty, it is necessary to determine whether the starting offset position of the current file matches the offset of the first sub-element in the array. The purpose of this judgment is to ensure the continuity of the data block. In specific implementation, the starting offset position of the current file is compared with the read_offset of the first element in the array. If the offsets are the same, it means that the data blocks are continuous. The data blocks can be directly written into the cache in the order recorded in the log playback array. At this time, the processing of the data blocks is continuous and orderly, and there will be no data loss or duplication. If the offsets are different, it means that there may be gaps between the data blocks or the file has changed. It is necessary to clear the log playback array and start reading again, and then read the data blocks from the starting offset position of the current file and write them into the cache in sequence according to the preset data block size to ensure that the correct data block sequence is re-established in the case of discontinuous data.

[0085] In the embodiment of the present invention, if the log playback array is an empty set, the head end offset, size, and sending status of each data block are mapped to corresponding sub-elements, and are stored in the log playback array in the order of the data block head end offset from small to large. By storing the offset in sequence, it can be ensured that the data blocks are processed in sequence according to the order of the files; if the log playback array is not an empty set, it is further determined whether the starting offset position of the current file is the same as the head end offset of the first sub-element in the playback array. When the offsets are the same, it means that the file reading process is continuous, and the subsequent data blocks can be directly written into the cache area in sequence to avoid repeated reading. When the offsets do not match, it may be due to the change of the file and the discontinuity of the data blocks. By clearing the array and re-reading, it is ensured that no omissions or misreading and writing are made, thereby ensuring that the data blocks in the data reading and writing process are continuous, thereby improving the collection efficiency of the log.

[0086] In an optional embodiment, the initial state information further includes a content signature of the file to be read, and the location information further includes a file path; after step S4 of the embodiment of the present invention, the following is further included:

[0087] If the location information exists, then the file identification information is used to search whether there is a corresponding file path in the location information; if there is no corresponding file path, then the checkpoint database is queried according to the content signature and file identification information of the file to be read to determine whether there is a target log file with the same content signature and file identification information as the file to be read. If so, it is determined that the log file to be read is a rolling file, and the file name of the target log file is obtained. A new file path is generated according to the file name of the target log file, and the new file path is used as the file path of the log file to be read and stored in the checkpoint database. If it does not exist, it is determined that the log file to be read is not a rolling file.

[0088] It should be noted that in this embodiment, the content signature of the file to be read is added to the initial state information. The signature is the hash value of the file content. In specific implementation, the hash value of the first 1024 bytes of the file or other preset size content can be calculated. The content signature is used to uniquely identify the file content. Even if the file name changes, as long as the file content remains unchanged, its content signature remains unchanged. A file path (file_path) field is added to the site information to record the complete path information of the file.

[0089] Specifically, the corresponding file path is searched in the site information according to the file identification information (devnode). The file identification information is a unique identifier of the file and can be obtained from the metadata of the file. Even if the file is renamed, the identifier will not change. This is achieved through a database query statement.

[0090] For example, see Figure 4, the line checks whether there is site information, and further determines whether there is a corresponding file path. When it is found that the file path does not exist, it is necessary to determine whether it is a rolling file scenario. Among them, a rolling file means that after the log file reaches a certain size, it will be renamed and a new log file will be created. The content signature (signature) and file identification information (devnode) of the file are queried in the checkpoint database. If the query result exists, it means that this is a rolling file. This is because the rolling file has the same file identification information and content signature (that is, the beginning of the file content has not changed). For the case of confirming that it is a rolling file, the file path information needs to be updated, the file name of the target log file is obtained, and then a new complete file path is generated based on the file name. For example, if the original file a.log is renamed to a.log.1, the file path needs to be updated from / xx / xx / a.log to / xx / xx / a.log.1. If no matching record is found in the checkpoint database (that is, neither the content signature nor the file identification information matches), it means that this file is not a rolling file, it may be a brand new file or a deleted file, and further processing is required.

[0091] Further, if it is determined that the log file to be read is not a rolling file, the location information corresponding to the log file to be read is deleted.

[0092] The embodiment of the present invention deletes the location information corresponding to the log file that is not a rolling file, determines that the log file is a new file, and deletes the corresponding location information, thereby avoiding erroneous reading and writing of data, thereby improving log collection efficiency.

[0093] In the embodiment of the present invention, if the log playback array is an empty set, the head end offset, size, and sending status of each data block are mapped to corresponding sub-elements, and are stored in the log playback array in the order of the data block head end offset from small to large. By storing the offset in sequence, it can be ensured that the data blocks are processed in sequence according to the order of the files; if the log playback array is not an empty set, it is further determined whether the starting offset position of the current file is the same as the head end offset of the first sub-element in the playback array. When the offsets are the same, it means that the file reading process is continuous, and the subsequent data blocks can be directly written into the cache area in sequence to avoid repeated reading. When the offsets do not match, it may be due to the change of the file and the discontinuity of the data blocks. By clearing the array and re-reading, it is ensured that no omissions or misreading and writing are made, thereby ensuring that the data blocks in the data reading and writing process are continuous, thereby improving the collection efficiency of the log.

[0094] In an optional embodiment, step S6 specifically includes: constructing a preset number of multiple threads; obtaining the first data block with a sending status of to-be-sent from the cache area based on any one of the multiple threads, and sending the data block to-be-sent to the target location; if the sending is successful, obtaining the next data block with a sending status of to-be-sent, and sending the data block to-be-sent to the target location, until all the data blocks to be sent are sent to the target location, and at the same time, setting the sending status of the data block in the location information of the data block that has been sent to sent; if the sending fails, confirming that the current thread is blocked, and continuing to send the data block that failed to send to the target location based on any other unblocked thread, until all the data blocks to be sent are sent to the target location, and at the same time, setting the sending status of the data block in the location information of the data block that has been sent to sent.

[0095] It should be noted that, in this embodiment, constructing a preset number of threads refers to creating a thread pool, the number of threads can be flexibly adjusted according to the configuration, and the threads are all working threads of the sending module, which are used to process the sending task of the data block.

[0096] For details, please refer to Figure 5 , after the component has a preset number of threads, the process of obtaining the data block to be sent from the cache adopts a preemptive polling mechanism. When a thread successfully obtains a data block, it immediately sends the data block to the target location. After the sending is successful, the committed state of the corresponding data block in the checkpoint database is updated to 1, and the ts field is updated to the current timestamp. After detecting the failure of sending, the failed data block is put back into the cache, and the current thread is marked as blocked. Other unblocked threads can continue to obtain data blocks (previously failed to send data blocks) from the cache for sending. Through multi-threaded parallel processing and automatic failure transfer mechanism, the overall processing flow will not be affected by the failure of sending a single data block, thereby improving the efficiency of data sending.

[0097] The embodiment of the present invention constructs a preset number of multiple threads, and multi-threaded parallel processing avoids the back pressure mechanism of a single thread. Based on any one of the multiple threads, the first data block with a sending status of to-be-sent is obtained from the cache area, and the data block to-be-sent is sent to the target location. The multiple threads can sequentially obtain data from the cache area and send it, thereby improving the sending efficiency. If the sending is successful, the next data block with a sending status of to-be-sent is obtained, and the data block to-be-sent is sent to the target location until all the data blocks to be sent are sent to the target location, thereby maintaining processing continuity. If the sending fails, it is confirmed that the current thread is blocked, and based on any other unblocked thread, the data block that failed to be sent continues to be sent to the target location, and the processing can be switched to other unblocked threads immediately for continued processing, thereby realizing automatic thread transfer after a failure. In case of failure, the thread can be quickly switched to continue processing, thereby effectively improving the overall efficiency of log collection and sending.

[0098] In an optional embodiment, the above method also includes: when the data block is written to the cache area in step S3 or step S4, or when the sub-elements are stored in the log playback array in ascending order of the data block head end offset, the current moment is converted into a corresponding timestamp, and the timestamp information in the location information is updated; determine whether the corresponding location information is expired according to the timestamp information, and when the location information has expired, clear the location information.

[0099] Specifically, after the data block is written into the memory buffer in step S3 and step S4, a pre-write checkpoint operation is required. When inserting or updating the corresponding site information in the checkpoint table, the current time is converted to a timestamp format, and the ts field of the site information is updated to a new timestamp; or, when the sub-elements are stored in the log playback array in the order of the data block head end offset from small to large, the ts field of the sub-element site information is updated to the current timestamp. Regularly check and clean up expired site information, wherein the current system time is obtained and converted to a corresponding timestamp format, all site information in the checkpoint table is traversed, the ts field of each record is read, the current timestamp is compared with the ts of each record, and the time difference is calculated. If the time difference exceeds the preset expiration threshold (for example, 7 days), the site information is considered expired, and the expired site information is deleted from the checkpoint table. By automatically cleaning up expired data, too many invalid records can be avoided from accumulating in the checkpoint database.

[0100] In the embodiment of the present invention, whenever a data block is written into the cache area or the log playback array is updated, the timestamp of the current moment is recorded. By adding the timestamp to the location information, the latest processing time of each data block can be recorded. By clearing the expired location information, a large number of useless location records can be effectively avoided from being accumulated, and the invalid data storage of the log collector in the checkpoint database can be reduced, which directly improves the efficiency of querying the location information that needs to be processed currently, and further improves the efficiency of log collection.

[0101] In actual application scenarios, the overall implementation process of a log collection and sending method provided by an embodiment of the present invention can be referred to Figure 6 , divided into acquisition module and sending module group. The acquisition module loads and verifies the checkpoint, arranges the playback segment, and reads the file block along the playback position if there is a playback segment; otherwise, read along the latest position, write the buffer, pre-write the checkpoint, and the status is uncommitted. The sending module group reads the file block from the buffer and sends the data. If the sending fails, it continues to read the file block and send the data. If the sending succeeds, it updates the checkpoint status to committed and cleans up the checkpoint file.

[0102] In the actual prior art, the process of collecting and sending log data can be referred to Figure 7 and Figure 8 , if a file has 3 file blocks S1, S2, and S3 to be collected. After S1 is sent successfully, the collector will save the end position of the S1 file block to the checkpoint. If the collector is restarted or an exception occurs, the collector will continue to collect the subsequent file block S2 along the end position of S1. Each file will only save one collection site in the checkpoint, and the latest site will overwrite the old site. At the same time, the data before the collection site in the checkpoint file has been successfully collected and sent to the target location. That is to say, the next file will be read only after the file block is read and sent successfully. If a send fails, the collector will send and block here in a loop until the send is successful. During the blocking process of sending, the collection thread is also blocked due to the back pressure mechanism of the cache queue.

[0103] Therefore, compared with the log collection method in the prior art, the embodiments of the present invention have the following beneficial technical effects:

[0104] (1) By introducing the starting address of the file block, the size of the file block, the submission status, etc., the states of the collector sites are made independent of each other. The design of this data format is the core part of the checkpoint mechanism. It decouples the sending module and the collection module, allowing the sending module to be expanded in parallel, improving the overall collection performance, and avoiding the situation where the states of the collection sites in the existing methods are dependent on each other, thereby greatly improving the overall efficiency of the collector.

[0105] (2) Use the checkpoint data format to replay the log and check the rollback log file to ensure the breakpoint continuation and rolling scene collection after the log is interrupted, and ensure the integrity of the log.

[0106] (3) Independent relationship between the acquisition module and the sending module. Different from the prior art, the acquisition module has to wait for the sending module to update the acquisition location. In the embodiment of the present invention, the sending module group can be linearly expanded according to the preset value to improve the sending efficiency.

[0107] In this embodiment, a log collection and sending device is also provided. The system is used to implement the above-mentioned embodiments and preferred implementation modes. The descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0108] This embodiment provides a log collection and sending device, such as Fig. 9 As shown, including:

[0109] A file monitoring module, configured to obtain initial status information of the changed log file to be read when a change in the log file is detected, wherein the initial status information includes file identification information of the log file to be read, a starting offset position of the file, and a size of a preset data block, and the log file to be read includes multiple data blocks to be read;

[0110] A checkpoint query module is used to query a preset checkpoint database according to the file identification information to determine whether there is location information corresponding to the log file to be read, wherein the location information represents a variety of data status information of the data block to be read in the log file to be read, each data block to be read corresponds to a location information, and the location information at least includes a sending status of the data block;

[0111] An initial data writing module is used to write each data block in the log file to be read into the cache area in sequence according to the size of the preset data block, starting from the starting offset position of the file if the location information does not exist, and store the location information corresponding to the current data block into the checkpoint database; wherein the sending status of the data block in the location information is to be sent;

[0112] A data update module, for constructing / updating sub-elements in a log playback array according to the location information if the location information exists, writing each data block in the log file to be read into the buffer area in turn according to each sub-element information of the log playback array, and storing the location information corresponding to the current data block into the checkpoint database; wherein the sending status of the data block in the location information is to be sent;

[0113] A location recording module is used to write all the data blocks to be read in the log file to be read into the buffer area; repeat the above initial data writing module and data updating module;

[0114] The data sending module is used to obtain the data block with the sending status of waiting to be sent from the buffer area, send the data block to be sent to the target location, and at the same time, set the sending status of the data block in the location information of the data block that has been sent to sent.

[0115] In some optional implementations, the location information further includes a data block head end offset and a data block size; the data update module specifically includes:

[0116] A log playback array empty set judgment unit, used to judge whether the log playback array is an empty set;

[0117] The data block mapping storage unit is used to map the corresponding data block head end offset, data block size, and data block sending status of each data block to each corresponding sub-element information if the log playback array is an empty set, and store the sub-elements in the log playback array in the order of the data block head end offset from small to large;

[0118] The offset matching processing unit is used to determine whether the file starting offset position is the same as the data block head end offset corresponding to the first sub-element in the log playback array if the log playback array is not an empty set. If they are the same, the data blocks corresponding to each subsequent sub-element are written into the cache area in sequence according to the position of the data block head end offset corresponding to the first sub-element in the log playback array; if they are not the same, the sub-elements in the log playback array are cleared, and starting from the file starting offset position, each data block in the log file to be read is written into the cache area in sequence according to the preset data block size.

[0119] In some optional implementations, the initial state information further includes a content signature of the file to be read, and the location information further includes a file path; the above-mentioned device further includes:

[0120] A path detection module is used to find out whether there is a file path corresponding to the location information according to the file identification information if the location information exists;

[0121] The rolling file detection module is used to query the checkpoint database based on the content signature and file identification information of the file to be read if there is no corresponding file path, and determine whether there is a target log file with the same content signature and file identification information as the file to be read. If so, it is determined that the log file to be read is a rolling file, and the file name of the target log file is obtained. A new file path is generated according to the file name of the target log file, and the new file path is used as the file path of the log file to be read and stored in the checkpoint database. If it does not exist, it is determined that the log file to be read is not a rolling file.

[0122] In some optional implementations, if it is determined that the log file to be read is not a rolling file, the location information corresponding to the log file to be read is deleted.

[0123] In some optional implementations, the data sending module specifically includes:

[0124] A thread pool construction unit is used to construct a preset number of threads;

[0125] A data block sending unit, configured to obtain a first data block in a sending state to be sent from a buffer area based on any one of the multiple threads, and send the data block to be sent to a target location;

[0126] A sending success processing unit is used for obtaining the next data block with a sending status of to-be-sent if the sending is successful, and sending the data block to-be-sent to the target location until all the data blocks to be sent are sent to the target location, and at the same time, setting the sending status of the data block in the location information of the data block that has been sent to sent;

[0127] The sending failure processing unit is used to confirm that the current thread is blocked if the sending fails, and continue to send the data block that failed to be sent to the target location based on any other unblocked thread until all the data blocks to be sent are sent to the target location. At the same time, the sending status of the data block in the location information of the data block that has been sent is set to sent.

[0128] In some optional implementations, the location information further includes timestamp information; and the method further includes:

[0129] A timestamp update module is used to convert the current moment into a corresponding timestamp and update the timestamp information in the location information when a data block is written into the cache in the initial data writing module or the data updating module, or when the sub-elements are stored into the log playback array in the order of the offset of the head end of the data block from small to large;

[0130] The expired site clearing module is used to determine whether the corresponding site information is expired according to the timestamp information, and clear the site information when the site information has expired.

[0131] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0132] The log collection and sending device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0133] The embodiment of the present invention also provides a computer device having the above Fig. 9 The log collection and sending device shown.

[0134] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0135] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0136] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0137] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0139] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0140] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0141] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0142] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the defined scope.

Claims

1. A log collection and sending method, characterized in that: The method comprises: Step S1, when a change in a log file is detected, initial state information of the changed log file to be read is obtained, wherein the initial state information includes file identification information of the log file to be read, a starting offset position of the file, and a size of a preset data block, and the log file to be read includes multiple data blocks to be read; Step S2, querying a preset checkpoint database according to the file identification information to determine whether there is location information corresponding to the log file to be read, wherein the location information represents a variety of data status information of a data block to be read in the log file to be read, each data block to be read corresponds to a location information, and the location information at least includes a sending status of the data block; Step S3, if the location information does not exist, starting from the starting offset position of the file, each data block in the to-be-read log file is sequentially written into the cache area according to the size of the preset data block, and the location information corresponding to the current data block is stored in the checkpoint database; wherein the sending status of the data block in the location information is to be sent; Step S4, if the location information exists, construct / update the sub-element in the log playback array according to the location information, write each data block in the log file to be read into the buffer area in turn according to each sub-element information of the log playback array, and store the location information corresponding to the current data block into the checkpoint database; wherein the sending status of the data block in the location information is to be sent; repeat the above steps S1 to S4; Step S5, writing all the data blocks to be read in the log file to be read into the buffer area; Step S6, obtaining a data block with a sending status of "to be sent" from the buffer area, sending the data block to be sent to the target location, and at the same time, setting the sending status of the data block in the location information of the data block that has been sent to "sent".

2. The method according to claim 1, characterized in that: The location information also includes the data block head end offset and the data block size; step S4 specifically includes: Determine whether the log playback array is an empty set; If the log playback array is an empty set, the corresponding data block head end offset, data block size, and data block sending status of each data block are mapped to each corresponding sub-element information, and the sub-elements are stored in the log playback array in the order of the data block head end offset from small to large; If the log playback array is not an empty set, determine whether the starting offset position of the file is the same as the data block head offset corresponding to the first sub-element in the log playback array; if they are the same, write the data blocks corresponding to each subsequent sub-element into the cache area in sequence according to the position of the data block head offset corresponding to the first sub-element in the log playback array; if they are not the same, clear the sub-elements in the log playback array, and start from the starting offset position of the file, write each data block in the log file to be read into the cache area in sequence according to the size of the preset data block.

3. The method according to claim 2, characterized in that The initial state information also includes a content signature of the file to be read, and the location information also includes a file path; the method further includes: If the location information exists, searching, according to the file identification information, whether there is a file path corresponding to the location information; If there is no corresponding file path, the checkpoint database is queried according to the content signature of the file to be read and the file identification information to determine whether there is a target log file with the same content signature and file identification information as the file to be read. If so, it is determined that the log file to be read is a rolling file, and the file name of the target log file is obtained. A new file path is generated according to the file name of the target log file, and the new file path is used as the file path of the log file to be read and stored in the checkpoint database. If it does not exist, it is determined that the log file to be read is not a rolling file.

4. The method according to claim 3, characterized in that: If it is determined that the log file to be read is not a rolling file, the location information corresponding to the log file to be read is deleted.

5. The method according to claim 4, characterized in that Step S6 specifically includes: Build a preset number of multiple threads; Based on any one of the multiple threads obtaining a first data block in a sending state to be sent from the buffer area, sending the data block to be sent to a target location; If the sending is successful, the next data block with a sending status of to-be-sent is obtained, and the data block to-be-sent is sent to the target location until all the data blocks to be sent are sent to the target location. At the same time, the sending status of the data block in the location information of the data block that has been sent is set to sent; If the sending fails, it is confirmed that the current thread is blocked, and any other unblocked thread will continue to send the data block that failed to be sent to the target location until all the data blocks to be sent are sent to the target location. At the same time, the sending status of the data block in the location information of the data block that has been sent is set to sent.

6. The method according to any one of claims 1 to 5, characterized in that: The location information also includes timestamp information; the method also includes: When the data block is written into the cache in step S3 or step S4, or when the sub-elements are stored in the log playback array in the order of the data block head end offset from small to large, the current moment is converted into a corresponding timestamp, and the timestamp information in the location information is updated; Determine whether the corresponding location information is expired according to the timestamp information, and clear the location information when the location information is expired.

7. A log collection and sending device, characterized in that: The device comprises: A file monitoring module, configured to obtain initial status information of the changed log file to be read when a change in the log file is detected, wherein the initial status information includes file identification information of the log file to be read, a starting offset position of the file, and a size of a preset data block, and the log file to be read includes multiple data blocks to be read; A checkpoint query module is used to query a preset checkpoint database according to the file identification information to determine whether there is location information corresponding to the log file to be read, wherein the location information represents a variety of data status information of the data block to be read in the log file to be read, each data block to be read corresponds to a location information, and the location information at least includes a sending status of the data block; An initial data writing module is used to write each data block in the log file to be read into the cache area in sequence according to the size of the preset data block, starting from the starting offset position of the file if the location information does not exist, and store the location information corresponding to the current data block into the checkpoint database; wherein the sending status of the data block in the location information is to be sent; A data update module, for constructing / updating the sub-elements in the log playback array according to the location information if the location information exists, and writing each data block in the log file to be read into the buffer area in turn according to each sub-element information of the log playback array, and storing the location information corresponding to the current data block into the checkpoint database; wherein the sending status of the data block in the location information is to be sent; repeating the above initial data writing module and data update module; A location recording module, used for writing all the data blocks to be read in the log file to be read into the buffer area; The data sending module is used to obtain the data block with the sending status of waiting to be sent from the buffer area, send the data block to be sent to the target location, and at the same time, set the sending status of the data block in the location information of the data block that has been sent to sent.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the log collection and sending method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the log collection and sending method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the log collection and sending method according to any one of claims 1 to 6.

Images