Log collection method and device across nodes

Abstract

The application is suitable for the technical field of data processing, and provides a cross-node log collection method and device. The method comprises the following steps: acquiring link node information and current collection state information; wherein, the link node information comprises all micro-service nodes in a node link and calling sequences between the micro-service nodes, the collection state information comprises paging state information and node state information, the paging state information represents a paging pulling progress in a single micro-service node which is currently collected, and the node state information represents a micro-service node which is currently collected; determining a target collection position according to the link node information and the current collection state information; wherein, the target collection position represents a micro-service node and / or a paging position which is to be collected; performing a log data pulling operation at the target collection position, and updating the collection state information. The link node information across nodes is constructed, the paging state and the node state are jointly maintained, and the precision and efficiency of log collection are improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a method and apparatus for cross-node log collection. Background Technology

[0002] With the development of microservice architecture and cloud-native technologies, a single business request in a distributed system typically spans multiple microservice nodes, including API (Application Programming Interface) gateways, application services, message components, caching components, and downstream third-party interfaces, resulting in scattered log records.

[0003] How to accurately collect logs across nodes is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a method and apparatus for cross-node log collection, which enables cross-node log collection and improves the efficiency and accuracy of log collection.

[0005] In a first aspect, embodiments of this application provide a cross-node log collection method, including: Obtain link node information and current collection status information; wherein, the link node information includes all microservice nodes in the node link and the call order between each microservice node, the collection status information includes at least one of pagination status information and node status information, the pagination status information represents the pagination pull progress within the currently collected single microservice node, and the node status information represents the currently collected microservice node. Based on the link node information and the current collection status information, the target collection location is determined; wherein, the target collection location represents the microservice node and / or pagination location to be collected; Perform log data retrieval at the target collection location and update the collection status information.

[0006] Secondly, embodiments of this application provide a cross-node log collection device, comprising: The information acquisition module is used to acquire link node information and current collection status information; wherein, the link node information includes all microservice nodes in the node link and the retrieval order between each microservice node, and the collection status information includes at least one of pagination status information and node status information, wherein the pagination status information represents the pagination retrieval progress within the currently collected single microservice node, and the node status information represents the currently collected microservice node. The location determination module is used to determine the target collection location based on the link node information and the current collection status information; wherein, the target collection location represents the microservice node to be collected and / or the pagination location; The log retrieval module is used to retrieve log data at the target collection location and update the collection status information.

[0007] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the method described in the first aspect above.

[0010] In this embodiment, within a distributed system, business requests can invoke multiple microservice nodes. Based on the invocation order among these microservice nodes, link node information can be obtained. This link node information includes all microservice nodes in the node link and the invocation order between them, thereby constructing a unified state space across nodes. Current collection status information can also be obtained, including pagination status information and node status information. Pagination status information represents the pagination retrieval progress within a single microservice node currently being collected, while node status information represents the microservice node currently being collected, enabling joint maintenance of pagination and node states. Based on the link node information and the current collection status information, the microservice node to be collected and / or the pagination position is determined as the target collection position. Log data retrieval is performed at the target collection position to achieve accurate log collection. The collection status information is updated to facilitate subsequent collection, thus transforming the originally stateless log collection process, which required full re-collection, into a measurable and controllable convergence process, making the collection process observable and improving the accuracy and efficiency of log collection. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the cross-node log collection method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the log collection process provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the cross-node log collection method provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the cross-node log collection method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the cross-node log collection device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0018] The cross-node log collection method provided in this application embodiment can be applied to terminal devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, computers, servers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application embodiment does not impose any restrictions on the specific type of terminal device.

[0019] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: Microservice nodes: Independent service units (processes / containers / instances) that participate in processing business requests in a distributed system, and are the source of log generation. In this embodiment, microservice nodes include, but are not limited to, gateways, business services, middleware (messages / caches), and third-party interfaces.

[0020] Node chain: A chain of calls that a single business request passes through in sequence through multiple microservice nodes, linked together according to their call relationships.

[0021] Pagination: Since the log interface cannot return all logs at once due to large data volumes, it can return them in batches, i.e., page by page. The data collector needs to retrieve the data page by page and can use "page position / page cursor" to record which page they have reached.

[0022] With the development of microservice architecture and cloud-native technologies, a single business request in a distributed system typically spans multiple microservice nodes, including API gateways, application services, message components, caching components, and downstream third-party interfaces, generating scattered records across multiple log storage systems. To achieve fault diagnosis, problem backtracking, and external data verification, it is necessary to collect relevant logs across nodes and instances.

[0023] In scenarios where external log integrity is ensured, the collection task is typically not aimed at a single fixed log file, but rather at supplementing the collection of logs along the entire node chain corresponding to a single business request. This chain is inherently dynamic in actual operation; for example, microservice nodes may change due to elastic scaling, rolling deployments, and fault migration.

[0024] In the current distributed log collection scenario, the collection process lacks a unified state modeling mechanism for dynamic links, making it impossible to continuously track the pagination progress of each node, the set of discovered nodes, the set of nodes to be processed, and the task execution status. As a result, it is difficult to support continuous collection and supplementary collection in complex scenarios.

[0025] Furthermore, current external log collection systems generally lack effective breakpoint mechanisms, and a full re-collection is usually required after a failure. In the event of network anomalies, authentication failures, interface rate limiting, or interruption of the collection task, it is usually necessary to re-initiate a full query with the initial conditions, which increases the pressure on the external log platform, reduces the timeliness of re-collection, and deteriorates the consistency of results.

[0026] Therefore, there is currently no technical solution that can provide unified state modeling, breakpoint maintenance, dynamic supplementary collection, and feedback convergence control for the distributed external log collection process of dynamic microservice nodes.

[0027] General Description of Embodiments in this Application According to one embodiment of this application, a method for ensuring the integrity of outgoing logs in a distributed system is provided. By jointly maintaining the pagination state, node state, and breakpoint state, and introducing a feedback control mechanism based on a multidimensional evaluation function, the data collection process, which was originally stateless, had no determinable termination conditions, and required full re-collection after failure, is transformed into a measurable, controllable, and convergent process that can continue execution from anomaly points.

[0028] Figure 1 This is a flowchart illustrating a cross-node log collection method, such as... Figure 1 As shown, a cross-node log collection method according to one embodiment of this application may include: Step 110: Obtain link node information and current collection status information; wherein, the link node information includes all microservice nodes in the node link and the retrieval order between each microservice node, and the collection status information includes at least one of pagination status information and node status information, the pagination status information represents the pagination retrieval progress within the currently collected single microservice node, and the node status information represents the currently collected microservice node. Step 120: Determine the target collection location based on the link node information and the current collection status information; wherein, the target collection location represents the microservice node and / or pagination location to be collected; Step 130: Retrieve log data at the target collection location and update the collection status information.

[0029] Steps 110-130 are described in detail below.

[0030] In step 110, a single business request passes through multiple microservice nodes in a preset order, forming a node link for that business request. Link node information refers to the composition of the node link corresponding to a single business request. For each type of business request, the currently known link node information can be directly obtained. During log collection, microservice nodes may change due to elastic scaling, rolling deployment, fault migration, etc. The link node information can be updated subsequently based on the updated microservice nodes. Link node information can include all microservice nodes in the node link and the call order between each microservice node. The call order reflects the upstream and downstream dependencies between nodes in the node link, and its function is to depict the topology of the node link, ensuring that the link node information not only includes the set of nodes but also the connection logic between nodes, thereby supporting the expansion of the collection scope and the location of the target collection position. Link node information constitutes a unified state space across nodes, providing the node range and call basis for the log collection process, enabling subsequent planning based on the sequential call order of nodes when determining the target collection position.

[0031] The collection status information refers to the global status content maintained during the log collection task's execution, which can include pagination status information and node status information. Pagination status information represents the pagination retrieval progress within a single microservice node. It can record the position identifier corresponding to the pagination query, the cumulative data volume retrieved, and whether the pagination query has been exhausted, etc., and is used to manage the progress of batch data retrieval within a single microservice node. For example, the pagination status information can record the running context of data pagination retrieval within a single node, including the pagination cursor, pagination offset position, and next page valid identifier, to continuously advance the batch data retrieval process within a single node. Node status information represents the microservice nodes that have been collected. Node status information can maintain a set of discovered microservice nodes, pending microservice nodes, and completed microservice nodes, used to manage the traversal and flow of all microservice nodes. Collection status information can be obtained in real time during the log collection task's execution.

[0032] In step 120, the link node information is the link topology association relationship, and the current collection status information is the collection progress that has been formed. Combining the link topology association relationship and the collection progress that has been formed, the operation objects corresponding to the next log retrieval are selected. This can be done by selecting the microservice node to be collected, selecting the specific page position to be read within a microservice node, or simultaneously determining the microservice node to be collected and the page position corresponding to that microservice node.

[0033] For example, the last microservice node currently being collected can be determined based on the node status information. Then, the node following this node can be found in the link node information as the next microservice node to be collected. This next microservice node is the target collection location. Alternatively, the page position that has not been fully retrieved can be read from the page status information to determine the page position to be collected, which is also the target collection location.

[0034] In step 130, a log query request is initiated for the identified microservice nodes and / or pagination positions to be collected, i.e., log data is pulled at the target collection position. After obtaining the log data, the pagination status information and node status information are modified synchronously, and the new pull progress and node processing status are written into the collection status information. After the collection status information is updated, it can be persistently saved to provide updated status basis for subsequent collection operations.

[0035] For example, after the fetch operation completes the collection of one page of log data from a single microservice node, the pagination fetch progress in the pagination status information corresponding to that microservice node can be updated. If all paginated data for that microservice node has been fetched, the node status information is synchronously modified to change the microservice node to a node that has completed processing. Link node information can be carried by a graph structure or a queue structure, and the collected status information can be stored in a database, cache, or distributed consistent storage. The reading and updating of the collected status information can be combined with a snapshot mechanism to complete state retention and support state recovery in abnormal scenarios.

[0036] In this embodiment, by acquiring link node information and collection status information, the target collection location is determined, and log retrieval and collection status information are executed. The log collection process is driven by the link topology and real-time collection progress, enabling state-driven log collection across microservice nodes. This facilitates continuous tracking of pagination retrieval progress and microservice node traversal.

[0037] In one implementation, the method further includes: receiving user-inputted collection entry parameter information, determining link node information corresponding to the collection entry parameter information; wherein the collection entry parameter information represents the node link to be called for collecting log data; and generating initialized pagination status information and node status information.

[0038] Specifically, users can input data collection entry parameters through the human-computer interaction interface. These parameters are the input received when starting the log collection task and can include at least one of the following: Trace ID, keywords, or interface path. A complete node chain is called a Trace; the Trace ID is the unique identifier of the node chain; keywords are used for fuzzy matching of the search text in the Trace record; and the interface path is the interface access address provided by the microservice to locate the target interface. The data collection entry parameters characterize the node chain required for collecting log data, defining the retrieval target and starting range of the log collection task, thereby deriving the initial link node information and providing the initial basis for pagination status information and node status information.

[0039] The system receives user-input data collection entry parameters, constructs link node information, and initializes data collection status information. This status information includes pagination status and node status information, and may also include breakpoint recovery status information. Breakpoint recovery status information does not directly store pagination status information such as the pagination cursor, nor does it directly store node status information such as the list of nodes to be collected. Instead, it records where to retrieve a complete snapshot containing all of the above information.

[0040] During the task startup phase, the externally input collection entry parameter information is read as the collection retrieval conditions. Based on the retrieval conditions, the microservice node corresponding to the start of the collection task and the call association between the nodes are parsed to obtain the link node information. The initial collection status information can be empty.

[0041] For example, the microservice node and call order corresponding to the business link to which the TraceID belongs can be obtained by parsing the received TraceID, thereby forming link node information; the microservice to which the interface belongs and the associated call node can be obtained by matching the received interface path, thereby forming link node information; or the business link that meets the search characteristics can be obtained by matching the received keywords, and the microservice node corresponding to the business link can be extracted to generate link node information.

[0042] Based on the obtained link node information, a pagination retrieval progress record and microservice node management record are constructed in the initial state of the data collection task, completing the initial construction of the data collection status information. The initialized pagination status information sets the pagination position identifier of each microservice node to the initial state of not having started retrieval, without carrying historical pagination offset information; the initialized node status information assigns the microservice nodes obtained from parsing the link node information to the set of microservice nodes to be processed, while setting the remaining node sets to empty. The initialized pagination status information and node status information can form an initial snapshot, i.e., the initialized breakpoint recovery status information, used for subsequent state recovery in abnormal scenarios.

[0043] In this embodiment, by receiving the collection entry parameter information and parsing it to obtain the corresponding link node information, and then generating the initial pagination status information and node status information, the state modeling of the collection task can be completed based on the external input retrieval conditions, providing an initial operating foundation for state-driven cross-node log collection.

[0044] In one implementation, the method further includes: determining the indicator information of preset collection indicators according to a preset time period; wherein the preset collection indicators include at least one of page exhaustion rate, node coverage rate, and anomaly coverage rate, wherein the page exhaustion rate represents the ratio of the number of microservice nodes whose pagination has been exhausted to the number of microservice nodes that have started pagination fetching, the node coverage rate represents the ratio of the number of microservice nodes that have started collection to the number of microservice nodes in the link node information, and the anomaly coverage rate represents the ratio of the number of collected data identified as abnormal to the preset number of anomalies; if the indicator information of any preset collection indicator meets the preset collection termination condition, the log data fetching operation is terminated.

[0045] Specifically, several preset collection metrics are established. These preset metrics are evaluation dimensions used to assess the completeness of log collection tasks and may include at least one of page exhaustion rate, node coverage rate, and anomaly coverage rate. These preset metrics are used to quantitatively evaluate the entire collection process from multiple perspectives, including the completeness of paginated collection, the extent of microservice node topology traversal, and the completeness of anomaly data acquisition. The resulting metrics can serve as a basis for determining whether to continue log data retrieval operations, thereby providing feedback and adjusting subsequent collection behavior.

[0046] Page exhaustion rate represents the ratio of the number of microservice nodes whose pagination has been exhausted to the number of microservice nodes that have started pagination fetching. It reflects the completion status of log pagination collection within each microservice node and indicates whether sufficient log data has been collected from the node. Node coverage rate represents the ratio of the number of microservice nodes that have started collection to the number of microservice nodes in the link node information. It reflects the actual collection progress of all microservice nodes included in the link node information and indicates the completeness of microservice node traversal at the link topology level. Anomaly coverage rate represents the ratio of the number of collected data identified as anomalies to the preset number of anomalies. It reflects the completeness of obtaining anomaly-related log records corresponding to the collection task target and is suitable for log supplementation scenarios for fault diagnosis.

[0047] Based on a preset time period and the accumulated statistical data during the collection process, the corresponding indicator information for each preset collection indicator is calculated. For example, a periodic timer can be triggered after each round of microservice node pagination fetching is completed. After the timer expires, the page exhaustion rate indicator information is obtained by counting the microservice nodes that have exhausted pagination and the microservice nodes that have started pagination fetching. The various sets of microservice nodes recorded in the node status information are read and combined with the microservice nodes recorded in the link node information to calculate the node coverage indicator information. The log records that have been fetched and marked as abnormal data are counted and compared with the preset number of abnormalities to calculate the abnormal coverage indicator information.

[0048] The calculation of the preset collection indicators can be achieved based on the multidimensional evaluation function. The multidimensional evaluation function can be constructed in the form of a rule engine and a statistical model. The statistical source data required for the indicator information calculation process can be taken from the pagination status information and the node status information.

[0049] Pre-set collection termination conditions: If any preset collection metric meets the pre-defined termination conditions, the log data retrieval operation is terminated. In other words, the calculated metric information is compared with the pre-configured termination conditions. When at least one preset collection metric meets the corresponding criterion, the cross-node log retrieval process is stopped, thus proactively exiting the collection task and avoiding the waste of system resources from continuous, ineffective collection. Different preset collection metrics can correspond to different collection termination conditions.

[0050] For example, a preset threshold can be configured as the termination condition for data collection. When the information of the preset collection indicators reaches the corresponding threshold, the log data retrieval operation is terminated. Alternatively, multiple preset collection indicators can be integrated and evaluated. When the integrated result meets the termination condition, the log data retrieval operation is terminated. The indicator information corresponding to the preset collection indicators can be persistently stored and output along with the collection results to represent the complete evaluation of this log collection task. After the conditions are met, the system outputs the collection results, which include at least the node topology, data volume statistics for each node, a list of abnormal logs, and the integrity evaluation results.

[0051] In this embodiment, page exhaustion rate, node coverage rate, and anomaly coverage rate are acquired according to a preset time period, and the start and stop of log data retrieval operations are controlled based on the collection termination conditions. The page exhaustion level, node coverage rate, and anomaly evidence coverage rate are calculated periodically. When the calculation results do not reach the threshold, pagination retrieval or node expansion continues. When the calculation results meet the preset conditions, the collection process terminates. This achieves a quantitative assessment of the completeness of log collection, overcoming the limitation of terminating tasks solely based on interface return results, and reducing unnecessary resource overhead while ensuring the integrity of log collection.

[0052] In one implementation, the method further includes: if the log data retrieval operation is interrupted, after restarting the retrieval operation, determining the current retrieval location based on the current retrieval status information; wherein the current retrieval location represents the currently retrieval microservice node and / or pagination location; and determining the current retrieval location as the target retrieval location.

[0053] Specifically, the collection status information refers to the global status content maintained during the log collection task's execution, which may include pagination status information, node status information, and breakpoint recovery status information. This collection status information can be updated and persistently saved after each log data retrieval is completed, fully preserving the pagination progress and microservice node traversal status before task interruption. This provides complete contextual material for task interruption recovery scenarios, facilitating the determination of the current collection location.

[0054] The current collection location represents the microservice node and / or page position currently being collected. It can be obtained by parsing the collection status information retained at the time of the interruption. It is used to reproduce the operation point of the collection process when the interruption occurred, so as to facilitate the subsequent recovery of the collection.

[0055] When network anomalies, authentication anomalies, or interface rate limiting anomalies occur, the system can recover the pagination and node status information at the time of interruption based on the collected status information, such as the pagination and node status information at the time of interruption, based on the breakpoint recovery status information. This allows the system to determine the pagination position, the set of discovered nodes, the set of nodes to be processed, and the set of completed nodes at the time of interruption, thus identifying the current collection position, which is also the anomaly point. This current collection position is then designated as the target collection position, and a differentiated recovery strategy is executed based on the anomaly type, continuing data collection from the anomaly point.

[0056] The collected status information can be persisted using a snapshot mechanism. Snapshots can be stored in a database, cache system, or distributed consistent storage. The reading and parsing of the collected status information can be combined with differentiated recovery strategies to adapt to different types of interruption causes. For example, for interruptions caused by network anomalies, the recovery strategy can start from the moment of interruption; for interruptions caused by authentication anomalies, the recovery strategy can start from the moment before the interruption.

[0057] In this embodiment, after the log retrieval operation is interrupted and restarted, the current retrieval location is determined based on the retrieval status information, and the current retrieval location is used as the target retrieval location. This allows the retrieval to be resumed directly from the interrupted point after the retrieval task is recovered from an abnormal interruption, avoiding the need to re-collect all data from the beginning, reducing the overhead of repeated data retrieval, and ensuring the consistency of the retrieval context.

[0058] Figure 2 This is a schematic diagram of the log collection process. Figure 2 The system reads the current status in real time. If the current status is normal, it can perform paginated log collection to obtain log data. It monitors the current pagination status, node status, and breakpoint recovery status in real time. After collection, it updates the collection status information, including pagination status, node status, and breakpoint recovery status. It then determines whether the conditions for continuing collection are met. If not, collection continues; otherwise, collection is complete.

[0059] In this embodiment, within a distributed system, business requests can invoke multiple microservice nodes. Based on the invocation order among these microservice nodes, link node information can be obtained. This link node information includes all microservice nodes in the node link and the invocation order between them, thereby constructing a unified state space across nodes. Current collection status information can also be obtained, including pagination status information and node status information. Pagination status information represents the pagination retrieval progress within a single microservice node currently being collected, while node status information represents the microservice node currently being collected, enabling joint maintenance of pagination and node states. Based on the link node information and the current collection status information, the microservice node to be collected and / or the pagination position is determined as the target collection position. Log data retrieval is performed at the target collection position to achieve accurate log collection. The collection status information is updated to facilitate subsequent collection, thus transforming the originally stateless log collection process, which required full re-collection, into a measurable and controllable convergence process, making the collection process observable and improving the accuracy and efficiency of log collection.

[0060] Figure 3 This is a flowchart illustrating a cross-node log collection method, such as... Figure 3 As shown, a cross-node log collection method according to one embodiment of this application may include: Step 310: Obtain link node information and current collection status information; wherein, the link node information includes all microservice nodes in the node link and the retrieval order between each microservice node, and the collection status information includes pagination status information and node status information. The pagination status information represents the pagination retrieval progress within a single microservice node currently being collected, and the node status information represents the microservice node currently being collected. Step 320: Determine the current collection location based on the current collection status information; where the current collection location represents the microservice node and / or pagination location currently collected; Step 330: Determine the target acquisition location based on the current acquisition location and link node information; Step 340: Retrieve log data at the target collection location and update the collection status information.

[0061] Steps 310 and 340 have been described in detail above and will not be repeated here. Steps 320 and 330 will be described in detail below.

[0062] In step 320, the collected status information refers to the global status content maintained during the log collection task's execution. It records the pagination progress of each microservice node and the flow status of each microservice node throughout the entire log collection process. The collected status information can be updated after each log data retrieval is completed, and it outputs the currently executed collection point, providing the original status-level basis for determining the current collection position.

[0063] The current collection position represents the currently collected microservice node and / or pagination position, which can be parsed from the collection status information, reflecting the current execution point of the collection process. For example, by reading the pagination fetch progress and microservice node processing status recorded within the collection status information, the current operation point of the collection process can be parsed and used as the current collection position. For instance, the first microservice node in the pending processing state in the node status information can be read and identified as the current collection position; the pagination position that has not been fetched can be read from the pagination status information and identified as the current collection position; or both node status information and pagination status information can be read simultaneously, and the corresponding microservice node and its corresponding incomplete pagination position can be jointly identified as the current collection position. The collection status information can be stored in a database, cache system, or distributed consistent storage, and can be read from a persistent state snapshot.

[0064] The current collection location serves as an intermediate output, and the target collection location is determined by combining it with the link node information. That is, the target collection location is determined based on the current collection location and the link node information. Taking into account the node topology constraints of the link and the completed collection progress, the target collection location corresponding to subsequent log retrieval is output, considering both the existing collection progress and the inherent node call order of the business link, to select the operation object for subsequent log query execution. For example, the next microservice node can be selected as the target collection location based on the current collection location and the microservice node call order recorded in the link node information.

[0065] In this embodiment, the current collection location is determined based on the collection status information, and then the target collection location is determined by combining the current collection location with the link node information. The real-time running progress of the collection task and the topology of the business link can be taken into account at the same time, so that the selection of the target collection location takes into account the existing collection progress and the calling logic of the link itself, thereby improving the rationality of the cross-node log collection process.

[0066] In one implementation, determining the current collection location based on the current collection status information includes: if the current collection status information includes pagination status information, then determining the current collection pagination based on the pagination status information in the current collection status information; wherein the current collection pagination represents the currently collected pagination location; if the current collection status information includes node status information, then determining the current collection node based on the node status information in the current collection status information; wherein the current collection node represents the currently collected microservice node.

[0067] Specifically, pagination status information is a component of the collection status information, representing the pagination fetching progress within a single microservice node. The pagination status information can record the location identifier corresponding to the pagination query, the cumulative data volume fetched, and whether the pagination query has been exhausted, among other things. Based on the pagination status information, the collection point at the pagination level can be parsed, providing a data source for determining the current pagination to be collected. The current pagination represents the currently collected pagination position, the point where the log fetching within a specific microservice node has stopped, but it does not specify which microservice node it is.

[0068] Node status information is also a component of the collection status information, representing the microservice nodes that have been collected so far. Node status information can maintain a set of discovered microservice nodes, microservice nodes awaiting processing, and microservice nodes that have been processed, recording the flow of microservice nodes throughout the entire collection process. Collection points at the node level can be read from the node status information, providing a basis for determining the current collection node. The current collection node represents the microservice node that has been collected so far.

[0069] The current collection location can include the current collection page and the current collection node. If the current collection status information includes page status information, the current collection page can be determined; if the current collection status information includes node status information, the current collection node can be determined.

[0070] If the current collection status information does not include pagination status information, then the current collection location does not contain the current collection pagination; if the current collection status information does not include node status information, then the current collection location does not contain the current collection node.

[0071] In this embodiment, by determining the current collection page when the collection status information includes pagination status information, and determining the current collection node when the collection status information includes node status information, the actual execution point of collection can be analyzed from the pagination dimension and the microservice node dimension, respectively, to adapt to the location identification requirements under different status retention conditions.

[0072] In one implementation, determining the target collection location based on the current collection location and link node information includes: if the current collection page represents the end point of the current collection node's page, then based on the current collection node and link node information, the next microservice node of the current collection node is determined as the target collection node, and the target collection page is determined as the starting point of the target collection node's page; wherein, the target collection node represents the microservice node to be collected, and the target collection page represents the page position to be collected; or, if the current collection page does not represent the end point of the current collection node's page, then the next page corresponding to the current collection page is determined as the target collection page, and the current collection node is determined as the target collection node.

[0073] Specifically, the current collection page represents the page position where the log retrieval within the current collection node is currently in progress. The page endpoint indicates that all available log data within the corresponding microservice node has been retrieved, and there is no further page data to be read. The current collection node is the microservice node currently being retrieved by the collection process. The link node information includes all microservice nodes in the node link and the call order between each microservice node, preserving the upstream and downstream topological relationships between microservice nodes.

[0074] The process involves determining whether the current page being fetched is the end point of the current fetch node. If so, it's confirmed that the current fetch node can be fully read. Based on the link node information, the next microservice node is identified as the target fetch node, and its page is designated as the starting point. In other words, once it's determined that the current microservice node's page fetching is complete, the microservice node logically following the current fetch node is selected as the target fetch node based on the microservice node call relationship recorded in the link node information. The initial read point of this newly selected microservice node is then used as the page to be fetched, i.e., the target page. The target fetch node is the microservice node accessed by subsequent log fetch operations, and the target page is the page to be read by subsequent log fetch operations.

[0075] If the current page being collected does not represent the end point of the current page collection node, then the next page corresponding to the current page collection can be directly determined as the target page collection, and the current page collection node can be determined as the target page collection node. In other words, if there is still unread log data within the current microservice node, the microservice node object is not switched; instead, the user is simply redirected within the current microservice node to the next page to be read, thus obtaining the target page collection node and the target page collection.

[0076] In this embodiment, by distinguishing whether the current collection page is the end point of the current collection node, cross-microservice node jump or intra-microservice node pagination progression is executed respectively. This can adapt to two different operating scenarios: single node collection completed and single node collection not yet completed. The subsequent microservice nodes and page reading positions are reasonably selected to ensure the orderly progression of the cross-node log collection process.

[0077] In this embodiment, within a distributed system, business requests can invoke multiple microservice nodes. Based on the invocation order among these microservice nodes, link node information can be obtained. This link node information includes all microservice nodes in the node link and the invocation order between them, thereby constructing a unified state space across nodes. Current collection status information can also be obtained, including pagination status information and node status information. Pagination status information represents the pagination retrieval progress within a single microservice node currently being collected, while node status information represents the microservice node currently being collected, enabling joint maintenance of pagination and node states. Based on the link node information and the current collection status information, the microservice node to be collected and / or the pagination position is determined as the target collection position. Log data retrieval is performed at the target collection position to achieve accurate log collection. The collection status information is updated to facilitate subsequent collection, thus transforming the originally stateless log collection process, which required full re-collection, into a measurable and controllable convergence process, making the collection process observable and improving the accuracy and efficiency of log collection.

[0078] Figure 4 This is a flowchart illustrating a cross-node log collection method, such as... Figure 4 As shown, a cross-node log collection method according to one embodiment of this application may include: Step 410: Obtain link node information and current collection status information; wherein, the link node information includes all microservice nodes in the node link and the retrieval order between each microservice node, and the collection status information includes pagination status information and node status information. The pagination status information represents the pagination retrieval progress within a single microservice node currently being collected, and the node status information represents the microservice node currently being collected. Step 420: Obtain the currently retrieved log data; Step 430: If there is no cross-node association information in the currently retrieved log data, determine the target collection location based on the link node information and the current collection status information; where cross-node association information represents the next microservice node that has a calling relationship with the current collection node, and the current collection node represents the microservice node that is currently being collected. Step 440: If there is cross-node association information in the currently retrieved log data, determine the next microservice node based on the cross-node association information; Step 450: If the next microservice node does not exist in the link node information, and the number of nodes in the link node information is less than the preset first quantity threshold, then add the next microservice node to the link node information and determine the next microservice node as the target collection node; wherein, the target collection node represents the microservice node to be collected. Step 460: Retrieve log data at the target collection location and update the collection status information.

[0079] Steps 410 and 460 have been described in detail above and will not be repeated here. Steps 420-450 will be described in detail below.

[0080] In step 420, the currently retrieved log data is the log record obtained after the retrieval operation was performed at the previous target collection location. The currently retrieved log data may include service identifier, instance identifier, Trace ID, Span ID, and upstream and downstream call related content. A Span is the basic unit representing an independent call segment in a distributed tracing system. A call segment can be the execution of a method within a service or a remote call between different microservice nodes. A complete business request generates a Trace, which is assembled from multiple Spans. Each Span corresponds to a segment of execution within the business process. The Span ID is a unique identifier for this Span itself.

[0081] In step 430, the cross-node association information represents the next microservice node that has a call relationship with the node corresponding to the currently pulled log data. The cross-node association information can come from the call topology content recorded inside the log data, which can reveal newly exposed business call objects during the collection process. It is used to discover microservice nodes that have not yet been included in the link node information and realize the dynamic expansion of the collection scope.

[0082] The system checks if cross-node correlation information exists in the currently retrieved log data. If not, it means there's no need to dynamically expand the link node information; the target collection location can be determined directly based on the original link node information and the current collection status information. In other words, the current collection location is determined based on the current collection status information, and the target collection location is determined based on the current collection location and the link node information.

[0083] In step 440, if cross-node association information exists in the currently retrieved log data, it indicates that dynamic expansion of microservice nodes may be necessary. Based on this cross-node association information, the next microservice node—that is, the next node to be collected—can be determined. This next microservice node may not exist in the link node information.

[0084] In step 450, this embodiment adapts to the bounded expansion of non-fixed log links. Microservice nodes can be dynamically expanded based on associations such as service identifiers, instance identifiers, Trace IDs, and Span IDs in the logs. The expansion process is subject to limits on diffusion depth, node quantity, and loop detection, achieving bounded control over the expansion. For example, a first quantity threshold can be preset. This preset threshold serves as a boundary condition constraining the scale of link node information, limiting the number of nodes and preventing the uncontrolled addition of microservice nodes during the collection process, which could lead to an out-of-control collection scope.

[0085] If the next microservice node exists in the link node information, there is no need to expand the link node information. The current collection location can be determined based on the current collection status information, and the target collection location can be determined based on the current collection location and the link node information. If the next microservice node does not exist in the link node information, and the number of nodes in the link node information is less than a preset first threshold, it means that the number of nodes in the link node information has not reached the upper limit. The next microservice node can be added to the link node information and designated as the target collection node. If the number of nodes in the link node information is equal to or greater than the preset first threshold, no more nodes can be added to the link node information, meaning the next microservice node cannot be accessed. A prompt message can be sent to the user indicating that the log acquisition process was interrupted.

[0086] Before adding the next microservice node to the link node information, loop detection can be performed synchronously to exclude microservice nodes with call loops before adding nodes and setting target collection nodes. Loop detection is used to identify circular call relationships between microservice nodes. A circular call relationship refers to a topology where multiple microservice nodes initiate calls to each other, forming a closed loop. If no detection and interception are performed, the link expansion process will repeatedly include the same microservice nodes in the pending set, causing the collection process to loop infinitely and the collection scope to become out of control. In this embodiment, the loop detection process is not specifically limited. Link node information can be stored using a graph structure or a queue structure. After the link node information is updated, it can be persistently saved, and the updated link node information can participate in the calculation of the target collection position in subsequent rounds.

[0087] In this embodiment, the retrieved log data is parsed for cross-node association information. Under the premise of identifying a new next microservice node and meeting the scale constraints, the link node information is updated and the target collection node is determined. The collection link range can be dynamically expanded based on the log content. At the same time, the boundary control of the expansion process is achieved by using the quantity threshold, taking into account both the integrity of log collection and system resource consumption.

[0088] In one implementation, the method further includes: determining the business function corresponding to the next microservice node; if the business function corresponding to the next microservice node does not exist in the business functions corresponding to each microservice node in the link node information, and the number of business functions corresponding to each microservice node in the link node information is less than a preset second quantity threshold, then the next microservice node is added to the link node information, and the next microservice node is determined as the target collection node.

[0089] Specifically, the next microservice node is a candidate microservice node obtained by parsing cross-node correlation information within log data. This next microservice node has the potential to be included in the link node information. Business functions refer to the business processing capabilities undertaken by the microservice node; different microservice nodes can correspond to the same or different business functions. Business functions can be represented by service names or business category tags. Business functions are used to categorize and distinguish microservice nodes from a business perspective. Using business functions as the unit for quantity constraints avoids repeatedly adding microservice nodes corresponding to a large number of instances of the same business function to the link node information, preventing excessive expansion of the collection scope due to multi-instance service deployments, and limiting the diffusion depth of link node information. The more types of business functions contained in the link node information, the deeper the diffusion depth.

[0090] The business function corresponding to the next microservice node is determined, as well as the business function corresponding to each microservice node in the link node information. A second quantity threshold is preset. The preset second quantity threshold is an extended constraint condition set for the business function dimension, which is used to constrain the diffusion depth of the link node information. This constraint can form a set of constraints for bounded diffusion control together with the node quantity limit and loop detection.

[0091] If the next microservice node exists in the link node information, then the business function of the next microservice node must also belong to the business function corresponding to each microservice node in the link node information. At this time, the current collection position is determined based on the current collection status information, and the target collection position is determined based on the current collection position and the link node information.

[0092] If the next microservice node does not exist in the link node information, it is further determined whether the business function corresponding to the next microservice node exists in the business functions corresponding to each microservice node in the link node information. If so, the number of nodes only needs to be judged based on the first quantity threshold; if not, the number of business functions corresponding to each microservice node in the link node information is determined. If the number is less than the preset second quantity threshold, the next microservice node can be added to the link node information and the next microservice node can be determined as the target collection node. If the number is equal to or greater than the preset second quantity threshold, the addition of the next microservice node is abandoned, the next microservice node is not set as the target collection node, and a prompt message can be issued to the user indicating that the log acquisition process is interrupted.

[0093] In this embodiment, by identifying the business function corresponding to the next microservice node and making a judgment based on whether the business function exists and the threshold number of business functions, the addition of microservice nodes can be controlled. This can constrain the expansion of the link from the business level, avoid the loss of control over the collection scope caused by a large number of instance nodes under the same business function, and control the overall scale of collection while covering different businesses as much as possible.

[0094] In this embodiment, within a distributed system, business requests can invoke multiple microservice nodes. Based on the invocation order among these microservice nodes, link node information can be obtained. This link node information includes all microservice nodes in the node link and the invocation order between them, thereby constructing a unified state space across nodes. Current collection status information can also be obtained, including pagination status information and node status information. Pagination status information represents the pagination retrieval progress within a single microservice node currently being collected, while node status information represents the microservice node currently being collected, enabling joint maintenance of pagination and node states. Based on the link node information and the current collection status information, the microservice node to be collected and / or the pagination position is determined as the target collection position. Log data retrieval is performed at the target collection position to achieve accurate log collection. The collection status information is updated to facilitate subsequent collection, thus transforming the originally stateless log collection process, which required full re-collection, into a measurable and controllable convergence process, making the collection process observable and improving the accuracy and efficiency of log collection.

[0095] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0096] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0097] Description of apparatus and devices in embodiments of this application Reference Figure 5 , Figure 5 This application provides a schematic diagram of the structure of a cross-node log collection device 500, which includes: The information acquisition module 501 is used to acquire link node information and current collection status information. The link node information includes all microservice nodes in the node link and the retrieval order between each microservice node. The collection status information includes at least one of pagination status information and node status information. The pagination status information represents the pagination retrieval progress within the currently collected single microservice node, and the node status information represents the currently collected microservice node. The location determination module 502 is used to determine the target collection location based on the link node information and the current collection status information; wherein, the target collection location represents the microservice node to be collected and / or the pagination location; The log retrieval module 503 is used to retrieve log data at the target collection location and update the collection status information.

[0098] In one example, the location determination module 502 includes: The current determination unit is used to determine the current collection position based on the current collection status information; wherein, the current collection position represents the microservice node and / or page position currently collected; The target determination unit is used to determine the target acquisition location based on the current acquisition location and link node information.

[0099] In one example, the currently determined unit is specifically used for: If the current collection status information includes pagination status information, then the current collection page is determined based on the pagination status information in the current collection status information; where the current collection page represents the currently collected page position; If the current collection status information includes node status information, then the current collection node is determined based on the node status information in the current collection status information; where the current collection node represents the microservice node currently collected.

[0100] In one example, the target determination unit is specifically used for: If the current page being collected represents the end point of the current page collection node, then based on the current page collection node and link node information, the next microservice node of the current page collection node is determined as the target page collection node, and the target page collection is determined as the starting point of the page collection at the target page collection node; wherein, the target page collection node represents the microservice node to be collected, and the target page collection represents the page position to be collected; or, If the current page being collected does not represent the end point of the current page collection node, then the next page corresponding to the current page collection will be determined as the target page collection node, and the current page collection node will be determined as the target page collection node.

[0101] In one example, it also includes: The parameter receiving module is used to receive the user-input data collection entry parameter information and determine the link node information corresponding to the data collection entry parameter information; wherein, the data collection entry parameter information represents the node link that needs to be called to collect log data; The initialization module is used to generate initial pagination status information and node status information.

[0102] In one example, the location determination module 502 is specifically used for: Retrieve the currently fetched log data; If there is no cross-node association information in the currently retrieved log data, the target collection location is determined based on the link node information and the current collection status information. Among them, the cross-node association information represents the next microservice node that has a call relationship with the current collection node, and the current collection node represents the microservice node that is currently being collected. If cross-node association information exists in the currently retrieved log data, then the next microservice node is determined based on the cross-node association information. If the next microservice node does not exist in the link node information, and the number of nodes in the link node information is less than the preset first quantity threshold, then the next microservice node is added to the link node information, and the next microservice node is determined as the target collection node; where the target collection node represents the microservice node to be collected.

[0103] In one example, it also includes: The function determination module is used to determine the business function corresponding to the next microservice node; The node addition module is used to add the next microservice node to the link node information and determine the next microservice node as the target collection node if the business function corresponding to the next microservice node does not exist in the business functions corresponding to each microservice node in the link node information, and the number of business functions corresponding to each microservice node in the link node information is less than a preset second quantity threshold.

[0104] In one example, it also includes: The indicator determination module is used to determine the indicator information of the preset collection indicators according to the preset time period. The preset collection indicators include at least one of page exhaustion rate, node coverage rate, and anomaly coverage rate. The page exhaustion rate represents the ratio of the number of microservice nodes that have exhausted pagination to the number of microservice nodes that have started pagination fetching. The node coverage rate represents the ratio of the number of microservice nodes that have started collection to the number of microservice nodes in the link node information. The anomaly coverage rate represents the ratio of the number of collected data identified as abnormal to the preset number of anomalies. The data collection termination module is used to terminate the log data retrieval operation if the data information of a preset data collection indicator meets the preset data collection termination condition.

[0105] In one example, it also includes: The restart module is used to determine the current collection position based on the current collection status information after restarting the collection operation if the log data retrieval operation is interrupted. The current collection position represents the microservice node and / or pagination position currently being collected. The current collection position is then determined as the target collection position.

[0106] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0108] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 of this embodiment includes: at least one processor 601 ( Figure 6 (Only one is shown in the diagram), memory 602, and computer program 603 stored in the memory 602 and executable on the at least one processor 601, wherein the processor 601 executes the computer program 603 to implement the steps in any of the above embodiments of the cross-node log collection method.

[0109] Electronic device 600 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This electronic device may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 600 and does not constitute a limitation on electronic device 600. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0110] The processor 601 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0111] In some embodiments, the memory 602 may be an internal storage unit of the electronic device 600, such as a hard disk or memory of the electronic device 600. In other embodiments, the memory 602 may be an external storage device of the electronic device 600, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 600. Furthermore, the memory 602 may include both internal and external storage units of the electronic device 600. The memory 602 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 602 can also be used to temporarily store data that has been output or will be output.

[0112] This application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0113] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0114] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the various method embodiments.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0116] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for cross-node log collection, characterized in that, include: Obtain link node information and current collection status information; wherein, the link node information includes all microservice nodes in the node link and the call order between each microservice node, the collection status information includes at least one of pagination status information and node status information, the pagination status information represents the pagination pull progress within the currently collected single microservice node, and the node status information represents the currently collected microservice node. Based on the link node information and the current collection status information, the target collection location is determined; wherein, the target collection location represents the microservice node and / or pagination location to be collected; Perform log data retrieval at the target collection location and update the collection status information.

2. The method according to claim 1, characterized in that, Determining the target acquisition location based on the link node information and the current acquisition status information includes: Based on the current collection status information, determine the current collection location; wherein, the current collection location represents the currently collected microservice node and / or pagination location; The target acquisition location is determined based on the current acquisition location and the link node information.

3. The method according to claim 2, characterized in that, Determining the current acquisition location based on the current acquisition status information includes: If the current collection status information includes pagination status information, then the current collection pagination is determined based on the pagination status information in the current collection status information; wherein, the current collection pagination represents the currently collected pagination position; If the current collection status information includes node status information, then the current collection node is determined based on the node status information in the current collection status information; wherein, the current collection node represents the microservice node currently collected.

4. The method according to claim 3, characterized in that, Determining the target collection location based on the current collection location and the link node information includes: If the current collection page represents the end point of the current collection node's page, then based on the current collection node and the link node information, the next microservice node of the current collection node is determined as the target collection node, and the target collection page is determined as the starting point of the page of the target collection node; wherein, the target collection node represents the microservice node to be collected, and the target collection page represents the page position to be collected; or, If the current page being collected does not represent the end point of the current page collection node, then the next page corresponding to the current page collection is determined as the target page collection, and the current page collection node is determined as the target page collection node.

5. The method according to claim 1, characterized in that, The method further includes: Receive user-inputted data collection entry parameter information and determine the link node information corresponding to the data collection entry parameter information; wherein, the data collection entry parameter information represents the node link that needs to be called to collect log data; Generate initial pagination status information and node status information.

6. The method according to claim 1, characterized in that, Determining the target acquisition location based on the link node information and the current acquisition status information includes: Retrieve the currently fetched log data; If there is no cross-node association information in the currently retrieved log data, the target collection location is determined based on the link node information and the current collection status information; wherein, the cross-node association information represents the next microservice node that has a calling relationship with the current collection node, and the current collection node represents the microservice node that is currently being collected; If cross-node association information exists in the currently retrieved log data, then the next microservice node is determined based on the cross-node association information. If the next microservice node does not exist in the link node information, and the number of nodes in the link node information is less than a preset first quantity threshold, then the next microservice node is added to the link node information, and the next microservice node is determined as the target collection node; wherein, the target collection node represents the microservice node to be collected.

7. The method according to claim 6, characterized in that, The method further includes: Determine the business function corresponding to the next microservice node; If the business function corresponding to the next microservice node does not exist in the business functions corresponding to each microservice node in the link node information, and the number of business functions corresponding to each microservice node in the link node information is less than a preset second quantity threshold, then the next microservice node is added to the link node information, and the next microservice node is determined as the target collection node.

8. The method according to claim 1, characterized in that, The method further includes: Based on a preset time period, the indicator information of preset collection indicators is determined; wherein, the preset collection indicators include at least one of page exhaustion rate, node coverage rate, and anomaly coverage rate, wherein the page exhaustion rate represents the ratio of the number of microservice nodes whose pagination has been exhausted to the number of microservice nodes that have started pagination fetching, the node coverage rate represents the ratio of the number of microservice nodes that have started collection to the number of microservice nodes in the link node information, and the anomaly coverage rate represents the ratio of the number of collected data identified as abnormal to the preset number of anomalies. If the information of a preset collection metric meets the preset collection termination condition, the log data retrieval operation will be terminated.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: If the log data retrieval operation is interrupted, the current retrieval position will be determined based on the current retrieval status information after the retrieval operation is restarted; wherein, the current retrieval position represents the microservice node and / or pagination position currently being retrieval; The current acquisition location is determined as the target acquisition location.

10. A cross-node log collection device, characterized in that, include: The information acquisition module is used to acquire link node information and current collection status information; wherein, the link node information includes all microservice nodes in the node link and the retrieval order between each microservice node, and the collection status information includes at least one of pagination status information and node status information, wherein the pagination status information represents the pagination retrieval progress within the currently collected single microservice node, and the node status information represents the currently collected microservice node. The location determination module is used to determine the target collection location based on the link node information and the current collection status information; wherein, the target collection location represents the microservice node to be collected and / or the pagination location; The log retrieval module is used to retrieve log data at the target collection location and update the collection status information.

Images