A method, system and related devices for cross-platform log correlation derivation

Abstract

The application discloses a method, system and related device for cross-platform log association derivation, the method comprising: obtaining a log query request in natural language form and determining a log query parameter; obtaining a first log from a task scheduling platform according to the log query parameter; parsing the first log to extract an association identifier, and obtaining a second log from a resource management platform based on the association identifier; receiving a response result returned by the resource management platform, and determining the acquisition state of the second log according to the response result, deriving the second log when the second log is successfully acquired, and deriving the first log when the second log fails to be acquired. The application effectively solves the technical problems of cumbersome query operation, cross-platform association difficulty and no alternative solution after the second log expires caused by the independent deployment of the task scheduling platform and the resource management platform, the domain storage of logs and the different retention strategies.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, system and related apparatus for cross-platform log association export. Background Technology

[0002] Workflow task scheduling platforms and resource management platforms are two types of core infrastructure. The task scheduling platform is responsible for task orchestration, triggering, and dependency management; the resource management platform is responsible for the allocation of computing resources and the actual execution of tasks. The complete execution process of a data processing task naturally has its runtime logs distributed across these two independent systems: the task scheduling platform records scheduling logs such as task submission and status changes, while the resource management platform records container logs such as standard output and standard error during the actual execution of the task.

[0003] In existing technologies, when a log query requirement occurs, operations and maintenance personnel typically need to log into two systems separately. First, they need to search for the target job in the task scheduling platform by project, workflow, and task, and then view the scheduling logs. Next, they need to find the task identifier assigned by the resource management platform from the scheduling logs, and then switch to the resource management platform to query the corresponding container logs. The entire process involves logging into and switching between multiple systems and multiple page jumps, resulting in numerous steps and low efficiency.

[0004] Some existing technologies attempt to solve the problem of cross-platform log viewing through log aggregation systems. These solutions typically use log collection agents to aggregate logs scattered across different systems to a unified storage and analysis platform. However, these solutions require additional deployment and maintenance costs, and still require manual querying and analysis of the aggregated logs, and cannot automatically export them across platforms. Summary of the Invention

[0005] This application provides a method, system, and related apparatus for cross-platform log association and export, which solves the problems of cumbersome query operations and difficulties in cross-platform association caused by the independent deployment of two platforms, the separate storage of logs, and different retention strategies in the prior art.

[0006] The first aspect of this application provides a method for cross-platform log association and export, the method comprising: Obtain a log query request in natural language form, and determine the log query parameters based on the log query request; Based on the log query parameters, a first log retrieval request is initiated to the task scheduling platform, and the first log returned by the task scheduling platform is received. The first log is parsed to extract the association identifier, and a second log acquisition request is sent to the resource management platform based on the association identifier, and the response result returned by the resource management platform is received. The acquisition status of the second log is determined based on the response result. The second log is the log in the resource management platform that corresponds to the associated identifier. If the second log is successfully retrieved, the second log will be identified as the log to be exported. If the second log fails to be retrieved, the first log is identified as the log to be exported; the log to be exported is then output.

[0007] Optionally, before initiating a first log retrieval request to the task scheduling platform based on the log query parameters and receiving the first log returned by the task scheduling platform, the process further includes: The log query parameters are used to query whether there are log cache records in the local cache. The log cache records include cache execution status and log access links. The cache execution status is the job execution status recorded when the logs were exported last time. The log access links are the access addresses of the log files generated in this export. When the log cache record exists, query the task scheduling platform for the real-time execution status of the job corresponding to the log query parameter, and determine the validity of the log cache record based on the comparison result between the cached execution status and the real-time execution status, as well as the cache duration of the log cache record; when the log cache record is determined to be valid, identify the log pointed to by the log access link in the log cache record as the log to be exported and output it. When the log cache record does not exist, the step of initiating the first log retrieval request to the task scheduling platform based on the log query parameters is executed.

[0008] Optionally, determining the validity of the log cache record based on the comparison result between the cache execution status and the real-time execution status, and the caching duration of the log cache record, includes: When the real-time execution status is in the final state, if the cache execution status is in the final state, then the log cache record is determined to be valid; if the cache execution status is in the running state, then the log cache record is determined to be invalid, and the step of initiating the first log acquisition request to the task scheduling platform according to the log query parameters is executed; wherein, the final state includes execution success, execution failure, and execution termination; When the real-time execution status is running, the cache duration of the log cache record is calculated, where the cache duration is the time difference between the current time and the creation time of the log cache record. If the cache duration is less than a preset threshold, the log cache record is determined to be valid. If the cache duration is greater than or equal to the preset threshold, the log cache record is determined to be invalid, and the step of initiating the first log retrieval request to the task scheduling platform according to the log query parameters is executed.

[0009] Optionally, the step of initiating a first log retrieval request to the task scheduling platform based on the log query parameters, and receiving the first log returned by the task scheduling platform, includes: A task scheduling platform session is created based on the log query parameters. An authentication request is then sent to the task scheduling platform through the task scheduling platform session to obtain the session identifier. Initiate an execution information query request to the task scheduling platform based on the session identifier to obtain the execution identifier; Based on the execution identifier, a first log retrieval request is initiated to the task scheduling platform, and the first log returned by the task scheduling platform is received.

[0010] Optionally, the step of parsing the first log to extract the association identifier, initiating a second log retrieval request to the resource management platform based on the association identifier, and receiving the response result returned by the resource management platform includes: The first log is parsed to extract a string that conforms to the task identifier format in the resource management platform, and the last matching result is taken as the association identifier. A second log retrieval request is generated based on the association identifier, and the second log retrieval request is sent to the resource management platform. Receive the response result returned by the resource management platform.

[0011] Optionally, determining the acquisition status of the second log based on the response result, wherein the second log is the log in the resource management platform corresponding to the associated identifier, includes: Extract the status code from the response result, compare the status code with a pre-configured status code rule base, and determine the acquisition status of the second log. The status code rule base includes at least a first type of status code and a second type of status code. When the status code belongs to the first type of status code, it is determined that the second log was successfully obtained; When the status code belongs to the second type of status code, it is determined that the second log acquisition failed.

[0012] Optionally, before parsing the first log to extract the association identifier, the method further includes: The first log is stored locally; The method further includes: If the second log acquisition fails, the first log stored locally is read as the log to be exported.

[0013] A second aspect of this application provides a cross-platform log association and export system, the system comprising: The parsing unit is used to obtain log query requests in natural language form and determine log query parameters based on the log query requests. The first acquisition unit is used to initiate a first log acquisition request to the task scheduling platform according to the log query parameters, and to receive the first log returned by the task scheduling platform. The second acquisition unit is used to parse the first log to extract the association identifier, initiate a second log acquisition request to the resource management platform based on the association identifier, and receive the response result returned by the resource management platform. A status identification unit is used to determine the acquisition status of the second log based on the response result, wherein the second log is the log in the resource management platform corresponding to the associated identifier; The log export unit is used to determine the second log as the log to be exported if the second log is successfully acquired; and to determine the first log as the log to be exported if the second log is not acquired; and to output the log to be exported.

[0014] A third aspect of this application provides an apparatus for cross-platform log association export, the apparatus comprising: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program that the processor invokes to execute the first aspect and any optional method of cross-platform log association export in the first aspect.

[0015] The fourth aspect of this application provides a computer-readable storage medium on which a program is stored, which, when executed on a computer, performs a method for cross-platform log association export as described in the first aspect and any optional method in the first aspect.

[0016] As can be seen from the above technical solutions, this application has the following advantages: By obtaining log query requests in natural language format to determine log query parameters, the system automatically retrieves the first log and extracts its associated identifier, establishing a log association index between the task scheduling platform and the resource management platform without manual intervention. Then, based on this index, it proactively initiates a second log retrieval request to the resource management platform and identifies the retrieval status of the second log based on the returned response. If the second log is successfully retrieved, it is designated as the log to be exported; if the second log retrieval fails, the first log is designated as the log to be exported and output. This forms a complete automated chain from user request to valid log output, effectively solving the technical problems in existing technologies caused by the independent deployment of the task scheduling platform and the resource management platform, the separate storage of logs in different domains, and different retention policies, resulting in cumbersome query operations, difficulties in cross-platform association, and the lack of alternative solutions after the second log expires. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an embodiment of cross-platform log association export provided in this application; Figure 2 A flowchart illustrating an embodiment of log query parameter query caching in cross-platform log association export provided in this application; Figure 3 A flowchart illustrating an embodiment of obtaining the first log in cross-platform log association export provided in this application; Figure 4 A flowchart illustrating an embodiment of obtaining a second log in cross-platform log association export provided in this application; Figure 5 A schematic diagram of the structure of an embodiment of the cross-platform log association export system provided in this application; Figure 6 This is a schematic diagram of one embodiment of the cross-platform log association export apparatus provided in this application. Detailed Implementation

[0019] It should be noted that the method, system, and related apparatus for cross-platform log association export provided in this application can be applied to terminals, systems, and servers. The terminal can be a smartphone, computer, tablet, smart TV, smartwatch, portable computer terminal, or a fixed terminal such as a desktop computer. Users can send natural language query requests through instant messaging tools running on the aforementioned terminal devices. The server can be a physical server, a cloud server, or a distributed cluster server, used to execute the processing logic of log acquisition, task identifier extraction, cross-platform log query, and status identification described in this application. The system can be an independent log management system, integrated into a task scheduling platform or resource management platform, or embedded in a third-party system as a software development kit. For ease of explanation, a log management system is used as an example in this application.

[0020] Please see Figure 1 , Figure 1 A flowchart illustrating an embodiment of cross-platform log association export provided in this application is shown, the embodiment including: S101. Obtain the log query request in natural language form, and determine the log query parameters based on the log query request; Users initiate log query requests through intelligent chatbots within instant messaging tools. These tools can be third-party instant messaging platforms that support intelligent chatbot functionality, such as WeChat Work, DingTalk, and Lark. The intelligent chatbot maintains real-time communication with the instant messaging server via a persistent connection. Specifically, upon system startup, the intelligent chatbot uses a pre-configured chatbot identifier and authentication key to initiate a connection request to the instant messaging server. Once the connection is established, it remains persistent. When a user sends a natural language message to the intelligent chatbot within the instant messaging tool, the instant messaging server pushes the natural language text to the intelligent chatbot in real-time through the persistent connection.

[0021] After receiving natural language text, the intelligent robot forwards it to the AI ​​engine module. The AI ​​engine module parses the natural language text based on the function call mechanism of the pre-trained language model. The system predefines descriptions of one or more log export tools. These descriptions include at least the tool name, function description, and acceptable input parameter definitions, which include parameter name, parameter type, whether the parameter is required, and parameter format validation rules. When the natural language text is input into the pre-trained language model, the model identifies the user's log query intent based on the tool descriptions and determines the target tool to be invoked. Log query intents include at least two types: task scheduling platform log export intent and resource management platform log export intent. Specifically, when the natural language text contains keywords such as project name, workflow name, or task name, the pre-trained language model determines the user's log query intent as a task scheduling platform log export intent; when the natural language text contains strings in the resource management platform task identifier format, the model determines the user's log query intent as a resource management platform log export intent.

[0022] After recognizing the user's log query intent, the pre-trained language model parses the parameters required to call the corresponding tool from natural language text based on predefined tool description information, and generates a structured call request that conforms to function signature requirements. The structured call request is organized using a structured data format, containing both the tool name and the call parameters. The system parses the structured call request and extracts the structured log query parameters. In this way, users do not need to remember any command formats or parameter names; they only need to describe their log query needs in natural language to complete the parameter input. For example, when a user enters "Export the data loading task logs from the daily extraction workflow under the data pipeline project," the structured call request generated by the pre-trained language model includes three parameters: project name, workflow name, and task name; when a user enters "View the execution logs of application ID 1234567890123," the structured call request generated by the pre-trained language model includes one parameter: application ID.

[0023] S102. Initiate a first log retrieval request to the task scheduling platform according to the log query parameters, and receive the first log returned by the task scheduling platform; A task scheduling platform session is created based on the log query parameters. An authentication request is initiated through this session to obtain a session identifier, which is then carried in all subsequent requests to maintain authentication. After successfully obtaining the session identifier, the system uses the project name, workflow name, and task name from the log query parameters as query conditions to request the execution information query interface of the task scheduling platform, retrieving the execution identifier of the most recent run of the corresponding task. After obtaining the execution identifier, the system uses this identifier as a parameter to initiate a log download request to the log download interface of the task scheduling platform, retrieving the content of the first log entry in a paginated manner. Specifically, each request retrieves a fixed number of characters of log content, and subsequent content is retrieved sequentially with increasing offsets until the length of the returned content is less than the requested number of characters, at which point the download is considered complete. For example, if each request is set to 50,000 characters, the request starts from offset zero, retrieving the first 50,000 characters; then the offset is increased by 50,000, requesting the next 50,000 characters, and so on, until the actual number of characters returned is less than 50,000, at which point the log is considered completely retrieved.

[0024] S103. Parse the first log to extract the association identifier, and based on the association identifier, send a second log acquisition request to the resource management platform and receive the response result returned by the resource management platform. Before parsing the first log to extract the association identifier, the obtained first log can be normalized first, and then the normalized first log can be saved to local storage. Next, the first log is parsed to extract the association identifier assigned by the resource management platform. Specifically, the system uses regular expressions to match strings in the first log that conform to the task identifier format of the resource management platform. When the resource management platform receives tasks submitted by the task scheduling platform, it assigns a globally unique task identifier to each task and records this identifier in the first log. Strings in this format are matched using regular expressions, and the last matched string is taken as the association identifier. The normalization process includes three steps: first, the HTML entities in the log are encoded and decoded into raw characters to avoid garbled text; second, different line breaks are uniformly replaced with the same line break character to avoid consecutive blank lines due to inconsistent line break formats; finally, two or more consecutive blank lines are compressed into a single blank line using regular expressions. After the above normalization process, the readability of the log is improved, which also helps to improve the efficiency of subsequent automated parsing.

[0025] If no string matching the format is found in the first log, it means that the task has not been submitted to the resource management platform or the task type does not involve scheduling by the resource management platform. In this case, the first log that was previously saved to local storage can be identified as the log to be exported. When outputting, the source of the log should be marked as the task scheduling platform and the reason why it is not associated with the second log of the resource management platform, so as to ensure that a log query request always returns log results.

[0026] Based on the association identifier, a second log retrieval request is initiated to the resource management platform to attempt to obtain the corresponding second log. Specifically, a log query request is constructed based on the association identifier and sent to the log query interface of the resource management platform to request the standard output log and standard error log of all containers corresponding to the association identifier. Since the resource management platform has a retention period for the second log, it may have been cleaned up after expiration. Therefore, this request is only an attempt to obtain the log, and its success depends on whether the second log corresponding to the association identifier still exists on the resource management platform side.

[0027] S104. Determine the acquisition status of the second log based on the response result. The second log is the log in the resource management platform that corresponds to the associated identifier. After receiving a log query request initiated by the system based on an association identifier, the resource management platform processes the request and generates a response result to return to the system. The response result contains at least two parts: a status code field and a log content field. The status code field carries the standardized status code of the resource management platform for the processing result of this request, and the log content field carries the corresponding second log data if the request is successful. It should be noted that in scenarios where the second log retrieval fails, the log content field may be empty or may not contain valid log data.

[0028] After receiving the response from the resource management platform, the system parses the status code from a specified field in the response. This status code is a standardized code generated by the resource management platform based on the query results after processing the log query request, and it corresponds one-to-one with the processing result of the request. For example, when the resource management platform successfully queries the corresponding second log, the returned status code field value is a success indicator; when the resource management platform fails to query the corresponding second log, the returned status code field value is a corresponding exception indicator, such as the task not existing or the log being cleared.

[0029] The parsed status code is compared with a pre-configured status code rule base to determine the acquisition status of the second log. The status code rule base pre-configures the correspondence between status codes and acquisition statuses, with different status code values ​​mapping to different acquisition status categories. Specifically, the status code rule base includes at least two categories: a first category of status codes and a second category of status codes. When the status code belongs to the first category, it indicates that the second log was successfully acquired; when the status code belongs to the second category, it indicates that the second log acquisition failed.

[0030] S105. If the second log is successfully retrieved, the second log is designated as the log to be exported; if the second log is not retrieved, the first log is designated as the log to be exported; output the log to be exported.

[0031] When the status code indicates successful retrieval of the second log, the second log is extracted from the response returned by the resource management platform and used as the log to be exported. Specifically, in response to the second log retrieval request, the resource management platform returns the retrieved log data to the system if it can find the log data corresponding to the associated identifier. After receiving the response, if the system determines that the status code belongs to the first type of status code, it further retrieves the corresponding second log from the response and processes the retrieved second log to form the log data for output.

[0032] If the second log is successfully retrieved, the processed second log will be designated as the log to be exported. Compared to the first log, the second log reflects specific operational information during the task execution process on the resource management platform side, thus providing users with more detailed operational data and facilitating the analysis and localization of task execution anomalies.

[0033] When the status code indicates that the second log retrieval failed, the system reads the first log that was saved to local storage before initiating the second log retrieval request and identifies the read first log as the log to be exported. Before executing the second log retrieval request, the system has already saved the first log to local storage. Therefore, upon receiving the response from the resource management platform and confirming the second log retrieval failure, there is no need to request the first log from the task scheduling platform again; instead, the system directly reads the saved first log from local storage and identifies the read result as the log to be exported. The failure to retrieve the second log may be caused by various reasons, such as the task corresponding to the associated identifier not existing, the associated identifier being invalid, the second log in the resource management platform exceeding its preset retention period and being cleaned up, the resource management platform being currently inaccessible, or an exception occurring in the resource management platform while processing the second log retrieval request. For the above different failure scenarios, although the system cannot retrieve the second log from the resource management platform, it can still complete the log output if the first log has already been saved.

[0034] Since the first log records the task execution process on the task scheduling platform side, it can contain information such as task submission records, task start execution records, task status change records, task execution results, and identifiers associated with tasks on the resource management platform. Although the first log lacks some detailed runtime information from the resource management platform side compared to the second log, it can still provide users with execution information at the task scheduling level for basic fault analysis and execution status judgment. Therefore, when the second log fails to be retrieved, using the first log as the log to be exported does not involve re-executing the cross-platform log retrieval process, but rather directly using the already saved first log for rollback output, avoiding the failure of the entire log export process or the return of empty results to the user due to the temporary unavailability of the second log.

[0035] After identifying the logs to be exported, the system performs output processing and outputs the logs to the user's designated log display location. Specifically, the system asynchronously uploads the corresponding log file to a remote file storage server via a file transfer protocol and generates an access link for that log file to avoid impacting service throughput from synchronous file transfers. The log display location can be a message window in an instant messaging tool, a log query page, or other preset log output locations.

[0036] When the log to be exported is a second log, the system outputs the second log from the resource management platform; when the log to be exported is a first log, the system outputs the first log stored locally. Furthermore, the system also labels the log source when outputting the log to be exported: when the log to be exported is a second log, the source is labeled as the resource management platform; when the log to be exported is a first log, the source is labeled as the task scheduling platform. A catch-all explanation is added to the header of the first log explaining why the second log is unavailable, allowing users to clearly understand which level of log they are currently viewing. Therefore, regardless of whether the second log in the resource management platform can be obtained normally, the system can select the appropriate log data to complete the output based on the log acquisition status.

[0037] In this embodiment, log query parameters are determined by obtaining log query requests in natural language form. Driven by this demand, the system automatically retrieves the first log and extracts its associated identifier, establishing a log association index between the task scheduling platform and the resource management platform without manual intervention. Then, based on this index, the system proactively initiates a second log retrieval request to the resource management platform and identifies the retrieval status of the second log based on the returned response. If the second log is successfully retrieved, it is designated as the log to be exported. If the second log fails to be retrieved, the first log is designated as the log to be exported and output. This forms a complete automated link from user request to valid log output, effectively solving the technical problems in the prior art caused by the independent deployment of the task scheduling platform and the resource management platform, the separate storage of logs, and different retention strategies, resulting in cumbersome query operations, difficulties in cross-platform association, and the lack of alternative solutions after the second log expires.

[0038] Before initiating a log retrieval request to the task scheduling platform based on the log query parameters, a caching query step is also included; please refer to [link to relevant documentation]. Figure 2 , Figure 2 This application provides a flowchart illustrating an embodiment of log query parameter caching in cross-platform log association export, which includes: S201. Query whether there are log cache records in the local cache according to the log query parameters. The log cache records include the cache execution status and log access links. The system checks if a corresponding log cache record exists in the cache based on the parsed log query parameters. Specifically, it uses at least a portion of the information from the log query parameters as the cache key and performs a key-value lookup operation in the cache database. The cache key must include at least the project name, workflow name, and task name, and may further include an execution identifier. The cache database can be implemented using a key-value database, such as a remote dictionary service, or a relational database.

[0039] The log cache records stored in the cache database include cache execution status and log access links. The cache execution status is the job execution status recorded when the logs were exported last time, and the log access link is the access address of the log file generated in this export. By checking whether there is a corresponding log cache record in the cache before executing the log export process, it can be determined whether there is reusable cache data in the first log corresponding to the current log query request, thereby avoiding repeated access to the task scheduling platform when there is already a reusable log.

[0040] If there is no corresponding log cache record in the cache, it means that the log corresponding to the current log query request has not previously formed a reusable cache result. Then, the step of initiating the first log retrieval request to the task scheduling platform according to the log query parameters is executed to obtain the first log. If there is a corresponding log cache record in the cache, it is further determined whether the log cache record is still valid.

[0041] S202. When a log cache record exists, query the task scheduling platform for the real-time execution status of the job corresponding to the log query parameters, and determine the validity of the log cache record based on the comparison result between the cached execution status and the real-time execution status, as well as the cache duration of the log cache record; when the log cache record is determined to be valid, identify the log pointed to by the log access link in the log cache record as the log to be exported and output it. When a log cache record is found, it is not directly reused. Instead, an execution status query request is first sent to the task scheduling platform to obtain the real-time execution status of the job. The task scheduling platform responds to this request and returns the real-time execution status, which includes "Running" and "Final," with the latter including "Completed," "Failed," and "Terminated." The cached execution status stored in the log cache record is the historical execution status, that is, the job execution status recorded when the logs were last exported, while the real-time execution status reflects the actual running status at the latest moment. Since the job's running status changes over time, it is necessary to retrieve the real-time execution status again each time the cache is hit and compare the cached execution status with the real-time execution status to determine whether the log in the cache is still a complete log, thereby ensuring the accuracy of the cache validity judgment.

[0042] When the real-time execution status is "final," it indicates that the job has finished running and the log content no longer changes. At this point, the cache execution status is further compared with the log cache record. If the cache execution status is also "final," it means the cached log is the complete log for this run, and the cache is deemed valid, eliminating the need to re-execute the log export process. If the cache execution status is "running," it means the job was not yet finished when the cache record was generated, and the cached log content is incomplete. The cache is deemed invalid, and the process jumps to step S102 to initiate a log retrieval request to the task scheduling platform to obtain the complete first log. For example, if a job was still running when its logs were first exported at 10:00 AM yesterday, and the cache execution status was recorded as "running," and today the job has finished running, the real-time execution status found is "completed." The cache execution status is inconsistent with the real-time execution status, indicating that the cached log is an incomplete log from the middle of the job's execution. The cache is deemed invalid and re-exported. For example, if the logs of a certain job were fully exported yesterday and the cached execution status is "completed", and the real-time execution status is still "completed" when queried again today, since the log content has been fixed, yesterday's cached result can be reused directly without making a duplicate request to the task scheduling platform.

[0043] When the real-time execution status is "running," it indicates that the current job execution has not yet ended. The cache execution status is consistent with the real-time execution status, and there is no state transition. At this time, since the log content is still being continuously generated and updated, the log content in the cache record may be incomplete. Further, the cache duration is used to determine whether the cache has expired. Specifically, the cache duration of the cache record is calculated, which is the time difference between the current time and the creation time of the cache record, and this cache duration is compared with a preset short-period threshold. If the cache duration is less than the preset short-period threshold, it means that the cache record is still within the valid window period, and the cache is determined to be valid; if the cache duration is greater than or equal to the preset short-period threshold, it means that the cache record has expired, and the cache is determined to be invalid. The process then jumps to step S102 to execute the process of initiating a log retrieval request to the task scheduling platform to obtain the latest first log containing the latest log content. The aforementioned preset short-cycle threshold is configurable. For example, if a task is running and receives three log query requests at 10:00 AM, 10:05 AM, and 10:15 AM respectively: The first query finds the cache missing, so the complete log export process is executed and the result is written to the cache; the second query finds a cache hit, but the cache duration is only five minutes, less than the ten-minute threshold, so the cached result is reused; the third query finds the cache duration has reached fifteen minutes, exceeding the ten-minute threshold, so the cache is deemed invalid, and the log export process is re-executed to obtain the latest log content. Through this threshold mechanism, the frequency of repeated log exports for running jobs is effectively controlled without sacrificing real-time performance.

[0044] When a log cache record is determined to be valid, the log access link in that record is directly read, the log it points to is identified as the log to be exported, and it is output to the user's corresponding log display location. Since the log file pointed to by the log access link has already been generated and stored during the previous export, there is no need to access the task scheduling platform again, nor is it necessary to re-execute the download of the first log and the acquisition of the second log, thus avoiding duplicate requests to the task scheduling platform.

[0045] S203. When the log cache record does not exist, execute the step of initiating the first log retrieval request to the task scheduling platform according to the log query parameters.

[0046] When no corresponding log cache record is found in the cache, or when the cache validity check indicates that the cache is invalid, the step of initiating a log retrieval request to the task scheduling platform based on the log query parameters is executed. Specifically, in the case of a non-existent or expired cache, existing cache results cannot be reused, therefore a complete log export process needs to be performed. A log retrieval request is initiated to the task scheduling platform based on the log query parameters to retrieve the first log entry.

[0047] In this embodiment, a cache query step is introduced before executing the first log acquisition process. When the cache is hit and valid, the historical export results are directly reused, avoiding repeated requests to the task scheduling platform, reducing system load and improving response speed. At the same time, by querying the real-time execution status from the task scheduling platform each time the cache is hit and comparing the real-time execution status with the cache execution status in the log cache record, log cache records with both real-time and cache execution status in the final state are permanently reused, avoiding repeated export of completed jobs. Log cache records with cache execution status in running but real-time execution status has transitioned to the final state are immediately determined to be invalid and re-exported, avoiding returning incomplete historical logs to the user. For log cache records with real-time execution status still in running, the cache refresh frequency is controlled by a configurable short-cycle threshold, balancing data real-time performance and system performance. The threshold mechanism also prevents frequent user refreshes from causing a large number of concurrent requests to overwhelm the backend service, thus playing a role in service rate limiting protection.

[0048] Based on the log query parameters, a log retrieval request is sent to the task scheduling platform to obtain the first log entry. Please refer to [link / reference needed]. Figure 3 , Figure 3 A flowchart illustrating an embodiment of obtaining the first log in cross-platform log association export provided in this application is shown. This embodiment includes: S301. Create a task scheduling platform session based on log query parameters, and send an authentication request to the task scheduling platform through the task scheduling platform session to obtain the session identifier; Log query parameters must include at least the project name, workflow name, and task name. The task scheduling platform session encapsulates the connection configuration and status information required for communication with the task scheduling platform, including the platform's interface address, connection timeout, read timeout, and authentication credentials. By creating a task scheduling platform session, the system obtains a unified entry point for subsequent interactions with the platform; all subsequent requests are sent through this session object, eliminating the need for repeated configuration of connection parameters. The session object creation process does not involve actual network communication; it only initializes the local object. The created session initiates an authentication request to the task scheduling platform to obtain a session identifier. Specifically, the session object constructs an authentication request message using pre-configured authentication credentials and sends it to the platform's authentication interface. The credentials include the login username and password. Upon receiving the authentication request, the task scheduling platform verifies the credentials. If successful, it generates a time-sensitive session identifier and returns it to the session object. This session identifier is typically a string with an expiration date, during which it can be used for authentication requests. The session object receives and stores the session identifier, and carries the session identifier in the request header in all subsequent interaction requests with the task scheduling platform in order to maintain the authenticated session state.

[0049] S302. Initiate an execution information query request to the task scheduling platform based on the session identifier to obtain the execution identifier; An execution identifier is a unique number assigned by the task scheduling platform to each run instance, used to distinguish multiple run records of the same task at different times. For example, the same task may run once a day, each run generating a unique execution identifier, which can be used to locate the log of a specific run. Specifically, the project name, workflow name, and task name in the log query parameters are used as query conditions to construct an execution information query request, which is then sent to the task scheduling platform's execution information query interface through the task scheduling platform session. The execution information query interface is an application programming interface provided by the task scheduling platform for querying task execution information. It receives query conditions such as project name, workflow name, and task name, and returns the corresponding execution record information. After receiving the execution information query request, the task scheduling platform matches the project name, workflow name, and task name to query the most recently run execution record and returns the corresponding execution identifier to the system. If the task scheduling platform does not find an execution record matching the log query parameters, it returns an error message indicating that the task does not exist, terminates the current process, and returns an error message to the upper layer.

[0050] S303. Initiate a first log retrieval request to the task scheduling platform based on the execution identifier, and receive the first log returned by the task scheduling platform; Based on the execution identifier, a log download request is initiated to the task scheduling platform to obtain the first log. Specifically, a log download request carrying the execution identifier and the maximum log length parameter is constructed, using a fixed number of characters as the upper limit of the log length for a single request, and sent to the log download interface of the task scheduling platform through the task scheduling platform session. After receiving the log download request, the task scheduling platform reads the specified length of log content from the log file corresponding to the execution identifier, starting from the starting offset, and returns the read log content and the actual length of the read to the system. Each time log content is received, the acquired log content is appended to the end of the acquired log content, the offset is increased by the actual length read, and then the next log download request is initiated to the task scheduling platform, until the actual length returned in a certain instance is less than the maximum log length for a single request, at which point the log is considered to have been completely downloaded, and the loop terminates. For example, if each request is set to 50,000 characters, the request starts from the position with an offset of zero, acquiring the first 50,000 characters; then the offset is increased by 50,000, and the next 50,000 characters are requested, and so on, until the actual number of characters returned in a certain instance is less than 50,000, at which point the log is considered to have been completely acquired. In this embodiment, a unified task scheduling platform session is created to provide a reusable communication context for subsequent authentication, query, and download requests. This session object carries the connection configuration and authentication status, enabling the session identifier obtained from the authentication request to be continuously reused in subsequent execution information query and log download requests, thereby completing multiple interactions with only one authentication. On this basis, the corresponding first log file is located by the execution identifier, and large logs are completely obtained by paginated loop downloading, reducing timeout or memory overflow problems caused by excessive data volume in a single request.

[0051] In this embodiment, the authentication status is reused through a task scheduling platform session, enabling authentication, execution information query, and first log acquisition to be executed continuously within the same session, reducing the communication overhead caused by repeated authentication. Furthermore, an execution identifier is determined based on the project name, workflow name, and task name. This execution identifier is used to accurately locate the first log of the corresponding running instance, avoiding log confusion caused by multiple runs of the same task and improving the accuracy of log acquisition. Moreover, a paginated, loop-based approach is used to acquire the first log, reducing the amount of data in a single request and minimizing request timeouts or excessive resource consumption during large-scale log acquisition. This ensures complete acquisition of the first log while improving the stability of log acquisition.

[0052] The following provides a detailed explanation of how to obtain the second log. Please refer to [link / reference]. Figure 4 , Figure 4 A flowchart illustrating an embodiment of obtaining a second log in cross-platform log association export provided in this application is shown. This embodiment includes: S401. Parse the first log, extract the string that conforms to the task identifier format in the resource management platform, and take the last matching result as the association identifier. The acquired first log is parsed, specifically by using regular expressions to scan the entire first log. When the resource management platform receives tasks submitted by the task scheduling platform, it assigns a globally unique task identifier to each task and records this identifier in the text content of the first log. This task identifier has a fixed string format. The system pre-configures matching rules corresponding to the task identifier and uses these rules to scan the entire first log, extracting strings from the first log that conform to the resource management platform's task identifier format.

[0053] Since the same job may be submitted to the resource management platform multiple times during execution, the first log may contain multiple strings that conform to the task identifier format. The system arranges the matching results according to the order in which each matching string appears in the first log, and determines the string obtained from the last match as the association identifier. The association identifier is used to establish the association relationship between the first log and the corresponding second log in the resource management platform. For example, when multiple strings that conform to the task identifier format appear sequentially in the first log, the system uses the last matching result located at the end of the first log as the association identifier, so that the association identifier carried by subsequent second log retrieval requests corresponds to the task most recently submitted to the resource management platform for that job.

[0054] If no string matching the task identifier format is found in the first log, it is determined that there is no associated identifier in the first log that can be used to associate the second log of the resource management platform, and the second log acquisition process is terminated.

[0055] S402. Generate a second log retrieval request based on the association identifier, and send the second log retrieval request to the resource management platform; Based on the association identifier, a second log retrieval request is generated according to the log query interface call rules of the resource management platform. The association identifier is used as the query basis for this second log retrieval request, which is then sent to the resource management platform to request the second log corresponding to the association identifier. The second log is the log in the resource management platform corresponding to the association identifier, and may include standard output logs and standard error logs generated by one or more containers corresponding to the association identifier. Upon receiving the second log retrieval request, the resource management platform queries the corresponding log data based on the association identifier carried within the request and returns the query result to the system.

[0056] Because the second log in the resource management platform has a preset retention period, the resource management platform may not be able to return a valid second log if the retention period is exceeded, the associated identifier does not exist, or an anomaly occurs in the resource management platform. Therefore, after the system initiates a second log retrieval request, it further determines the retrieval status of the second log based on the actual response returned by the resource management platform.

[0057] S403. Receive the response result returned by the resource management platform.

[0058] The system receives the response from the resource management platform regarding the second log retrieval request and extracts the status code from the response. The status code indicates the processing result of the resource management platform for this second log retrieval request. The system pre-configures a status code rule base, which establishes the correspondence between different status codes and the second log retrieval status, including at least a first type of status code and a second type of status code. Specifically, a first type of status code corresponds to successful second log retrieval, and a second type of status code corresponds to failed second log retrieval.

[0059] After obtaining the status code, it is compared with a pre-configured status code rule base to determine its category. If the status code belongs to the first category, the second log is considered successfully retrieved, and the corresponding second log is obtained from the response result. If the status code belongs to the second category, the second log retrieval is considered unsuccessful. The second category of status codes can correspond to invalid association identifiers, non-existent logs corresponding to the association identifiers, second logs exceeding their retention period and being cleaned up, or abnormal processing by the resource management platform. By uniformly classifying different status codes, the different processing results returned by the resource management platform can be converted into a unified second log retrieval status, providing a basis for subsequent determination of logs to be exported.

[0060] In this embodiment, a string conforming to the task identifier format of the resource management platform is extracted from the first log, and the last matching result is determined as the association identifier, enabling the first log to be associated with the corresponding second log in the resource management platform. The second log is obtained based on the association identifier, and different status codes returned by the resource management platform are categorized using a status code rule base. This automatically determines the acquisition status of the second log, reducing the need for manual searching of association identifiers and judging log acquisition results. Furthermore, the second log or the first log is selected as the log to be exported based on the acquisition status of the second log. Even if the second log in the resource management platform is unavailable, the first log can still be used to complete the log output, improving the stability and fault tolerance of cross-platform log export.

[0061] This application also provides a cross-platform log association and export system; please refer to [link / reference]. Figure 5 , Figure 5A schematic diagram of an embodiment of the cross-platform log association export system provided in this application is shown. The system includes: Parsing unit 501 is used to obtain log query requests in natural language form and determine log query parameters based on the log query requests; The first acquisition unit 502 is used to initiate a first log acquisition request to the task scheduling platform according to the log query parameters and receive the first log returned by the task scheduling platform. The second acquisition unit 503 is used to parse the first log to extract the association identifier, initiate a second log acquisition request to the resource management platform based on the association identifier, and receive the response result returned by the resource management platform. The status identification unit 504 is used to determine the acquisition status of the second log based on the response result. The second log is the log in the resource management platform that corresponds to the associated identifier. Log export unit 505 is used to determine the second log as the log to be exported if the second log is successfully obtained; and to determine the first log as the log to be exported if the second log is not obtained; and to output the log to be exported.

[0062] This application also provides a device for cross-platform log association and export; please refer to [link / reference]. Figure 6 , Figure 6 A schematic diagram of an embodiment of the cross-platform log association export apparatus provided in this application is shown. The apparatus includes: Processor 601, memory 602, input / output unit 603, bus 604; The processor 601 is connected to the memory 602, the input / output unit 603, and the bus 604; The memory 602 stores a program, and the processor 601 calls the program to execute any of the methods described above for cross-platform log association export.

[0063] This application also relates to a computer-readable storage medium on which a program is stored. When the program is run on a computer, the computer executes any of the cross-platform log association export methods described above. For detailed implementation methods, please refer to... Figures 1-5 The specific implementation examples are not described here.

[0064] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0065] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0066] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0067] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0068] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for cross-platform log association and export, characterized in that, The method includes: Obtain a log query request in natural language form, and determine the log query parameters based on the log query request; Based on the log query parameters, a first log retrieval request is initiated to the task scheduling platform, and the first log returned by the task scheduling platform is received. The first log is parsed to extract the association identifier, and a second log acquisition request is sent to the resource management platform based on the association identifier, and the response result returned by the resource management platform is received. The status code is extracted from the response result, and the status code is compared with a pre-configured status code rule base to determine the acquisition status of the second log. The status code rule base includes at least a first type of status code and a second type of status code. When the status code belongs to the first type of status code, the acquisition of the second log is determined to be successful. When the status code belongs to the second type of status code, the acquisition of the second log is determined to be unsuccessful. If the second log is successfully retrieved, the second log will be identified as the log to be exported. If the second log fails to be retrieved, the first log is identified as the log to be exported; the log to be exported is then output.

2. The method according to claim 1, characterized in that, Before initiating a first log retrieval request to the task scheduling platform based on the log query parameters and receiving the first log returned by the task scheduling platform, the process also includes: The log query parameters are used to query whether there are log cache records in the local cache. The log cache records include cache execution status and log access links. The cache execution status is the job execution status recorded when the logs were exported last time. The log access links are the access addresses of the log files generated in this export. When the log cache record exists, query the task scheduling platform for the real-time execution status of the job corresponding to the log query parameter, and determine the validity of the log cache record based on the comparison result between the cached execution status and the real-time execution status, as well as the cache duration of the log cache record; when the log cache record is determined to be valid, identify the log pointed to by the log access link in the log cache record as the log to be exported and output it. When the log cache record does not exist, the step of initiating the first log retrieval request to the task scheduling platform based on the log query parameters is executed.

3. The method according to claim 2, characterized in that, The step of determining the validity of the log cache record based on the comparison result between the cache execution status and the real-time execution status, and the cache duration of the log cache record, includes: When the real-time execution status is in the final state, if the cache execution status is in the final state, then the log cache record is determined to be valid; if the cache execution status is in the running state, then the log cache record is determined to be invalid, and the step of initiating the first log acquisition request to the task scheduling platform according to the log query parameters is executed; wherein, the final state includes execution success, execution failure, and execution termination; When the real-time execution status is running, the cache duration of the log cache record is calculated, where the cache duration is the time difference between the current time and the creation time of the log cache record. If the cache duration is less than a preset threshold, the log cache record is determined to be valid. If the cache duration is greater than or equal to the preset threshold, the log cache record is determined to be invalid, and the step of initiating the first log retrieval request to the task scheduling platform according to the log query parameters is executed.

4. The method according to claim 1, characterized in that, The step of initiating a first log retrieval request to the task scheduling platform based on the log query parameters and receiving the first log returned by the task scheduling platform includes: A task scheduling platform session is created based on the log query parameters. An authentication request is then sent to the task scheduling platform through the task scheduling platform session to obtain the session identifier. Initiate an execution information query request to the task scheduling platform based on the session identifier to obtain the execution identifier; Based on the execution identifier, a first log retrieval request is initiated to the task scheduling platform, and the first log returned by the task scheduling platform is received.

5. The method according to claim 1, characterized in that, The process of parsing the first log to extract the association identifier, initiating a second log retrieval request to the resource management platform based on the association identifier, and receiving the response result returned by the resource management platform includes: The first log is parsed to extract a string that conforms to the task identifier format in the resource management platform, and the last matching result is taken as the association identifier. A second log retrieval request is generated based on the association identifier, and the second log retrieval request is sent to the resource management platform. Receive the response result returned by the resource management platform.

6. The method according to any one of claims 1 to 5, characterized in that, Before parsing the first log to extract the association identifier, the following steps are also included: The first log is stored locally; The method further includes: If the second log acquisition fails, the first log stored locally is read as the log to be exported.

7. A system for cross-platform log association and export, characterized in that, The system includes: The parsing unit is used to obtain log query requests in natural language form and determine log query parameters based on the log query requests. The first acquisition unit is used to initiate a first log acquisition request to the task scheduling platform according to the log query parameters, and to receive the first log returned by the task scheduling platform. The second acquisition unit is used to parse the first log to extract the association identifier, initiate a second log acquisition request to the resource management platform based on the association identifier, and receive the response result returned by the resource management platform. A status recognition unit is used to extract a status code from the response result, compare the status code with a pre-configured status code rule base, and determine the acquisition status of the second log. The status code rule base includes at least a first type of status code and a second type of status code. When the status code belongs to the first type of status code, it is determined that the second log acquisition was successful. When the status code belongs to the second type of status code, it is determined that the second log acquisition failed. The log export unit is used to determine the second log as the log to be exported if the second log is successfully acquired; and to determine the first log as the log to be exported if the second log is not acquired; and to output the log to be exported.

8. A device for cross-platform log association and export, characterized in that, The device includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a program stored thereon, the program performing the method as described in any one of claims 1 to 6 when executed on a computer.

Images