Log processing method and device

Abstract

The invention discloses a log processing method and device, and relates to the field of computing services. In the method, a plurality of logs are not located in the same storage directory, namely, the storage directory of at least one log is different from the storage directory of at least one of the other logs, so that a specified directory is configured, and reference files of the plurality of logs are created in the specified directory; the reference file of each log is used for storing the path of the log. Therefore, since the reference files of the plurality of logs are unified in the specified directory, the reference files of the plurality of logs can be quickly searched in the specified directory, so that the plurality of logs can be quickly acquired according to the reference files of the plurality of logs, and the dispersed stored logs can be mapped to the unified specified directory; the log collection path complexity is simplified, the log collection time is shortened, the cross-directory I / O operation frequency and resource consumption are reduced, and the overall log collection efficiency, real-time performance and timeliness are improved.

Description

Technical Field

[0001] This application relates to the field of computing services, and more particularly to a log processing method and apparatus. Background Technology

[0002] A computing service cluster is a collection or system consisting of at least one node, where each node is a computing service device (such as a server or edge computing gateway) that participates in collaborative work within the cluster. Computing service clusters can support cloud computing, high-performance computing (HPC), edge computing, financial transaction auditing, autonomous driving analytics, chip design, Scalable File Service (SFS), big data processing, distributed storage, containerized applications, or artificial intelligence training, among other things.

[0003] Nodes in the computing service cluster generate a large number of logs during the execution of user-submitted jobs. These logs record the job execution status, resource usage, and error messages, serving as core evidence for monitoring job health, optimizing resource allocation, and tracing the root cause of problems.

[0004] However, these logs are often scattered across multi-level, multi-path, and nested directory structures, lacking unified naming rules and clear hierarchical divisions. This dispersion increases log collection time, leading to frequent cross-directory input / output (I / O) operations and increased resource consumption, thus reducing the overall efficiency, real-time performance, and timeliness of log collection. Summary of the Invention

[0005] This application provides a log processing method and apparatus to address problems such as path dispersion, low collection efficiency, and insufficient real-time performance in the log collection process.

[0006] Firstly, this application provides a log processing method, including: Obtain the paths of multiple logs, which are not located in the same storage directory; create a reference file for each log in the specified directory, and use the reference file to store the path of that log; collect multiple logs based on the reference files created in the specified directory.

[0007] As can be seen, since multiple logs are not located in the same storage directory (i.e., at least one of these logs has a different storage directory than at least one of the other logs), this application configures a designated directory and creates reference files for these multiple logs within that directory. Thus, because the reference files for these multiple logs are unified in the designated directory, they can be quickly located within that directory. This allows for rapid collection of the multiple logs based on these reference files, effectively mapping scattered logs to a unified designated directory. This simplifies the complexity of log collection paths, avoids traversing multi-level, multi-path, and nested directory structures, shortens log collection time, reduces the frequency of cross-directory I / O operations and resource consumption, and improves the overall efficiency, real-time performance, and timeliness of log collection.

[0008] In one possible example of the first aspect, creating a reference file for each log in a specified directory includes: creating a folder for each log in the specified directory, with each log folder corresponding to the job for that log; and creating a reference file for that log in each log folder.

[0009] As can be seen, since the job is known first and then executed to generate logs, there is a correlation between the logs and the jobs. When a log's job corresponds to a folder, the corresponding folder for that job can be quickly found in the specified directory. Then, a reference file for that log is created in the folder corresponding to the job, so that the reference file can be quickly found in the specified directory according to folder categories, improving search efficiency. Furthermore, the logs can be quickly collected based on the reference file, improving collection efficiency.

[0010] In one possible example of the first aspect, creating a folder for each log in a specified directory includes: creating the folder for each log in the specified directory based on the job information of the job for each log.

[0011] The job information for each log includes at least one of the following: job identifier, submission time, start time, expected end time, and execution status.

[0012] As can be seen, since the job information of each log indicates the relevant information of the job, only the folder created based on the job information can ensure that the folder corresponds to the job. This allows the user to quickly find the folder corresponding to the job and the reference files in the folder under the specified directory according to the job category, provided that the job is known.

[0013] In one possible example of the first aspect, creating a folder for each log in a specified directory based on the job information of the job for each log includes: for each log, if the job for that log is in a running state, creating a folder for that log in a specified directory based on at least one of the job identifier, submission time, start execution time, and expected end execution time.

[0014] As can be seen, this application only creates folders in the specified directory for running jobs, and ensures that the folders correspond to the jobs based on at least one of the job identifier, submission time, start execution time, and expected end execution time.

[0015] In one possible example of the first aspect, the folder for each log is created in a specified directory based on the job information of the job for each log, including: for each log, if the job for that log is in the running state, the folder for that log is created in the specified directory based on the job identifier and the start execution time.

[0016] As can be seen, this application only creates folders in the specified directory for running jobs, and ensures that the folders correspond to the jobs based on the job identifier and start execution time, so that the folders corresponding to the jobs and the reference files in the folders can be quickly found in the specified directory according to the job categories.

[0017] In one possible example of the first aspect, obtaining the paths of multiple logs includes: for each log, determining whether the job of the log is a new job based on the job identifier and historical job identifier of the job; if it is determined to be a new job, obtaining the path of the log.

[0018] As can be seen, since historically executed jobs have job identifiers (i.e., historical job identifiers) and currently executed jobs also have job identifiers (i.e., current job identifiers), this application determines whether a job is a new job based on the historical job identifiers and the current job identifiers.

[0019] Since the job is a newly added job, this application detects the new job in real time and uses the logs generated by the node executing the new job as the newly generated logs, thus achieving real-time detection of newly generated logs. Then, reference files for the newly generated logs are created in a specified directory. These reference files are used to store the paths of these newly generated logs, thereby collecting these newly generated logs in a timely manner based on the reference files, improving the real-time performance and timeliness of log collection.

[0020] In one possible example of the first aspect, determining whether a job in the log is a new job based on the job identifier of the job in the log and the historical job identifier includes: determining that the job in the log is a new job if the job identifier of the job in the log is not within the historical job identifiers.

[0021] As can be seen, since there is no job identifier in the historical job identifier, it means that the node did not execute any jobs before this time, thus the job is determined to be a newly added job.

[0022] In one possible example of the first aspect, a reference file for each log is created in a specified directory, including: Create a reference file for each log file in the specified directory using symbolic links, hard links, or binding mounts.

[0023] As can be seen, when a reference file for a log is created in a specified directory using a symbolic link, this application creates a symbolic link in the specified directory. This symbolic link is the reference file for the log, and its stored content is the path pointing to the log.

[0024] Alternatively, when a reference file for a log is created in a specified directory via a hard link, this application creates a hard link in the specified directory, which serves as the reference file for that log.

[0025] Alternatively, when a reference file for a log is created in a specified directory via a binding mount, this application creates a binding mount in the specified directory, which maps the path of the log to the reference file of the log.

[0026] In one possible example of the first aspect, after creating a reference file for each log in a specified directory, the method further includes: For each log file, if the reference file for that log file has existed for a period of time greater than or equal to a first preset time, delete the reference file for that log file.

[0027] Therefore, to prevent reference files of a certain log from remaining active for an extended period, this application introduces a first preset time and executes the deletion process based on the existence time of the reference file. In this deletion process, if the existence time of the log's reference file is greater than or equal to the first preset time, the log's reference file is deleted. This allows for a certain amount of time to automatically clean up long-lived reference text, achieving automatic control of the reference file's lifecycle and improving the timeliness and efficiency of reference file cleanup.

[0028] In one possible example of the first aspect, the method further includes, before deleting the reference file of the log: The log is collected according to the reference file within the first preset time period.

[0029] As can be seen, since the reference file of the log will be deleted, this application can collect the log again based on the reference file before deleting the log, thus avoiding the omission of log collection due to the deletion of the reference file.

[0030] In one possible example of the first aspect, after collecting multiple logs, the method further includes: For each log, if the execution status of the job in that log is either completed or terminated, and the execution status has been in the completed or terminated state for a period of time greater than or equal to a second preset time, then delete the reference file of that log.

[0031] As can be seen, this application queries the execution status of a job in a specific log. When the execution status of the job in the log is found to be either "complete" or "terminated," it indicates that the job in the log has been completed or terminated. Therefore, this application introduces a second preset time and initiates the deletion process. In this deletion process, if the execution status remains in the "complete" or "terminated" state for a period greater than or equal to the second preset time, the reference file of the log is deleted. This achieves automatic cleanup of the reference text in the log after the job in the log has been completed or terminated, thus automating the lifecycle control of the reference file and improving the timeliness and efficiency of reference file cleanup.

[0032] In one possible example of the first aspect, the method further includes, before deleting the reference file of the log: The log is collected according to the reference file within the second preset time period.

[0033] As can be seen, since the reference file of the log will be deleted, this application can collect the log again based on the reference file before deleting the log, thus avoiding the omission of log collection due to the deletion of the reference file.

[0034] In one possible example of the first aspect, after collecting multiple logs, the method further includes: For each log, if the number of job execution failures in that log exceeds a preset number, delete the reference file for that log.

[0035] As can be seen, this application continuously monitors the number of job execution failures for a specific log when the job execution for that log first fails. Therefore, this application introduces a preset number of failures. If the number of job execution failures for that log exceeds the preset number, it indicates that the job for that log has ultimately failed or terminated, and the reference file for that log is promptly deleted. This achieves timely and automatic cleanup of reference text, avoids invalid logs consuming resources, completes automatic control of the reference file's lifecycle, and improves the timeliness and efficiency of reference file cleanup.

[0036] In one possible example of the first aspect, the multiple logs are logs generated by nodes in the computing service cluster executing at least one job. That is, the nodes in the computing service cluster generate multiple logs and paths to these multiple logs when executing at least one job. Thus, this application can obtain the paths to these multiple logs.

[0037] Secondly, this application provides a log processing apparatus, comprising: The acquisition unit is used to acquire the paths of multiple logs that are not located in the same storage directory. Create a unit to create a reference file for each log in a specified directory. The reference file for each log is used to store the path of that log. The collection unit is used to collect multiple logs based on reference files created in a specified directory.

[0038] In one possible example of the second aspect, regarding the creation of a reference file for each log in a specified directory, the creation unit is used to: create a folder for each log in the specified directory, with each log folder corresponding to the job for that log; and create a reference file for that log in each log folder.

[0039] In one possible example of the second aspect, regarding the creation of a folder for each log in a specified directory, the creation unit is used to: create a folder for that log in a specified directory based on the job information of the job for each log.

[0040] The job information for each log includes at least one of the following: job identifier, submission time, start time, expected end time, and execution status.

[0041] In one possible example of the second aspect, in terms of creating a folder for each log in a specified directory based on the job information of the job for each log, the creation unit is configured to: for each log, if the execution status of the job for that log is running, create a folder for that log in a specified directory based on at least one of the job identifier, submission time, start execution time, and expected end execution time.

[0042] In one possible example of the second aspect, in terms of creating a folder for each log in a specified directory based on the job information of the job for each log, the creation unit is used to: for each log, if the execution status of the job for that log is running, create a folder for that log in a specified directory based on the job identifier and start execution time.

[0043] In one possible example of the second aspect, regarding retrieving the paths of multiple logs, the retrieval unit is used for: For each log, determine whether the job in the log is a new job based on the job identifier and historical job identifier of the job in that log; if it is determined to be a new job, obtain the path of the log.

[0044] In one possible example of the second aspect, in determining whether a job in the log is a new job based on the job identifier and historical job identifier of the job in the log, the acquisition unit is used to: determine that the job in the log is a new job if the job identifier of the job is not within the historical job identifiers of the log.

[0045] In one possible example of the second aspect, regarding the creation of a reference file for each log in a specified directory, the creation unit is used for: Create a reference file for each log file in the specified directory using symbolic links, hard links, or binding mounts.

[0046] In one possible example of the second aspect, the log processing apparatus further includes a deletion unit; the deletion unit is used for: For each log file, if the reference file for that log file has existed for a period of time greater than or equal to a first preset time, delete the reference file for that log file.

[0047] In one possible example of the second aspect, the acquisition unit is also used for: Within the first preset time period, logs in the specified directory are collected based on the reference file of the log.

[0048] In one possible example of the second aspect, the log processing apparatus further includes a deletion unit; the deletion unit is used for: For each log, if the execution status of the job in that log is either completed or terminated, and the execution status has been in the completed or terminated state for a period of time greater than or equal to a second preset time, then delete the reference file of that log.

[0049] In one possible example of the second aspect, the acquisition unit is also used for: The log is collected according to the reference file within the second preset time period.

[0050] In one possible example of the second aspect, the log processing apparatus further includes a deletion unit; the deletion unit is used for: For each log, if the number of job execution failures in that log exceeds a preset number, delete the reference file for that log.

[0051] In one possible example of the second aspect, multiple logs are logs generated by nodes in the compute service cluster executing at least one job.

[0052] Thirdly, this application provides a log processing apparatus, including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the method described in the first aspect above.

[0053] Fourthly, the method described in the first aspect above is applied to computing service devices.

[0054] Fifthly, this application provides a computing service device, including the log processing apparatus described in the second or third aspect above.

[0055] Sixthly, this application provides a computing service cluster, including the computing service device described in the fifth aspect above.

[0056] In a seventh aspect, a chip according to this application includes a processor, wherein the processor performs the method described in the first aspect above. Optionally, the chip further includes an interface circuit, the processor being connected to the interface circuit, the interface circuit being used for transmitting and receiving information.

[0057] Eighthly, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement the method described in the first aspect.

[0058] Ninthly, this application provides a computer program product, including a computer program or instructions, wherein the computer program or instructions, when executed, implement the method described in the first aspect. Exemplarily, the computer program product may be a software installation package.

[0059] It is worth noting that the beneficial effects of the technical solutions in aspects two through nine can be found in the technical effects of the technical solution in aspect one above, and will not be repeated here. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the architecture of a computing service cluster according to an embodiment of this application; Figure 2 This is a schematic diagram of another computing service cluster architecture according to an embodiment of this application; Figure 3 This is a schematic diagram of another computing service cluster architecture according to an embodiment of this application; Figure 4 This is a schematic diagram of the architecture of a log processing system according to an embodiment of this application; Figure 5 This is a flowchart illustrating a log processing method according to an embodiment of this application; Figure 6 This is a flowchart illustrating another log processing method according to an embodiment of this application; Figure 7 This is a functional unit block diagram of a log processing device according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a log processing device according to an embodiment of this application. Detailed Implementation

[0061] It should be understood that the terms "first," "second," etc., used in the embodiments of this application are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, software, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may also include steps or units not listed, or may also include other steps or units inherent to these processes, methods, products, or devices.

[0062] In the embodiments of this application, "at least one" or "at least one item" refers to one or more, and "multiple" refers to two or more.

[0063] In the embodiments of this application, "and / or" describes the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. Among them, A and B can be singular or plural.

[0064] In the embodiments of this application, "at least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, and c can represent the following seven cases: a; b; c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0065] The following is a detailed explanation of the computing service cluster.

[0066] A computing service cluster can be a collection or system consisting of one or more nodes.

[0067] In this computing service cluster, nodes can work together to provide efficient, reliable, and scalable computing services. They can be distributed in different physical locations and connected together through a network to form a unified computing platform.

[0068] For example, such as Figure 1 As shown, Figure 1This is a schematic diagram of the architecture of a computing service cluster according to an embodiment of this application. The computing service cluster 10 includes multiple nodes, such as node 110, node 120, and node 130, which are interconnected via a wired or wireless network.

[0069] Optionally, the computing service cluster integrates the computing, storage, and network resources of nodes through a unified management and scheduling mechanism, presenting a unified service capability to the outside world. It supports various application scenarios in cloud services, such as cloud service computing, HPC, edge computing, financial transaction auditing, intelligent driving analysis, chip design, SFS, big data processing, distributed storage, containerized applications, or artificial intelligence training. It also supports elastic scaling, load balancing, high availability, on-demand allocation, and fault redundancy (such as other nodes taking over the work when a single node fails).

[0070] Optionally, the computing service cluster includes a cloud computing service cluster, an HPC cluster, or an edge computing cluster. A cloud computing service cluster is a distributed system composed of multiple interconnected servers (such as physical machines or virtual machines), providing users with elastic, efficient, and scalable computing, storage, and network resource services through a unified management and scheduling platform. The cloud computing service cluster is the core carrier for the implementation of cloud computing technology, supporting the operation of various cloud services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).

[0071] The nodes are explained in detail below.

[0072] A node can be a basic physical or logical unit that constitutes a computing service cluster, or a computing service device that participates in collaborative work within the cluster. These computing service devices can include physical servers, virtual servers, or edge computing gateways, etc.

[0073] Of course, the "node" mentioned in this embodiment can also be replaced with "computing service device".

[0074] Optionally, a node has its own independent hardware and software resources (such as a central processing unit (CPU), graphics processing unit (GPU), memory, storage, hard disk, network bandwidth, I / O interface, etc.) and operating system to run specific services or applications.

[0075] Optionally, nodes can interact with each other and synchronize their states through the communication protocol within the computing service cluster, jointly supporting the overall functionality of the computing service cluster.

[0076] Optionally, the nodes include management nodes, compute nodes, and storage nodes.

[0077] The management node can manage and coordinate other nodes, and is responsible for resource scheduling, node monitoring, job allocation, monitoring of job execution status (such as waiting status, running status, completed status, terminated status, aborted status or failed status, etc.), service discovery and fault recovery of the computing service cluster.

[0078] Compute nodes are responsible for executing user-submitted jobs, running agent programs, reporting their own resource status (such as CPU utilization, memory usage, I / O operations, disk space, network status, etc.) to the management node, and receiving, executing, and managing jobs assigned by the management node.

[0079] Storage nodes are responsible for storing and managing data, providing distributed data storage services, and offering persistent and highly reliable data storage capabilities for the entire cluster.

[0080] Optionally, the management node includes a scheduler. The scheduler is responsible for dynamically allocating and managing resources to ensure efficient and fair execution of jobs, playing a key role in cloud computing, HPC, edge computing, financial transaction auditing, autonomous driving analytics, chip design, SFS, big data processing, distributed storage, containerized applications, or artificial intelligence training. For example, the scheduler includes the Donau scheduler.

[0081] Optionally, the management node includes a job status manager. The job status manager is a component that manages the execution status of jobs through the management node's (such as the Danube scheduler) application programming interface (API) polling mechanism or event listening mechanism. For example, the job status manager includes a job status changer, which is a component that captures changes in the execution status of jobs (such as changing from running to completed) through the management node's API polling mechanism or event listening mechanism.

[0082] Optionally, the compute node includes an execution of an HPC server, an Elastic Compute Service (ECS), or an HPC-ECS. Here, ECS is a high-performance, elastic, and secure cloud server, while HPC-ECS is an ECS instance used to execute HPC.

[0083] The assignment will be explained in detail below.

[0084] A job can refer to a specific work unit submitted by a user to a computing service cluster that needs to be executed collaboratively or independently by cluster nodes. It can also be the target of resource scheduling and computing services.

[0085] Of course, the term "job" mentioned in this embodiment can also be replaced with "task" or similar terms.

[0086] Optionally, a job is to start one or more processes on one or more nodes to execute, with clear resource requirements, objectives, execution logic and expected results, and is designed to use the resources of the cluster to complete specific computations, processing or user needs.

[0087] Optionally, job execution by a node includes steps such as submission, queuing, scheduling, and execution. During submission, the user submits the job to the management node in the computing service cluster via a user interface or application programming interface (API). During queuing, the management node places the job in a queue based on resources and policies. During scheduling, the management node allocates suitable compute nodes and / or storage nodes to execute the job based on factors such as current load and resource availability. During execution, compute nodes can start corresponding processes to execute the job, releasing occupied resources upon job completion, failure, or termination.

[0088] Optionally, the operation has the following characteristics: Independence and integrity: Each job is an independent unit of work, containing clearly defined input, processing logic, and output requirements. For example, processing 1 million user data entries and generating statistical reports, or running a machine learning model training exercise.

[0089] Schedulerability: The management node schedules appropriate computing nodes for the job to be executed based on resource availability, supporting distributed processing. This means that complex jobs can be broken down into multiple sub-jobs that can be executed in parallel by multiple nodes.

[0090] Optionally, the jobs include cloud computing service jobs, HPC jobs, edge computing jobs, financial transaction auditing jobs, intelligent driving analysis jobs, chip design jobs, SFS jobs, big data processing jobs, distributed storage jobs, containerized application jobs, data backup jobs, data migration jobs, or machine learning training jobs, etc.

[0091] Optionally, a job has a lifecycle. The job lifecycle refers to the series of state changes that occur from the moment the job is created by the user until it leaves the system and releases all resources. The job lifecycle can be divided into the following stages: Creation and Submission: Users prepare the program code, resource requirements (e.g., 4 CPUs, 16GB memory), input data locations, execution parameters, etc., needed for the job. Users submit the job description to the management node of the computing service cluster via command-line tools, web interface, or API.

[0092] Scheduling and queuing: The management node receives jobs, checks their resource requirements, and places them into a queue to wait based on the current cluster load and scheduling policy. The management node continuously monitors cluster resources, and when sufficient resources (such as idle CPU and memory) become available, it selects a job from the queue (e.g., based on priority, fairness, or other policies) and assigns it to a suitable compute node.

[0093] Preparation and Execution: The compute node prepares the runtime environment for the job (such as pulling container images, setting environment variables, etc.), allocates the requested CPU and memory resources, and starts the job process.

[0094] Monitoring and Operation: The job executes its computational tasks on the compute nodes. During this time, the nodes (such as management nodes) continuously monitor the job's status, resource usage, and progress.

[0095] Completion and Termination: After a job is completed, it enters a final, immutable state, including completion, failure, and termination. A completion state indicates that the job has executed normally and achieved its intended goal (e.g., successfully outputting results or returning a normal exit code). A failure state indicates that the job encountered an error during execution and exited abnormally (e.g., code error, insufficient resources, or data error), failing to complete normally. A termination state indicates that the job was actively canceled by the user, forcibly terminated by the system administrator, or forcibly stopped by the system due to a serious error (e.g., node crash).

[0096] Optionally, the job includes job information, which includes at least one of the following: job identifier, submission time, start time, expected end time, or execution status.

[0097] A job identifier is used to uniquely identify a job. It includes the job name or identifier (ID) and is used to associate the logs generated by the node executing the job, which facilitates traceability and debugging.

[0098] Submission time refers to the time when a user submits a job to the computing service cluster, accurate to the second or millisecond. It serves as the basis for calculating job queuing time, tracing the submission order of jobs, and assisting in the analysis of scheduling delays.

[0099] The start execution time can refer to the time when the computing service cluster allocates the job to a specific node and the job actually starts running (such as starting the process or executing code), accurate to the second or millisecond. It marks the starting point of the actual consumption of cluster resources in the job's life cycle and serves as the basis for the execution time of the computing job. Combined with the submission time, it can be used to evaluate the scheduling efficiency of the computing service cluster.

[0100] The estimated completion time can refer to the estimated job completion time based on the job's resource requirements, historical execution data, or user-preset parameters. It can be automatically calculated by the computing service cluster based on historical data models (such as the average execution time of similar jobs), or manually specified by the user when submitting the job.

[0101] Execution status describes the stage a job is in during its lifecycle, reflecting its current operational status and providing real-time progress updates. It is the core basis for users to monitor jobs, troubleshoot problems (e.g., checking error logs in a failed state), and trigger subsequent operations (e.g., automatically starting downstream jobs upon completion). Execution status includes running, suspended, completed, failed, and terminated states. Running status indicates the job has been allocated resources and is executing, consuming resources. Suspended status indicates the job has been forced to stop due to non-fatal issues (e.g., insufficient resources, missing dependencies), and can be resubmitted for execution.

[0102] The log is explained in detail below.

[0103] Logs can be text files, structured files, or binary files generated by nodes during job execution. They record various events, errors, warnings, debugging information, operation information, running status, and other operations that occur during job execution. This helps administrators or developers trace the job execution process, troubleshoot problems, or perform performance analysis, ensuring the correct execution of tasks and timely detection of issues.

[0104] Of course, the log mentioned in this embodiment can also be replaced with a log file. The following embodiment mainly uses the log as an example for specific explanation.

[0105] It is worth noting that each log has a path in the file system. The log path can be a string identifier used to uniquely locate and access a log, and can be either an absolute path or a relative path.

[0106] For example, the absolute path of log 1 is / var / log / jobs / job_123 / error.log, and the absolute path of log 2 is / eda / log / jobs / job_456 / error.log. In the absolute path / var / log / jobs / job_123 / error.log, the leading forward slash ( / ) represents the root directory, var / represents a subdirectory or folder under the root directory, log / represents a subdirectory or folder under var / , jobs / represents a subdirectory or folder under log / , job_123 / represents a subdirectory or folder under jobs / , and error.log represents the final log.

[0107] As can be seen, since var / , log / , jobs / , and job_123 / form a nested directory structure, log 1 is stored within this nested directory structure. Similarly, log 2 is also stored within a nested directory structure. Furthermore, because / var / log / jobs / job_123 / and / eda / log / jobs / job_456 are different multi-level, multi-path, and nested directory structures, the paths to log 1 and log 2 are dispersed.

[0108] Optionally, the log includes at least one of the following: timestamp, node identifier, node status, job information, error information, or audit log.

[0109] The log includes several key elements: timestamps record the specific time of the log entry, helping to pinpoint when a problem occurred; node identifiers provide unique identifiers to distinguish different nodes, such as node names or Internet Protocol (IP) addresses; node status records the health status of nodes during job execution (e.g., CPU utilization, memory usage, I / O operations, disk space, network conditions) for performance analysis and optimization; error messages record relevant error information and stack traces when problems occur during job execution, aiding developers in troubleshooting; and audit logs record information such as user actions and permission changes.

[0110] The log processing method of this application embodiment will be described in detail below.

[0111] Because nodes in the computing service cluster generate a large number of logs during job execution, and these logs are often scattered and stored in multi-level, multi-path, and nested directory structures, lacking unified naming rules and clear hierarchical division, this dispersion not only increases log collection time, but also leads to frequent cross-directory I / O operations and increased resource consumption, thus significantly reducing the overall log collection efficiency, real-time performance, and timeliness.

[0112] For example, such as Figure 2 As shown, the Danube scheduler 210 in the compute service cluster schedules resources and nodes for multiple Elastic File Service (SFS) jobs, including Elastic File Service 2201, Elastic File Service 2202, and Elastic File Service 2203. Multiple Filebeats are deployed on the nodes in the compute service cluster, including Filebeat 2301, Filebeat 2302, and Filebeat 2303. Each Filebeat is a lightweight log collection tool used for real-time log monitoring, querying and obtaining log paths, and collecting logs.

[0113] exist Figure 2 In the process, file transport 2301 is used to monitor, query and collect logs generated by nodes in the computing service cluster executing elastic file service 2201, file transport 2302 is used to monitor, query and collect logs generated by nodes in the computing service cluster executing elastic file service 2202, and file transport 2303 is used to monitor, query and collect logs generated by nodes in the computing service cluster executing elastic file service 2203.

[0114] exist Figure 2 In this process, the logs generated by node executing elastic file service 2202, the logs generated by node executing elastic file service 2203, and the logs generated by node executing elastic file service 2203 are stored in multi-level, multi-path, and nested directory structures. The paths of the logs generated by node executing elastic file service 2202, the logs generated by node executing elastic file service 2203, and the logs generated by node executing elastic file service 2203 are all different, which reduces the overall collection efficiency, real-time performance, and timeliness of these logs.

[0115] To collect these logs, you can try the following methods: One approach is log collection driven by human experience. This solution relies on operations and maintenance personnel manually configuring log collection paths and estimating log generation locations based on historical experience. However, this approach depends on the experience and judgment of operations and maintenance personnel, lacks automation and intelligence, cannot adapt to dynamic changes in log paths, cannot meet the needs of real-time monitoring and rapid response, and struggles to guarantee the accuracy, completeness, and real-time performance of the collected data.

[0116] One approach is a static log collection strategy. This method sets a fixed log collection directory and periodically scans these directories to obtain logs. However, this approach cannot cover all log paths, resulting in a large number of logs being missed or collected repeatedly, leading to wasted resources and low collection efficiency. It also cannot dynamically adapt to changes in log paths, failing to meet the needs of real-time monitoring and rapid response.

[0117] One approach is isolated log source management. This solution configures log collection parameters individually for each job, such as log path and collection frequency. While this improves the targeting of collection, it suffers from high configuration complexity, difficult management, and struggles to achieve centralized log management and analysis. As the number of jobs increases and log paths change dynamically, the configuration and maintenance costs of this solution rise sharply, failing to meet the needs of real-time monitoring and rapid response.

[0118] To address the issues of fragmented paths, low collection efficiency, and insufficient real-time and timeliness in log collection, this embodiment proposes a log processing method.

[0119] In the log processing method, this embodiment can obtain the path of at least one log generated by a node in the computing service cluster executing at least one job. Specifically, at least one of these logs has a different path than the remaining at least one log, and / or at least one of these logs has a different storage directory than the remaining at least one log, with different paths existing in different storage directories, resulting in the log paths being distributed, and / or the logs being stored in different storage directories.

[0120] Because the logs generated by the nodes executing these jobs are scattered across different paths, this embodiment configures only one designated directory. A reference file for each log is created in this designated directory, and the reference file stores the path of that log. The designated directory can refer to a directory pre-specified in this embodiment.

[0121] For example, the log generated by a node executing a job (job_123) is located at / var / log / jobs / job_123 / error.log, with the specified directory being / app / eda / log / . The path to create the folder corresponding to this job within the specified directory is / app / eda / log / .<job_123> / ,<job_123> / represents the folder corresponding to the job, and error.log represents the log generated by the job.

[0122] In this way, since multiple log reference files are used to store the paths of these multiple logs, and these multiple log reference files are all located in a specified directory, this embodiment can quickly find the multiple log reference files in the specified directory, so as to quickly collect the multiple logs based on the multiple log reference files. This realizes the mapping of scattered logs to a unified specified directory, simplifies the path complexity of log collection, avoids traversing multi-level, multi-path, and nested directory structures, shortens the log collection time, reduces the frequency of cross-directory I / O operations and resource consumption, and improves the overall log collection efficiency, real-time performance, and timeliness.

[0123] For example, such as Figure 3 As shown, the Danube scheduler 310 in the computing service cluster schedules resources and nodes for multiple SFS jobs, including Elastic File Service 3201, Elastic File Service 3202, and Elastic File Service 3203. A log processing system is deployed on the nodes in the computing service cluster, which includes an HPC server 330 and a file transport (i.e., file transport 340).

[0124] exist Figure 3 In the process, HPC server 330 creates folders corresponding to elastic file services 3201, 3202, and 3203 under the specified directory 350. Then, HPC server 330 creates reference file 1 in the folder corresponding to elastic file service 3201, which stores the path of logs generated by the node executing elastic file service 3201; creates reference file 2 in the folder corresponding to elastic file service 3202, which stores the path of logs generated by the node executing elastic file service 3202; and creates reference file 3 in the folder corresponding to elastic file service 3203, which stores the path of logs generated by the node executing elastic file service 3203.

[0125] exist Figure 3 In the process, the file transport 340 collects logs from the specified directory 350 according to reference file 1 to obtain the logs generated by the node executing the elastic file service 3201, collects logs from the specified directory 350 according to reference file 2 to obtain the logs generated by the node executing the elastic file service 3202, and collects logs from the specified directory 350 according to reference file 3 to obtain the logs generated by the node executing the elastic file service 3203.

[0126] It can be seen that, compared to the above Figure 2 , Figure 3 Reference files can be quickly retrieved by simply searching under the specified directory 350, and logs can be quickly collected based on the reference files. This shortens the log collection time, reduces the frequency of cross-directory I / O operations and resource consumption, and improves the overall efficiency, real-time performance and timeliness of log collection.

[0127] The following is an example of a log processing system according to this embodiment.

[0128] like Figure 4 As shown, Figure 4 This is a schematic diagram of the architecture of a log processing system according to an embodiment of this application. Figure 4 In the log processing system 40, there are a log path acquisition module 410, a log processing module 420, and a log collection module 430.

[0129] The log path acquisition module 410 is used to acquire the paths of multiple logs that are not located in the same storage directory, and stores the paths of these multiple logs in the database to provide data support for subsequent log processing.

[0130] It should be understood that multiple logs are not located in the same storage directory, that is, the storage directory of at least one of these logs is different from the storage directory of at least one of the other logs.

[0131] For example, “Log1, Log2 and Log3 are not located in the same storage directory” includes the following situations: (1) Log1, Log2 and Log3 have different storage directories; (2) Log1 and Log2 have the same storage directory, and Log3 has a different storage directory than Log1 and Log2; (3) Log1 and Log3 have the same storage directory, and Log2 has a different storage directory than Log1 and Log3; (4) Log2 and Log3 have the same storage directory, and Log1 has a different storage directory than Log2 and Log3.

[0132] The log processing module 420 is used to create a reference file for each log in a specified directory, and the reference file for each log is used to store the path of that log.

[0133] The log collection module 430 is used to collect multiple logs based on reference files created in a specified directory.

[0134] As can be seen, the log processing system 40 obtains the paths of multiple logs. Since these multiple logs are not located in the same directory, this embodiment configures a specified directory and creates a reference file for each log in the specified directory.

[0135] In this way, since the reference files for these multiple logs are all located in the specified directory, the log processing system 40 can quickly find the reference files for these multiple logs in the specified directory. This allows for the rapid collection of these multiple logs based on the reference files, thus mapping the scattered logs to a unified specified directory. This simplifies the path complexity of log collection, avoids traversing multi-level, multi-path, and nested directory structures, shortens log collection time, reduces the frequency of cross-directory I / O operations and resource consumption, and improves the overall efficiency, real-time performance, and timeliness of log collection.

[0136] In one possible example, the multiple logs are logs generated by nodes in the compute service cluster executing at least one job.

[0137] The following is an example of the log path acquisition module 410.

[0138] In one possible example, the log path acquisition module 410 is a component, software program, tool, or hardware deployed on a node in a computing service cluster. For example, in Figure 3 In the document transport 340, a log path acquisition module 410 is included.

[0139] In one possible example, the log path acquisition module 410 is used to periodically or periodically query the path of at least one log generated by a node in the computing service cluster executing at least one job, thereby obtaining the paths of multiple logs and being able to dynamically adjust the period length as needed.

[0140] In one possible example, the log path acquisition module 410 is also used to determine, for each log, whether the job in the log is a new job based on the job identifier and historical job identifier of the job in the log, and to acquire the path of the log if it is determined to be a new job.

[0141] It should be noted that in large-scale operation scenarios, existing tools struggle to detect newly generated logs in real time, failing to meet the demands for real-time monitoring and rapid response, resulting in poor real-time log collection. For example, in critical areas such as chip design, insufficient real-time log collection can lead to delayed problem detection, impacting the progress and quality of the entire design process.

[0142] Since a certain log entry is a newly added job, this embodiment can use the log path acquisition module 410 to detect the new job in real time and use the logs generated by the node executing the new job as the newly generated logs, thereby achieving real-time detection of newly generated logs. Then, reference files for the new job are created in a specified directory. These reference files are used to store the paths of these newly generated logs, thereby collecting these newly generated logs in a timely manner based on the reference files, improving the real-time performance and timeliness of log collection.

[0143] Optionally, determining whether a job in the log is a new job based on the job identifier and historical job identifier of the job in the log includes: if the job identifier of the job in the log is not in the historical job identifiers (i.e., the job identifier of the job in the log is not in the historical job identifiers), then the job in the log is determined to be a new job.

[0144] As can be seen, since there is no job identifier in the historical job identifier, it means that the node did not execute the job before this, so the log path acquisition module 410 can determine that the job is a newly added job.

[0145] The following is an example of the log processing module 420.

[0146] In one possible example, the log processing module 420 is a component, software program, tool, or hardware deployed on nodes in a computing service cluster. For example, in Figure 3 In the HPC server 330, there is a log processing module 420.

[0147] In one possible example, the log processing module 420 is used to create a folder for each log in a specified directory, and to create a reference file for that log in each log folder.

[0148] Each log file corresponds to a specific job, and these log files can be in the same or different folders.

[0149] Each log entry can be from the same job or different jobs.

[0150] As can be seen, since the job is known first and then executed to generate logs, there is a correlation between the logs and the jobs. When a log's job corresponds to a folder, the corresponding folder for that job can be quickly found in the specified directory. Then, a reference file for that log is created in the folder corresponding to the job, so that the reference file can be quickly found in the specified directory according to folder categories, improving search efficiency. Furthermore, the logs can be quickly collected based on the reference file, improving collection efficiency.

[0151] In one possible example, the log processing module 420 is used to create a folder for each log in a specified directory based on the job information of the log job. The job information of the log job includes at least one of the following: job identifier, submission time, start time, expected end time, and execution status.

[0152] It should be understood that the job information may include any one, two, three, four or all of the following: job identifier, submission time, start time, expected end time, and execution status. This list is not exhaustive.

[0153] It should be noted that the management node can obtain job information and send it to the log processing module 420. The log processing module 420 can then create a folder in a specified directory based on the job information. Since the job information indicates relevant job information, only folders created based on the job information can guarantee a correspondence between the folder and the job. This allows for quick location of the corresponding folder and its reference files within the specified directory, provided the job is known.

[0154] Optionally, create a folder for each log under the specified directory according to the job information of the job corresponding to each log, including: for each log, when the execution status of the job corresponding to this log is the running status, create a folder for this log under the specified directory according to at least one of the job identifier, submission time, start execution time, and estimated end execution time.

[0155] For example, the job identifier of the job corresponding to a certain log is job_123, the path of this log is / var / log / jobs / job_123 / error.log, and the specified directory is / app / eda / log / . If the execution status of the job corresponding to this log is the running status, create a folder for this log under the specified directory according to the job identifier of the job corresponding to this log, and the path of the folder for this log is / app / eda / log / <job_123> / , where <job_123> / represents a folder named after the job identifier (i.e., job_123) of the job corresponding to this log; Alternatively, if the execution status of the job corresponding to this log is the running status, create a folder for this log under the specified directory according to the submission time of the job corresponding to this log, and the path of the folder for this log is / app / eda / log / <submission time> / , where <submission time> / represents a folder named after the submission time of the job corresponding to this log; Alternatively, if the execution status of the job corresponding to this log is the running status, create a folder for this log under the specified directory according to the start execution time of the job corresponding to this log, and the path of the folder for this log is / app / eda / log / <start execution time> / , where <start execution time> / represents a folder named after the start execution time of the job corresponding to this log; Alternatively, if the execution status of the job corresponding to this log is the running status, create a folder for this log under the specified directory according to the estimated end execution time of the job corresponding to this log, and the path of the folder for this log is / app / eda / log / <estimated end execution time> / , where <estimated end execution time> / represents a folder named after the estimated end execution time of the job corresponding to this log; Alternatively, if the execution status of the job corresponding to this log is the running status, create a folder for this log under the specified directory according to the job identifier and start execution time of the job corresponding to this log, and the path of the folder for this log is / app / eda / log / <job_123 - start execution time> / , where <job_123 - start execution time> / represents a folder named after the job identifier (i.e., job_123) and start execution time of the job corresponding to this log.

[0156] As can be seen, the log processing module 420 only creates folders in the specified directory for running jobs, and ensures that the folders correspond to the jobs based on at least one of the job identifier, submission time, start execution time, and expected end execution time.

[0157] In one possible example, the log processing module 420 is used to create the first file in the first folder via a symbolic link, a hard link, or a bind mount.

[0158] A symbolic link can be a special file that stores the path to another file.

[0159] In this way, the log processing module 420 creates a reference file for a specific log in a specified directory using a symbolic link. This can be understood as creating a symbolic link in the specified directory. The symbolic link is the file for that log entry, and its stored content is the path to that log entry.

[0160] Hard links can be additional directory entry mappings to the file identifier (inode) of the original file, that is, assigning multiple filenames to the same original file. All hard links share the same inode and data blocks as the original file.

[0161] In this way, the log processing module 420 creates a reference file for a certain log in the specified directory through a hard link. This can be understood as creating a hard link in the specified directory, which is the reference file for the log. The reference file and the log are essentially the same file with different names and share the same inode.

[0162] Binding mount is a mechanism that mounts a directory or file in a file system to another directory, so that the two paths point to the same storage location.

[0163] In this way, the log processing module 420 creates a reference file for a certain log in a specified directory through a binding mount method. This can be understood as creating a binding mount in a specified directory, which maps the path of the log to the reference file of the log. Any operation on the reference file of the log will be directly reflected on the path of the log, and vice versa.

[0164] In one possible example, the log processing module 420 is also configured to delete the reference file of each log if the reference file of the log has existed for a first time or equal to a first preset time.

[0165] It should be noted that this embodiment introduces the lifecycle of the reference file and records the existence time of the reference file, so as to accurately calculate the lifespan of the reference file by the existence time of the reference file.

[0166] To prevent reference files of a particular log from remaining active for an extended period, especially if the reference file may have expired, this embodiment introduces a first preset time (such as 1 hour, 6 hours, 12 hours, 24 hours, 48 ​​hours, or 72 hours). This first preset time can be a default value or dynamically adjusted, and the deletion process is executed based on the existence time of the log's reference file. In this deletion process, if the existence time of the log's reference file is greater than or equal to the first preset time, the log's reference is deleted. This allows for a certain amount of time to automatically clean up long-lived reference text, achieving automatic control of the reference file's lifecycle and improving the timeliness and efficiency of reference file cleanup.

[0167] In one possible example, the log processing module 420 is also configured to delete the reference file of each log if the second execution state of the job in that log is a completed state or a terminated state, and the time in which the second execution state is in the completed state or terminated state is greater than or equal to a first preset time.

[0168] It should be noted that this embodiment introduces a reference file lifecycle and periodically or irregularly queries the execution status of a job in a certain log. When the execution status of a job in a log is found to be either completed or terminated, it indicates that the job in that log has been completed or terminated.

[0169] Therefore, this embodiment introduces a second preset time (such as 1 hour, 6 hours, 12 hours, 24 hours, 48 ​​hours, or 72 hours, etc.). The second preset time can be a default or dynamically adjusted time, and the deletion process begins. In this deletion process, if the time the second execution state is in the completed or terminated state is greater than or equal to the second preset time, the first file is deleted. This achieves automatic cleanup of the log's reference text after the job execution of the log is completed or terminated, completing automatic control of the reference file's lifecycle and improving the timeliness and efficiency of reference file cleanup.

[0170] In one possible example, the log processing module 420 is also used to delete the reference file of each log if the number of job execution failures for that log exceeds a preset number.

[0171] It should be noted that this embodiment introduces the lifecycle of the first file, and continuously detects the number of times the job execution of a certain log fails when the job execution of a certain log fails for the first time.

[0172] Therefore, this embodiment introduces a preset number of attempts (such as 2, 4, or 8 times), which can be a default or dynamically adjusted number. If the number of times the job execution of the log fails exceeds the preset number, it indicates that the job of the log has ultimately failed or terminated, and the reference file of the log is deleted in a timely manner. This achieves timely and automatic cleaning of reference text, avoids invalid logs occupying resources, completes the automatic control of the life cycle of reference files, and improves the timeliness and efficiency of reference file cleaning.

[0173] The following is an example of the log collection module 430.

[0174] In one possible example, the log collection module 430 is a component, software program, tool, or hardware deployed on nodes in a computing service cluster. For example, in Figure 3 In the document transport 340, a log collection module 430 is included.

[0175] In one possible example, the log collection module 430 is used to collect multiple logs according to a reference file for each log at a preset collection frequency. The preset collection frequency can be fixed or dynamically adjusted to ensure efficient, real-time, and timely log collection.

[0176] In one possible example, the log collection module 430 is also used to collect the log according to a reference file within a first preset time period.

[0177] As can be seen, since the reference file of the log will be deleted, the log collection module 430 can collect the log again based on the reference file before deleting the log, thus avoiding missing the log due to the deletion of the reference file.

[0178] In one possible example, the log collection module 430 is also used to collect the log according to a reference file within a second preset time period.

[0179] As can be seen, since the reference file of the log will be deleted, the log collection module 430 can collect the log again based on the reference file before deleting the log, thus avoiding missing the log due to the deletion of the reference file.

[0180] The other modules of the log processing system 40 will be described in detail below.

[0181] In one possible example, the log processing system 40 also includes a log generation module. This log generation module is used to generate multiple logs during the execution of at least one job.

[0182] Optionally, the log generation module includes compute nodes in the compute service cluster.

[0183] In one possible example, the log processing system 40 also includes a log storage module. This log storage module is used to receive and store at least one collected log, and the path to store that at least one log.

[0184] Optionally, the log storage module is a component, software program, tool, or hardware deployed on nodes in a computing service cluster. For example, log storage modules include Elasticsearch or the Hadoop Distributed File System (HDFS).

[0185] In one possible example, the log processing system 40 also includes a log query module. This log query module provides functions such as querying and visualizing at least one log entry.

[0186] Optionally, the log query module is a component, software program, tool, or hardware deployed on nodes in the computing service cluster.

[0187] In one possible example, the log processing system 40 also includes a system testing and optimization module. This module can be used to test the log processing system 40, verifying its ability to efficiently and accurately collect all logs, and to optimize the system based on the test results, such as adjusting log processing strategies and optimizing log collection performance. For example, the system testing and optimization module could include Grafana or a custom analytics engine.

[0188] Optionally, the system testing and optimization module is a component, software program, tool, or hardware deployed on nodes in the computing service cluster.

[0189] The following is an example of a log processing method according to an embodiment of this application.

[0190] like Figure 5 As shown, Figure 5 This is a flowchart illustrating a log processing method provided in an embodiment of this application. This method can be applied to computing service devices, chips, units, or devices. The method includes the following steps: S510. Obtain the paths of multiple logs that are not located in the same storage directory.

[0191] S520. Create a reference file for each log in the specified directory. The reference file for each log is used to store the path of that log.

[0192] S530. Collect multiple logs based on reference files created in the specified directory.

[0193] It should be noted that there are identical contents here to those in the above embodiments, so these identical contents can be found in the above embodiments and will not be repeated here.

[0194] As can be seen, to address the issues of path dispersion, low collection efficiency, and insufficient real-time performance in log collection, this embodiment configures a designated directory and creates multiple log reference files within that directory. Since these multiple log reference files are unified under the designated directory, this embodiment can quickly locate these reference files within the specified directory, enabling rapid collection of multiple logs based on them. This maps scattered log storage to a unified designated directory, simplifying the path complexity of log collection, avoiding traversing multi-level, multi-path, and nested directory structures, shortening log collection time, reducing the frequency of cross-directory I / O operations and resource consumption, and improving the overall efficiency, real-time performance, and timeliness of log collection.

[0195] In one possible example, creating a reference file for each log in a specified directory includes: creating a folder for each log in the specified directory, with each log folder corresponding to the job for that log; and creating a reference file for that log in each log folder.

[0196] Each log can be in the same or different folders, and each log can be in the same or different jobs.

[0197] As can be seen, since the job is known first and then executed to generate logs, there is a correlation between the logs and the jobs. When a log's job corresponds to a folder, the corresponding folder for that job can be quickly found in the specified directory. Then, a reference file for that log is created in the folder corresponding to the job, so that the reference file can be quickly found in the specified directory according to folder categories, improving search efficiency. Furthermore, the logs can be quickly collected based on the reference file, improving collection efficiency.

[0198] In one possible example, creating a folder for each log in a specified directory includes: creating the folder for each log in the specified directory based on the job information of the job for each log.

[0199] The job information for each log includes at least one of the following: job identifier, submission time, start time, expected end time, and execution status.

[0200] Each log entry can be from the same job or different jobs.

[0201] As can be seen, since the job information of each log indicates the relevant information of the job, only the folder created based on the job information can ensure that the folder corresponds to the job. This allows the user to quickly find the folder corresponding to the job and the reference files in the folder under the specified directory according to the job category, when the job is known.

[0202] In one possible example, based on the job information of each log job, a folder for that log is created in a specified directory, including: for each log, if the execution status of the job for that log is running, creating a folder for that log in the specified directory based on at least one of the job identifier, submission time, start execution time, and expected end execution time.

[0203] As can be seen, this application only creates folders in the specified directory for running jobs, and ensures that the folders correspond to the jobs based on at least one of the job identifier, submission time, start execution time, and expected end execution time.

[0204] In one possible example, based on the job information of each log job, a folder for that log is created in a specified directory, including: for each log, if the job execution status of that log is running, creating a folder for that log in a specified directory based on the job identifier and start execution time.

[0205] As can be seen, this application only creates folders in the specified directory for running jobs, and ensures that the folders correspond to the jobs based on the job identifier and start execution time, so that the folders corresponding to the jobs and the reference files in the folders can be quickly found in the specified directory according to the job categories.

[0206] In one possible example, the path to multiple logs is obtained, including: for each log, determining whether the job in the log is a new job based on the job identifier and historical job identifier of the job in that log; if it is determined to be a new job, obtaining the path to the log.

[0207] As can be seen, since historically executed jobs have job identifiers (i.e., historical job identifiers) and currently executed jobs also have job identifiers (i.e., current job identifiers), this application determines whether a job is a new job based on the historical job identifiers and the current job identifiers.

[0208] Since the job is a newly added job, this application detects the new job in real time and uses the logs generated by the node executing the new job as the newly generated logs, thus achieving real-time detection of newly generated logs. Then, reference files for the newly generated logs are created in a specified directory. These reference files are used to store the paths of these newly generated logs, thereby collecting these newly generated logs in a timely manner based on the reference files, improving the real-time performance and timeliness of log collection.

[0209] In one possible example, determining whether a job in a log is a new job is based on the job identifier and historical job identifier of the job in the log, including: determining that the job in the log is a new job if the job identifier of the job in the log is not among the historical job identifiers.

[0210] As can be seen, since there is no job identifier in the historical job identifier, it means that the node did not execute any jobs before this time, thus the job is determined to be a newly added job.

[0211] In one possible example, a reference file for each log entry is created in a specified directory, including by creating a reference file for each log entry in the specified directory via a symbolic link, a hard link, or a bound mount.

[0212] As can be seen, when a reference file for a log is created in a specified directory using a symbolic link, this application creates a symbolic link in the specified directory. This symbolic link is the reference file for the log, and its stored content is the path pointing to the log.

[0213] Alternatively, when a reference file for a log is created in a specified directory via a hard link, this application creates a hard link in the specified directory, which serves as the reference file for that log.

[0214] Alternatively, when a reference file for a log is created in a specified directory via a binding mount, this application creates a binding mount in the specified directory, which maps the path of the log to the reference file of the log.

[0215] In one possible example, after creating a reference file for each log in a specified directory, the method further includes: for each log, deleting the reference file of that log if the reference file has existed for a period of time greater than or equal to a first preset time.

[0216] Therefore, to prevent reference files of a certain log from remaining active for an extended period, this application introduces a first preset time and executes the deletion process based on the existence time of the reference file. In this deletion process, if the existence time of the log's reference file is greater than or equal to the first preset time, the log's reference file is deleted. This allows for a certain amount of time to automatically clean up long-lived reference text, achieving automatic control of the reference file's lifecycle and improving the timeliness and efficiency of reference file cleanup.

[0217] Optionally, before deleting the reference file of the log, the method further includes: collecting the log according to the reference file of the log within a first preset time period.

[0218] As can be seen, since the reference file of the log will be deleted, this application can collect the log again based on the reference file before deleting the log, thus avoiding the omission of log collection due to the deletion of the reference file.

[0219] In one possible example, after collecting multiple logs, the method further includes: for each log, if the second execution state of the job in that log is a completed state or a terminated state, and the time that the second execution state is in the completed state or terminated state is greater than or equal to a second preset time, deleting the reference file of that log.

[0220] As can be seen, this application queries the execution status of a job in a certain log. When the execution status of the job in the log is found to be either "complete" or "terminated," it indicates that the job in the log has been completed or terminated. Therefore, this application introduces a second preset time and initiates the deletion process. In this deletion process, if the time the second execution status is in the "complete" or "terminated" state is greater than or equal to the second preset time, the reference file of the log is deleted. This achieves automatic cleanup of the reference text of the log after the job in the log has been completed or terminated, completing automatic control of the lifecycle of the reference file and improving the timeliness and efficiency of reference file cleanup.

[0221] In one possible example, before deleting the reference file of the log, the method further includes: collecting the log according to the reference file of the log within a second preset time period.

[0222] As can be seen, since the reference file of the log will be deleted, this application can collect the log again based on the reference file before deleting the log, thus avoiding the omission of log collection due to the deletion of the reference file.

[0223] In one possible example, after collecting multiple logs, the method further includes: for each log, if the number of job execution failures for that log exceeds a preset number, deleting the reference file for that log.

[0224] As can be seen, this application continuously monitors the number of times a job in a particular log fails to execute when the job first fails. Therefore, this application introduces a preset count. If the number of job failures for a particular log exceeds the preset count, it indicates that the job for that log has ultimately failed or terminated, and the reference file for that log is promptly deleted. This achieves timely and automatic cleanup of reference text, avoids invalid logs consuming resources, completes automatic control of the reference file's lifecycle, and improves the timeliness and efficiency of reference file cleanup.

[0225] In one possible example, the multiple logs are logs generated by nodes in the compute service cluster executing at least one job. That is, the nodes in the compute service cluster generate multiple logs and paths for these multiple logs when executing at least one job. Thus, this embodiment can obtain the paths of these multiple logs.

[0226] The following example illustrates part of the log processing method in the above embodiment, using the execution of the first job by a node in the computing service cluster at the first moment as an example. The first job can be one or more jobs; the first moment can be a time that arrives periodically (e.g., every 5 minutes) or non-periodicly, or it can be the current moment.

[0227] Of course, in the initial moment, nodes in the computing service cluster may execute a second job in addition to the first job. The second job could be one or more jobs other than the first job. In this case, the log processing method for the second job can be understood by referring to the first job, and will not be elaborated further.

[0228] like Figure 6 As shown, Figure 6 This is a flowchart illustrating another log processing method provided in an embodiment of this application. The method includes the following steps: S601. Computing service device 610 is a resource and node for executing the first job scheduling.

[0229] Optionally, the computing service device 610 serves as a management node. In this way, the management node serves as the resource and node for the first job scheduling.

[0230] Optionally, the computing service device 610 includes a Danube scheduler. This allows the Danube scheduler to schedule resources and nodes for the first job.

[0231] Optionally, the computing service device 620 is a computing node. In this way, the first job is executed through the computing node.

[0232] S602. Computing service device 620 performs the first task.

[0233] S603. Computing service device 620 generates the first log.

[0234] The first log is one or more logs generated by the computing service device 620 when executing the first job at the first moment.

[0235] S604. The computing service device 630 acquires the job information of the first job and stores the job information of the first job.

[0236] The job information for the first job includes the job identifier for the first job.

[0237] Among them, the execution state of the first task at the first moment is the running state.

[0238] Optionally, the computing service device 630 obtains the job information of the first job, including: the computing service device 630 polls the API of the computing service device 630 to obtain the job information of the first job.

[0239] Optionally, the job information for the first job is stored in a database.

[0240] Optionally, the computing service device 630 acquires the job information of the first job periodically or at regular intervals. Optionally, the computing service device 630 includes document consignment. Thus, logs are collected through document consignment.

[0241] Optionally, the computing service device 630 includes an HPC server. The HPC server then processes the logs generated by the execution of the first job.

[0242] Optionally, the computing service device 630 includes a task status manager. This allows the task status manager to monitor and manage the execution status of the first job.

[0243] S605. The computing service device 630 determines whether the first job is a new job based on the job identifier of the first job and the historical job identifier.

[0244] Among them, the computing service device 630 determines whether the first job is a newly added job at the first moment based on the job identifier of the first job and the historical job identifier.

[0245] If not, proceed to step S606; if yes, proceed to steps S607 through S610. Optionally, historical job identifiers are stored in a database. The computing service device 630 can query and retrieve historical job identifiers from this database.

[0246] Optionally, the computing service device 630 includes a task status manager. The task status manager determines whether the first job is a newly added job based on the job identifier of the first job and the historical job identifier.

[0247] S606. The computing service device 630 determines whether there is a reference file for the first log in the specified directory.

[0248] If not, then execute S607; if yes, then execute S610.

[0249] S607. Path to the first log obtained by computing service device 630.

[0250] S608. The computing service device 630 creates a folder for the first log in the specified directory.

[0251] S609. The computing service device 630 creates a reference file for the first log in the folder of the first log.

[0252] Optionally, the computing service device 630 creates a reference file for the first log in the folder of the first log, including: the computing service device 630 creates a soft link in the folder of the first log, the soft link being the reference file for the first log, and the stored content pointing to the path of the first log.

[0253] Optionally, to avoid retaining the reference file of the first log for an extended period of time, the computing service device 630 deletes the reference file of the first log if the existence time of the reference file of the first log is greater than or equal to a first preset time; otherwise, it continues to retain the reference file of the first log.

[0254] S610. The computing service device 630 collects the first log according to the reference file of the first log.

[0255] Optionally, the computing service device 630 includes a file transport function. The file transport function collects the first log based on a reference file of the first log. S611. The computing service device 630 determines whether the first job has been completed or terminated.

[0256] If yes, then execute S612; otherwise, execute S613.

[0257] Optionally, the computing service device 630 determines whether the first job has been completed or terminated at a second time. The second time is after the first time, and can be a time that arrives periodically (e.g., every 10 minutes) or non-periodicly.

[0258] S612. The computing service device 630 executes the process of deleting the reference file of the first log.

[0259] In this deletion process, if the time during which the first job is in a completed or terminated state is greater than or equal to a second preset time, the reference file of the first log is deleted; otherwise, the reference file of the first log is retained.

[0260] S613. The computing service device 630 continues to retain a reference file of the first log.

[0261] S614. The computing service device 630 determines whether the number of times the first job has failed exceeds a preset number.

[0262] If yes, then execute S615; otherwise, execute S613.

[0263] S615. Computing service device 630 deletes the reference file of the first log.

[0264] The following is an example description of a log processing apparatus according to an embodiment of this application.

[0265] The above embodiments describe the log processing method from a methodological perspective. The following describes an example of a functional unit of a log processing device. The log processing device includes hardware structures and / or software modules corresponding to each function. For example, functional units can be divided according to each function, or two or more functions can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0266] It should be noted that the division of units in this embodiment is illustrative and is only a logical functional division. In actual implementation, there may be other division methods.

[0267] When using integrated units, Figure 7 This is a functional unit block diagram of a log processing device according to an embodiment of this application. The log processing device 700 includes an acquisition unit 701, a creation unit 702, and a collection unit 703.

[0268] Optionally, the acquisition unit 701 is a functional unit for acquiring the path of the log.

[0269] Optionally, creation unit 702 is a functional unit used to create files in a specified directory.

[0270] Optionally, the acquisition unit 703 is a functional unit for acquiring logs.

[0271] Optionally, the log processing apparatus 700 further includes a storage unit for storing computer program code or instructions executed by the log processing apparatus 700. The storage unit may be a memory.

[0272] Optionally, the log processing device 700 is a chip or a chip module.

[0273] Optionally, the log processing apparatus 700 further includes a processing unit. The processing unit may be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this embodiment. The processing unit may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0274] Optionally, the acquisition unit 701, the creation unit 702, and the acquisition unit 703 are integrated into the processing unit.

[0275] Optionally, the log processing device 700 is used to perform any of the steps performed by the computing service device, chip, or chip module as described in the above method embodiments.

[0276] In specific implementation, the acquisition unit 701, the creation unit 702, and the collection unit 703 are used to execute the steps as described in the above method embodiments, and when performing other actions, other units may be selectively invoked to complete the corresponding operations. A detailed explanation follows.

[0277] The acquisition unit 701 is used to acquire the paths of multiple logs that are not located in the same storage directory. Create unit 702, which is used to create a reference file for each log in the specified directory. The reference file for each log is used to store the path of that log. The acquisition unit 703 is used to acquire multiple logs based on reference files created in a specified directory.

[0278] As can be seen, since multiple logs are not located in the same storage directory (i.e., at least one of these logs has a different storage directory than at least one of the other logs), this application configures a designated directory and creates reference files for these multiple logs within that directory. Thus, because the reference files for these multiple logs are unified in the designated directory, they can be quickly located within that directory. This allows for rapid collection of the multiple logs based on these reference files, effectively mapping scattered logs to a unified designated directory. This simplifies the path complexity of log collection, avoids traversing multi-level, multi-path, and nested directory structures, shortens log collection time, reduces the frequency of cross-directory I / O operations and resource consumption, and improves the overall efficiency, real-time performance, and timeliness of log collection.

[0279] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above. The log processing device 700 can be used to execute the method embodiment described above in this embodiment, and will not be described again here.

[0280] The following is an example description of another log processing apparatus according to an embodiment of this application.

[0281] The above embodiments were mainly introduced from the perspective of the method. The following is an example of another log processing device.

[0282] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a log processing device according to an embodiment of this application. The log processing device 800 may include a processor 810, a memory 820, and a communication bus for connecting the processor 810 and the memory 820.

[0283] Optionally, the memory 820 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and the memory 820 is used to store the program code executed by the log processing device 800 and the data transmitted.

[0284] Optionally, the log processing device 800 also includes a communication interface for receiving and sending data.

[0285] Optionally, the processor 810 includes a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof.

[0286] In a specific implementation, the processor 810 in the log processing device 800 executes the computer program or instructions 821 stored in the memory 820 to perform the following operations: Obtain the paths of multiple logs, which are not located in the same storage directory; create a reference file for each log in the specified directory, and use the reference file to store the path of that log; collect multiple logs based on the reference files created in the specified directory.

[0287] As can be seen, since multiple logs are not located in the same storage directory (i.e., at least one of these logs has a different storage directory than at least one of the other logs), this application configures a designated directory and creates reference files for these multiple logs within that directory. Thus, because the reference files for these multiple logs are unified in the designated directory, they can be quickly located within that directory. This allows for rapid collection of the multiple logs based on these reference files, effectively mapping scattered logs to a unified designated directory. This simplifies the path complexity of log collection, avoids traversing multi-level, multi-path, and nested directory structures, shortens log collection time, reduces the frequency of cross-directory I / O operations and resource consumption, and improves the overall efficiency, real-time performance, and timeliness of log collection.

[0288] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above. The log processing device 800 can be used to execute the method embodiment described above in this embodiment, and will not be described again here.

[0289] The following provides examples illustrating other relevant aspects of this embodiment.

[0290] In one possible example, this application embodiment provides a computing service device, which includes the log processing device 700 or the log processing device 800 described above.

[0291] In one possible example, this application embodiment provides a computing service cluster, including the computing service device described above.

[0292] In one possible example, this application provides a chip including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps described in the above method embodiments.

[0293] In one possible example, this application provides a chip module including a transceiver component and a chip. The chip includes a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps described in the above method embodiments.

[0294] In one possible example, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed, implement the steps described in the above method embodiments.

[0295] In one possible example, embodiments of this application provide a computer program product, including a computer program or instructions that, when executed, implement the steps described in the above method embodiments.

[0296] It should be noted that, for the sake of simplicity, the above embodiments are all described as a series of actions. Those skilled in the art should understand that this application is not limited to the described order of actions, as some steps in the embodiments of this application can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of this application.

[0297] In the above embodiments, the descriptions of each embodiment in this application have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0298] The steps of the methods or algorithms described in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (EEPROM), registers, hard disk, portable hard disk, read-only optical disk (CD-ROM), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Furthermore, the ASIC can reside in a terminal device or management device. Alternatively, the processor and storage medium can exist as discrete components in the terminal device or management device.

[0299] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0300] The modules or units included in the various devices and products described in the above embodiments can be software modules or units, hardware modules or units, or a combination of both. For example, for devices and products applied to or integrated into a chip, all modules or units can be implemented using hardware methods such as circuits, or at least some modules or units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules or units can be implemented using hardware methods such as circuits. For devices and products applied to or integrated into a chip module, all modules or units can be implemented using hardware methods such as circuits. Different modules or units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules or units can be implemented using hardware methods such as circuits. The implementation is achieved through a software program that runs on a processor integrated within the chip module. The remaining modules or units (if any) can be implemented using hardware methods such as circuits. For various devices or products applied to or integrated into terminal equipment, each of its modules or units can be implemented using hardware methods such as circuits. Different modules or units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal equipment. Alternatively, at least some modules or units can be implemented using a software program that runs on a processor integrated within the terminal equipment. The remaining modules or units (if any) can be implemented using hardware methods such as circuits.

[0301] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above descriptions are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A log processing method, characterized in that, include: Obtain the paths of multiple logs, which are not located in the same storage directory; Create a reference file for each log in the specified directory. The reference file for each log is used to store the path of that log. Collect the multiple logs based on the reference files created in the specified directory.

2. The method according to claim 1, characterized in that, The step of creating a reference file for each log in the specified directory includes: Create a folder for each log in the specified directory, with each log folder corresponding to the job for that log; Create a reference file for each log in its own folder.

3. The method according to claim 2, characterized in that, The step of creating a folder for each log in the specified directory includes: Based on the job information of each log, create a folder for that log in the specified directory; The job information for each log includes at least one of the following: job identifier, submission time, start time, expected end time, and execution status.

4. The method according to claim 3, characterized in that, The step of creating a folder for each log in the specified directory based on the job information of each job includes: For each log, if the job in that log is in the running state, create a folder for that log in the specified directory based on at least one of the job identifier, the submission time, the start execution time, and the expected end execution time.

5. The method according to any one of claims 1-4, characterized in that, The paths for obtaining multiple logs include: For each log, determine whether the job in that log is a new job based on the job identifier and historical job identifier of the job in that log. If it is determined to be a newly added job, obtain the path of the log.

6. The method according to any one of claims 1-4, characterized in that, The step of creating a reference file for each log in the specified directory includes: Create a reference file for each log file in the specified directory using a soft link, hard link, or binding mount method.

7. The method according to any one of claims 1-4, characterized in that, After creating a reference file for each log in the specified directory, the method further includes: For each log file, if the reference file for that log file has existed for a period of time greater than or equal to a first preset time, then delete the reference file for that log file.

8. The method according to any one of claims 1-4, characterized in that, After collecting the multiple logs, the method further includes: For each log, if the execution status of the job in that log is either completed or terminated, and the time the execution status is in the completed or terminated state is greater than or equal to a second preset time, then the reference file of that log is deleted.

9. The method according to any one of claims 1-4, characterized in that, After collecting the multiple logs, the method further includes: For each log, if the number of job execution failures in that log exceeds a preset number, delete the reference file for that log.

10. The method according to any one of claims 1-4, characterized in that, The multiple logs are logs generated by nodes in the computing service cluster executing at least one job.

11. A log processing device, characterized in that, include: The acquisition unit is used to acquire the paths of multiple logs, which are not located in the same storage directory; Create a unit to create a reference file for each log in a specified directory. The reference file for each log is used to store the path of that log. The collection unit is used to collect the multiple logs based on the reference files created in the specified directory.

12. A log processing device, characterized in that, It includes a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the method as claimed in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program or instructions that, when executed, implement the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed, implement the method as described in any one of claims 1-10.

Images