Log processing method and device, electronic equipment and computer program product

Abstract

The invention discloses a log processing method and device, electronic equipment and a computer program product. Relates to the field of financial science and technology, and comprises the following steps: receiving a log query request, and extracting log generation time and a log tag of a target log file from the log query request, the log tag being used for indicating a file type of the target log file; generating a query condition according to the log generation time and the log tag, and obtaining a storage path of the target log file from a query index associated with the object storage system according to the query condition to obtain a target storage path; and extracting a target log file from the object storage system based on the target storage path, and sending the target log file to the client. Through the method and the device, the technical problem that the query efficiency is low when the target log file is acquired from the computing cluster in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and specifically, to a log processing method, device, electronic device, and computer program product. Background Art

[0002] In today's big data processing environment, ETL (Extract, Transform, and Load) jobs play a crucial role, integrating massive amounts of information from diverse data sources into a unified data warehouse or data lake to support subsequent data analysis and business decision-making. However, the complexity and distributed nature of ETL jobs make logging crucial during their operation. Logs not only reflect the operation status of the job but also provide valuable clues when a job is interrupted or fails, helping to quickly locate and resolve the problem.

[0003] However, the nodes in distributed computing clusters are designed primarily for efficient business data processing, and their local storage resources are typically prioritized for storage and computing tasks. Consequently, the storage space for log files is strictly limited. Excessive log usage can squeeze out valuable computing resources, impacting the normal operation of the cluster and the efficiency of other tasks. To address storage space limitations, cluster management policies often specify a short log retention period, typically retaining logs for only a few hours to a day. While this helps to utilize resources efficiently, it also carries the risk of losing critical fault information, significantly increasing the difficulty of debugging and problem tracing, and making it impossible to fully exploit the value of log data for historical trend analysis or predictive maintenance.

[0004] Furthermore, due to the distributed nature of ETL jobs, the log files generated by these jobs are stored separately across the nodes in the cluster. While this distributed storage structure facilitates parallel data processing, when global log information is needed for analysis, users must manually collect logs across nodes. This process is not only time-consuming but can also lead to analysis bias due to missing logs from specific nodes, increasing operational complexity and potential errors.

[0005] Regarding the technical problem of low query efficiency when obtaining target log files from a computing cluster in related technologies, no effective solution has been proposed yet. Summary of the Invention

[0006] The main purpose of this application is to provide a log processing method, device, electronic device and computer program product to solve the technical problem of low query efficiency when obtaining target log files from a computing cluster in the related art.

[0007] To achieve the above-mentioned objectives, according to one aspect of the present application, a log processing method is provided. The method comprises: receiving a log query request, extracting the log generation time and log tag of a target log file from the log query request, wherein the log tag is used to indicate the file type of the target log file; generating a query condition based on the log generation time and the log tag, and obtaining the storage path of the target log file from a query index associated with an object storage system based on the query condition to obtain a target storage path; extracting the target log file from the object storage system based on the target storage path, and sending the target log file to a client.

[0008] Furthermore, before receiving the log query request, the method also includes: obtaining M target log generation directories in the computing cluster, extracting log file names from the M target log generation directories, and obtaining M groups of log file names, wherein each target log generation directory refers to an updated log generation directory in the computing cluster, and M is a positive integer; obtaining M groups of log files according to the M groups of log file names, and storing the M groups of log files in the object storage system.

[0009] Furthermore, obtaining M target log generation directories includes: obtaining N log collection components in the computing cluster, and obtaining the log generation directory corresponding to each log collection component, to obtain N log generation directories, where N is greater than or equal to M and N is a positive integer; controlling each log collection component to detect each log generation directory in the computing cluster; if the N log generation directories include M updated log generation directories, the M updated log generation directories are determined as the M target log generation directories.

[0010] Furthermore, obtaining M groups of log files according to the names of the M groups of log files includes: obtaining the initial log file from the computing cluster according to the name of each log file to obtain M groups of initial log files; verifying each initial log file in each group of initial log files, and if the verification of the M groups of initial log files passes, field filtering is performed on each initial log file to obtain M groups of log files.

[0011] Furthermore, storing M groups of log files in the object storage system includes: compressing the M groups of log files to obtain M groups of compressed log files; obtaining a log tag in each compressed log file to obtain M groups of log tags, wherein each log tag is used to indicate the file type of each compressed log file; obtaining a file storage type of the object storage system, classifying the M groups of log tags according to the file storage type, and obtaining a classification result; and storing the compressed log files associated with the M groups of log tags in the object storage system based on the classification result.

[0012] Furthermore, the object storage system is associated with a first database and a second database, and storing M groups of compressed log files associated with log tags in the object storage system based on the classification results includes: calling an application programming interface associated with the object storage system, and uploading M groups of compressed log files to the first database of the object storage system based on the application programming interface; obtaining storage information of each compressed log file, wherein each storage information at least includes a storage path, log generation time, compression information and log tag of each compressed log file stored in the first database; establishing a query index for the M groups of compressed log files based on the storage information of each compressed log file to obtain a target query index, wherein the target query index includes the storage information of each compressed log file; and storing the target query index in the second database of the object storage system.

[0013] Furthermore, before obtaining M target log generation directories, the method also includes: when the data processing program is running, obtaining the files generated when the data processing program is running at a preset time interval to obtain Y log files, where Y is a positive integer; obtaining the job type associated with each log file, and dividing each log file into N log generation directories according to each job type, where each log generation directory includes the log file names of multiple log files of at least one job type; and storing the Y log files in containers under the N log generation directories in the computing cluster.

[0014] To achieve the above-mentioned purpose, according to another aspect of the present application, a log processing system is provided. The system includes: a log collection module for obtaining M target log generation directories in a computing cluster, extracting log file names from the M target log generation directories to obtain M groups of log file names, and obtaining M groups of log files based on the M groups of log file names, where M is a positive integer; a log archiving module for compressing the M groups of log files to obtain M groups of compressed log files, and storing the M groups of compressed log files in an object storage system; a log analysis module for receiving a log query request, extracting the log generation time and log tag of the target log file from the log query request, generating a query condition based on the log generation time and log tag, obtaining a storage path of the target log file from a query index associated with the object storage system based on the query condition, obtaining a target storage path, extracting the target log file from the object storage system based on the target storage path, and sending the target log file to a client.

[0015] To achieve the above-mentioned objectives, according to another aspect of the present application, a log processing device is provided. The device includes: a receiving unit, configured to receive a log query request and extract the log generation time and log tag of a target log file from the log query request, wherein the log tag is used to indicate the file type of the target log file; a generating unit, configured to generate a query condition based on the log generation time and the log tag, and obtain the storage path of the target log file from a query index associated with an object storage system based on the query condition to obtain the target storage path; and an extracting unit, configured to extract the target log file from the object storage system based on the target storage path and send the target log file to a client.

[0016] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned log processing methods.

[0017] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory stores an executable program, and the processor is used to run the program, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned log processing methods.

[0018] According to another aspect of an embodiment of the present invention, a computer program product is further provided. The computer program product includes a computer program, wherein when the computer program is executed by a processor, any one of the above-mentioned log processing methods is implemented.

[0019] In an embodiment of the present application, a log processing method is adopted. By receiving a log query request, the log generation time and log tag of the target log file are extracted from the log query request, wherein the log tag is used to indicate the file type of the target log file; a query condition is generated according to the log generation time and the log tag, and the storage path of the target log file is obtained from the query index associated with the object storage system according to the query condition to obtain the target storage path; the target log file is extracted from the object storage system based on the target storage path, and the target log file is sent to the client, thereby solving the technical problem of low query efficiency when obtaining the target log file from the computing cluster in the related art. By extracting the log generation time and log tag of the target log file from the log query request, generating a query condition according to the log generation time and the log tag, using the query condition to determine the target storage path, and obtaining the target log file based on the target storage path, the technical effect of improving the efficiency of querying the target log file from the computing cluster is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0021] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a log processing method;

[0022] Figure 2 This is a flowchart of a log processing method provided in accordance with an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a log processing system provided according to an embodiment of the present application;

[0024] Figure 4 This is a schematic diagram of a processing method of a log collection module provided in an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of a processing method of a log archiving module provided according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of a processing method of a log analysis module provided according to an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of an optional log processing system provided according to an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of a log processing device provided according to an embodiment of the present application;

[0029] Figure 9 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation storage for users to choose to agree or reject the automated decision results. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution; if the user chooses to refuse, the expert decision-making process will be entered. Users can view the purpose of data use in real time through authorization decoding, and have the right to withdraw authorization or delete data at any time. After withdrawing authorization, the system will terminate the relevant data processing within 24 hours.

[0033] It should be noted that the collected information used in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse use.

[0034] Example 1

[0035] According to an embodiment of the present application, an embodiment of a method for processing logs is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1It is a hardware structure diagram of a computer terminal (or mobile device) for implementing a log processing method, such as Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 The computer system 102 includes a processor 102 (the processor 102 may include but is not limited to a microcontroller unit (MCU) or a programmable logic device (FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the computer system 102 may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS), a network interface, a keyboard, a cursor control device, a power supply, and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the log processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned log processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC) and a network interface, which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] The display may be, for example, a touch screen liquid crystal display (LCD), which enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0041] Under the above operating environment, this application provides Figure 2 The log processing method shown. Figure 2 This is a flow chart of a log processing method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0042] Step S201: receiving a log query request, and extracting a log generation time and a log tag of a target log file from the log query request, wherein the log tag is used to indicate a file type of the target log file.

[0043] Specifically, a log query request refers to a request in which a user, through a graphical user interface or command line interface, provides specific query conditions to retrieve and analyze log information from a specific time period or job type in the past. Upon receiving the request, the parameters in the request can be parsed to obtain the log generation time and log tag of the target log file to be queried. The log generation time refers to the time when the log file was created or updated; the log tag is a type of metadata that indicates the file type of the target log file. During the execution of an ETL (Extract Transform Load) job, each log file represents various status, event, and error information generated during the ETL operation. Therefore, the log tag can also represent information such as the associated job unique identifier (i.e., job ID), job type (e.g., data extraction, transformation, or loading), and cluster node unique identifier. This information allows for rapid location of the corresponding log files in the object storage system, thereby determining which files meet the user-specified time range and job tag.

[0044] Step S202 : generating a query condition according to the log generation time and the log tag, and obtaining the storage path of the target log file from the query index associated with the object storage system according to the query condition to obtain the target storage path.

[0045] Specifically, after obtaining the log generation time and log tag, accurate query conditions can be generated by combining the log generation time and log tag. The query conditions can accurately retrieve the target log file in the object storage system. The object storage system is a cloud storage service or self-built storage facility used to store unstructured data, which can perform long-term storage of large-scale data.

[0046] Furthermore, as log files are archived to the object storage system, a query index is created simultaneously. The index records key information about each log file, such as the file name, storage path, generation time, and related tags, and can be stored in a relational database.

[0047] After obtaining the query conditions, the query index supporting the object storage system can be queried according to the query conditions, that is, the storage location of the target log file can be located and the target storage path of the target log file can be determined, thus avoiding linear search of massive data in the object storage system, greatly shortening the log retrieval time, and improving the system response speed and user satisfaction.

[0048] Step S203: extracting a target log file from the object storage system based on the target storage path, and sending the target log file to the client.

[0049] Specifically, after determining the target storage path representing the exact location of the archive storage, the required log files can be accurately found from the object storage system through the target storage path, and a request can be initiated to the object storage system. The latter extracts the target log files from its huge storage pool according to the request and sends them to the client to complete the data delivery.

[0050] The log processing method provided by the embodiment of the present application receives a log query request, extracts the log generation time and log tag of the target log file from the log query request, wherein the log tag is used to indicate the file type of the target log file; generates a query condition based on the log generation time and the log tag, and obtains the storage path of the target log file from the query index associated with the object storage system based on the query condition to obtain the target storage path; extracts the target log file from the object storage system based on the target storage path, and sends the target log file to the client, thereby solving the technical problem of low query efficiency when obtaining the target log file from the computing cluster in the related art. By extracting the log generation time and log tag of the target log file from the log query request, generating a query condition based on the log generation time and the log tag, using the query condition to determine the target storage path, and obtaining the target log file based on the target storage path, the technical effect of improving the efficiency of querying the target log file from the computing cluster is achieved.

[0051] Optionally, in the log processing method provided in the embodiment of the present application, before receiving the log query request, the method also includes: obtaining M target log generation directories in the computing cluster, extracting log file names from the M target log generation directories, and obtaining M groups of log file names, wherein each target log generation directory refers to an updated log generation directory in the computing cluster, and M is a positive integer; obtaining M groups of log files according to the M groups of log file names, and storing the M groups of log files in the object storage system.

[0052] Specifically, before receiving the log query request sent by the user, multiple log files need to be stored, that is, first scan all nodes in the computing cluster regularly or in real time to find those directories where log files have been generated or updated in the recent time, to ensure that the latest log information can be captured and prepare for subsequent archiving and storage. After locating multiple target log generation directories with updates, the names of all log files will be extracted from these directories to form corresponding different log file names. Each group of names corresponds to a log file under a target log generation directory, which facilitates the subsequent independent processing of the log files under each directory. By obtaining the log file names under each target directory, it is possible to identify which specific log files need to be collected and archived, rather than blindly processing all files.

[0053] Furthermore, after mastering the target log generation directory and the log file name therein, the actual content of the log file is read and obtained from the corresponding directory of the computing cluster based on this information, thereby obtaining M groups of log files, so that they can be smoothly transmitted and stored in the object storage system, realizing long-term log preservation and solving the problem of limited storage space in the computing cluster.

[0054] This embodiment comprehensively monitors the log generation situation in the computing cluster, can intelligently collect the latest log files, and store them efficiently and securely in the object storage system. This not only optimizes the utilization of storage resources and improves the efficiency of log management, but also lays a solid foundation for future maintenance and data analysis.

[0055] Optionally, in the log processing method provided in the embodiment of the present application, obtaining M target log generation directories includes: obtaining N log collection components in the computing cluster, and obtaining the log generation directory corresponding to each log collection component, to obtain N log generation directories, where N is greater than or equal to M and N is a positive integer; controlling each log collection component to detect each log generation directory in the computing cluster; if the N log generation directories include M updated log generation directories, the M updated log generation directories are determined as M target log generation directories.

[0056] Specifically, the log collection component refers to the log collection agent, which can be distributed on different nodes of the computing cluster. Each component monitors the log generation directory of one or more computing clusters to detect whether new log files are generated. When obtaining an updated log generation directory, the above-mentioned multiple log collection components can be first extracted, and each log collection component can be controlled to monitor the corresponding log generation directory to determine whether there are new log files therein, thereby achieving comprehensive coverage of the log generation directory and ensuring that log updates on any node will not be missed. It should be noted that the log generation directory monitored by each log collection component is all the directories in the computing cluster that may generate log files. These directories are pre-configured when each log collection component is deployed. Each directory contains one or more log files. When the log collection component monitors log files, it can scan the log generation directory to which it belongs according to a pre-set cycle or trigger condition to check whether there are new or updated log files.

[0057] Furthermore, after obtaining a plurality of log generation directories containing updated log files, these log generation directories can be used as target log generation directories for storing log files.

[0058] This embodiment uses the log collection component to locate the target log generation directory, which can improve the comprehensiveness and reliability of log management, avoid invalid detection of non-updated directories, thereby reducing overall resource consumption and improving efficiency.

[0059] Optionally, in the log processing method provided in the embodiment of the present application, obtaining M groups of log files according to the names of M groups of log files includes: obtaining the initial log file from the computing cluster according to the name of each log file to obtain M groups of initial log files; verifying each initial log file in each group of initial log files, and if the verification of the M groups of initial log files passes, performing field filtering on each initial log file to obtain M groups of log files.

[0060] Specifically, when obtaining log files based on the log file names stored in the target log generation directory, the entire set of original log files is first extracted, i.e., the corresponding initial log files. Each log file in each set of initial log files is then verified to check the integrity and correctness of the log files and confirm whether the log files have been completely written without damage or data loss.

[0061] Furthermore, useless or redundant information is removed from the initial log files that have passed the verification, and log fields closely related to the ETL job status analysis are retained. That is, field filtering is performed on each initial log file to obtain the final multiple groups of log files, thereby improving data purity and analysis efficiency, reducing storage space usage, and ensuring the refinement and pertinence of log data.

[0062] This embodiment ensures the accuracy and applicability of archived log data by checking and filtering fields on the initial log files, reduces analysis errors caused by data quality issues, reduces storage and transmission costs, increases the speed of data processing and analysis, and greatly improves the accuracy of problem location and troubleshooting.

[0063] Optionally, in the log processing method provided in the embodiment of the present application, storing M groups of log files in the object storage system includes: compressing the M groups of log files to obtain M groups of compressed log files; obtaining the log tag in each compressed log file to obtain M groups of log tags, wherein each log tag is used to indicate the file type of each compressed log file; obtaining the file storage type of the object storage system, classifying the M groups of log tags according to the file storage type, and obtaining a classification result; and storing the compressed log files associated with the M groups of log tags in the object storage system based on the classification result.

[0064] Specifically, after obtaining multiple newly added log files, data compression technology can be used to reduce the log file size to save storage space and transmission bandwidth. To accurately store the compressed log files, log tags containing metadata about the log file attributes can be obtained. Log tags can describe information such as the log file type, source, and timestamp, facilitating subsequent retrieval and management.

[0065] Furthermore, since different storage categories are defined in the object storage system, such as hot storage (high-frequency access), warm storage (medium-frequency access), and cold storage (low-frequency access), each category has different access speeds and storage costs. Therefore, log tags can be classified according to the file storage type to determine which category of object storage each log file can be stored in, such as hot storage, warm storage, or cold storage based on access frequency, thus obtaining the classification result. Finally, based on the above classification results, the compressed log files associated with each log tag are stored in the object storage system, ensuring the long-term preservation and ready accessibility of the data.

[0066] This embodiment reduces the storage space requirements of log files in the object storage system through compression processing, labeling and intelligent classification storage of log files, ensures the rational allocation of storage resources, reduces storage costs, and significantly improves the efficiency and cost-effectiveness of ETL job log management.

[0067] Optionally, in the log processing method provided in the embodiment of the present application, the object storage system is associated with a first database and a second database, and storing M groups of compressed log files associated with log tags in the object storage system based on the classification results includes: calling an application programming interface associated with the object storage system, and uploading M groups of compressed log files to the first database of the object storage system based on the application programming interface; obtaining storage information of each compressed log file, wherein each storage information includes at least the storage path, log generation time, compression information and log tag of each compressed log file stored in the first database; establishing a query index for the M groups of compressed log files based on the storage information of each compressed log file to obtain a target query index, wherein the target query index includes the storage information of each compressed log file; and storing the target query index in the second database of the object storage system.

[0068] Specifically, when storing the compressed log files according to the classification results, the application programming interface for data interaction between the object storage system and the external system or application can be called first, and then the compressed log files can be uploaded to the first database using the interface, where the first database refers to the database in the object storage system used to store frequently accessed or recently generated log files.

[0069] Furthermore, when archiving, it is necessary to record the metadata of each compressed log file, that is, to record the storage path, compression information, classification label, etc. of the log file, and then determine the storage information of each compressed log file based on this information. Then, based on this storage information, a query index is established for each log file, such as which log file a certain job unique identifier corresponds to, so as to obtain the target query index, and store the target query index in the relational database in the object storage system, that is, in the second database, to ensure the security and long-term availability of the index information.

[0070] This embodiment efficiently stores and manages compressed log files, meeting the need for immediate access while controlling long-term storage costs. Furthermore, the creation and maintenance of targeted query indexes enables rapid and accurate log file retrieval. Users can quickly find the required log information based on criteria such as time and log type, significantly reducing the time required to locate issues. This fully utilizes the resources of the object storage system while maintaining data orderliness and accessibility.

[0071] Optionally, in the log processing method provided in the embodiment of the present application, before obtaining M target log generation directories, the method also includes: when the data processing program is running, obtaining the files generated when the data processing program is running at a preset time interval to obtain Y log files, where Y is a positive integer; obtaining the job type associated with each log file, and dividing each log file into N log generation directories according to each job type, where each log generation directory includes the log file names of multiple log files of at least one job type; and storing the Y log files in containers under the N log generation directories in the computing cluster.

[0072] Specifically, a data processing program is a program or script that performs data processing tasks such as data extraction, transformation, and loading (ETL). When an ETL job runs, it transfers and processes data between different data sources. At this time, a timer or polling mechanism can generate various status, event, and error information files, also known as log files. To store these log files in the computing cluster, it is also necessary to obtain the job type of each log file. For example, the job type can be extraction, transformation, or loading, and each type corresponds to different log file recording standards.

[0073] Furthermore, different log files are recorded in different log generation directories based on the job type. Specifically, the log file name of each log file is stored in the log generation directory. This not only facilitates categorized log storage but also facilitates subsequent rapid retrieval and analysis based on job type. Furthermore, these log files are stored in containers within different log generation directories within the computing cluster to avoid storage conflicts.

[0074] This embodiment ensures the immediate generation and acquisition of log files by collecting log files in real time and classifying log files by job type, thereby improving the problem detection and response speed. In addition, based on the directory division of job type, it realizes the refined classification of log files, which facilitates rapid positioning and analysis according to different data processing requirements.

[0075] The embodiment of the present application further provides a log processing system, which can be used to execute the log processing method provided in the embodiment of the present application. The log processing system provided in the embodiment of the present application is introduced below.

[0076] Figure 3 is a schematic diagram of a log processing system provided according to an embodiment of the present application, such as Figure 3 As shown, the system includes:

[0077] The log collection module 301 is configured to obtain M target log generation directories in the computing cluster, extract log file names from the M target log generation directories to obtain M groups of log file names, and obtain M groups of log files based on the M groups of log file names, where M is a positive integer.

[0078] Specifically, Figure 4 is a schematic diagram of a processing method of a log collection module according to an embodiment of the present application, such as Figure 4 As shown, the log collection module 301 can monitor and obtain the ETL job logs generated in the cluster in real time through the log collection component Agent of the computing cluster, thereby identifying multiple target log generation directories with updates in the computing cluster, and then extracting the log file names of the updated log files in each target log generation directory, thereby obtaining multiple groups of log file names, wherein each group of log file names represents the log file name of a log file under a target log generation directory, for example, log file name A and log file name B are obtained.

[0079] Furthermore, based on the above log file names, the log collection module 301 extracts specific log files from the corresponding directory in the computing cluster to form multiple log files, and then sends these log files and log file names to the log archiving module 302 for storage.

[0080] The log archiving module 302 is configured to compress the M groups of log files to obtain the M groups of compressed log files, and store the M groups of compressed log files in the object storage system;

[0081] Specifically, Figure 5 Schematic diagram of a processing method of a log archiving module according to an embodiment of the present application. Figure 5As shown, after the log archiving module 302 receives multiple groups of log files and corresponding log file names sent by the log collection module 301, it can obtain the metadata of the log files when they are archived to the computing cluster, that is, record the storage path, compression information, classification labels, etc. of the log files, and generate log labels based on the log file names and the metadata of the above log files, and use data compression technology to reduce the volume of the log files to obtain compressed log files, so as to save storage space and transmission bandwidth.

[0082] Furthermore, an application programming interface of the object storage system is called, and the compressed log files are uploaded to the first database of the object storage system using the interface. Storage information of each compressed log file is determined based on this information, and a target query index is established for each log file using the storage information. The target query index is stored in the relational database of the object storage system, that is, in the second database, to ensure the security and long-term availability of the index information.

[0083] The log analysis module 303 is used to receive a log query request, extract the log generation time and log tag of the target log file from the log query request, generate query conditions based on the log generation time and log tag, obtain the storage path of the target log file from the query index associated with the object storage system based on the query conditions, obtain the target storage path, extract the target log file from the object storage system based on the target storage path, and send the target log file to the client.

[0084] Specifically, Figure 6 is a schematic diagram of a processing method of a log analysis module provided in an embodiment of the present application, such as Figure 6 As shown, when the log analysis module 303 receives a log query request representing a user query requirement, it can parse the parameters in the request to obtain the log generation time (i.e., time period) and log tag (i.e., job identifier (job ID)) of the target log file to be queried, and then generate precise query conditions based on the log generation time and log tag to guide the log analysis module 303 to find the target log file in the object storage system.

[0085] Furthermore, the relational database associated with the object storage system is obtained, and the storage path of the target log file is obtained from the query index associated with the relational database using the query condition to obtain the target storage path. The target storage path is then used to accurately find the required target log file from the object storage system, and a request is initiated to the object storage system. The latter extracts the target log file from its huge storage pool according to the request, decompresses the log file, loads its content, and sends it to the client to complete the data delivery.

[0086] The log processing system provided by the embodiment of the present application is used to obtain M target log generation directories in the computing cluster through the log collection module 301, extract the log file names from the M target log generation directories, obtain M groups of log file names, and obtain M groups of log files according to the M groups of log file names, where M is a positive integer; the log archiving module 302 is used to compress the M groups of log files to obtain M groups of compressed log files, and store the M groups of compressed log files in the object storage system; the log analysis module 303 is used to receive a log query request, extract the log generation time and log tag of the target log file from the log query request, and generate a query condition based on the log generation time and the log tag, and obtain the target log from the query index associated with the object storage system according to the query condition. The storage path of the file is obtained, and the target storage path is obtained. It is also used to extract the target log file from the object storage system based on the target storage path, and send the target log file to the client, which solves the technical problem of low query efficiency when obtaining the target log file from the computing cluster in the related technology. Through the coordinated operation of the log collection module 301, the log archiving module 302 and the log analysis module 303, the log file is stored in the object storage system, and after receiving the log query request, the log generation time and log label of the target log file are extracted from the log query request, and the query condition is generated according to the log generation time and the log label. The query condition is used to determine the target storage path, and the target log file is obtained based on the target storage path, thereby achieving the technical effect of improving the efficiency of querying the target log file from the computing cluster.

[0087] The present application also provides a log processing system. Figure 7 is a schematic diagram of an optional log processing system provided according to an embodiment of the present application, such as Figure 7 As shown, the system includes:

[0088] First, the log collection component Agent in the log collection module monitors and retrieves the ETL job logs generated in the cluster in real time, identifying multiple target log generation directories within the compute cluster with updates. It then extracts the log file names of the updated log files within each target log generation directory, generating multiple sets of log file names. Based on these log file names, the log collection module extracts specific log files from the corresponding directories within the compute cluster, generating multiple log files. These log files and their names are then sent to the log archiving module for storage.

[0089] Furthermore, after receiving multiple groups of log files and corresponding log file names sent by the log collection module, the log archiving module can obtain the metadata of the log files when they are archived to the computing cluster, that is, record the storage path, compression information, classification tags, etc. of the log files, and generate log tags based on the log file names and the metadata of the above log files, and use data compression technology to reduce the volume of the log files to obtain compressed log files, thereby saving storage space and transmission bandwidth. Then, the application programming interface of the object storage system is called, and the compressed log files are uploaded to the first database of the object storage system using the interface, and the storage information of each compressed log file is determined based on this information, and a target query index is established for each log file using the storage information, and the target query index is stored in the relational database of the object storage system, that is, in the second database, to ensure the security and long-term availability of the index information.

[0090] When the log analysis module receives a log query request representing a user's query needs, it can parse the parameters in the request to obtain the log generation time (i.e., time period) and log tag (i.e., job identifier (job ID)) of the target log file to be queried. Then, based on the log generation time and log tag, it generates precise query conditions to guide the log analysis module to find the target log file in the object storage system.

[0091] Furthermore, the relational database associated with the object storage system is obtained, and the storage path of the target log file is obtained from the query index associated with the relational database using the query condition to obtain the target storage path. The target storage path is then used to accurately find the required target log file from the object storage system, and a request is initiated to the object storage system. The latter extracts the target log file from its huge storage pool according to the request, decompresses the log file, loads its content, and sends it to the client to complete the data delivery.

[0092] This embodiment ensures the timeliness of log data through real-time collection, efficient cloud storage, and convenient analysis of ETL job logs, avoids the omission of important information, reduces storage costs, significantly improves query efficiency, and facilitates operation and maintenance personnel to quickly find problems in massive logs.

[0093] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0094] Example 2

[0095] The present application also provides a log processing device. It should be noted that the log processing device of the present application can be used to execute the log processing method provided in the present application. The log processing device provided in the present application is introduced below.

[0096] According to an embodiment of the present application, a device for implementing the above-mentioned log processing method is also provided. Figure 8 is a schematic diagram of a log processing device provided according to an embodiment of the present application, such as Figure 8 As shown, the device includes: a receiving unit 80, a generating unit 81, and an extracting unit 82.

[0097] The receiving unit 80 is configured to receive a log query request and extract a log generation time and a log tag of a target log file from the log query request, wherein the log tag is used to indicate a file type of the target log file;

[0098] The generating unit 81 is configured to generate a query condition based on the log generation time and the log tag, and obtain the storage path of the target log file from the query index associated with the object storage system according to the query condition to obtain the target storage path;

[0099] The extraction unit 82 is configured to extract a target log file from the object storage system based on the target storage path, and send the target log file to the client.

[0100] The log processing device provided by the embodiment of the present application receives a log query request through a receiving unit 80, extracts the log generation time and log tag of the target log file from the log query request, wherein the log tag is used to indicate the file type of the target log file; the generating unit 81 generates a query condition based on the log generation time and the log tag, and obtains the storage path of the target log file from the query index associated with the object storage system according to the query condition to obtain the target storage path; the extracting unit 82 extracts the target log file from the object storage system based on the target storage path, and sends the target log file to the client, thereby solving the technical problem of low query efficiency when obtaining the target log file from the computing cluster in the related art. By extracting the log generation time and log tag of the target log file from the log query request, generating a query condition based on the log generation time and the log tag, determining the target storage path by using the query condition, and obtaining the target log file based on the target storage path, the technical effect of improving the efficiency of querying the target log file from the computing cluster is achieved.

[0101] Optionally, in the log processing device provided in the embodiment of the present application, the device also includes: a first acquisition unit, used to obtain M target log generation directories in the computing cluster before receiving a log query request, extract log file names from the M target log generation directories, and obtain M groups of log file names, wherein each target log generation directory refers to an updated log generation directory in the computing cluster, and M is a positive integer; a second acquisition unit, used to obtain M groups of log files according to the M groups of log file names, and store the M groups of log files in the object storage system.

[0102] Optionally, in the log processing device provided in the embodiment of the present application, the device also includes: a first acquisition module, used to obtain N log collection components in the computing cluster, and obtain the log generation directory corresponding to each log collection component to obtain N log generation directories, where N is greater than or equal to M and N is a positive integer; a control module, used to control each log collection component to detect each log generation directory in the computing cluster; a determination module, used to determine the M log generation directories with updates as M target log generation directories if the N log generation directories include M log generation directories with updates.

[0103] Optionally, in the log processing device provided in the embodiment of the present application, the device also includes: a second acquisition module, used to obtain the initial log file from the computing cluster according to the name of each log file, to obtain M groups of initial log files; a verification module, used to verify each initial log file in each group of initial log files, if the M groups of initial log files pass the verification, field filtering is performed on each initial log file to obtain M groups of log files.

[0104] Optionally, in the log processing device provided in the embodiment of the present application, the device also includes: a processing module, used to compress M groups of log files to obtain M groups of compressed log files; a third acquisition module, used to obtain the log tag in each compressed log file to obtain M groups of log tags, wherein each log tag is used to indicate the file type of each compressed log file; a fourth acquisition module, used to obtain the file storage type of the object storage system, classify the M groups of log tags according to the file storage type, and obtain a classification result; a first storage module, used to store the compressed log files associated with the M groups of log tags to the object storage system based on the classification result.

[0105] Optionally, in the log processing device provided in the embodiment of the present application, the device also includes: a calling module, used to call the application programming interface associated with the object storage system, and upload M groups of compressed log files to the first database of the object storage system based on the application programming interface; a fifth acquisition module, used to obtain the storage information of each compressed log file, wherein each storage information includes at least the storage path, log generation time, compression information and log label of each compressed log file stored in the first database; an establishment module, used to establish a query index of M groups of compressed log files according to the storage information of each compressed log file, and obtain a target query index, wherein the target query index includes the storage information of each compressed log file; a second storage module, used to store the target query index to the second database of the object storage system.

[0106] Optionally, in the log processing device provided in the embodiment of the present application, the device also includes: a third acquisition unit, used to obtain the files generated when the data processing program is running at a preset time interval before obtaining M target log generation directories, and obtain Y log files, where Y is a positive integer; a fourth acquisition unit, used to obtain the job type associated with each log file, and divide each log file into N log generation directories according to each job type, where each log generation directory includes the log file names of multiple log files of at least one job type; a storage unit, used to store the Y log files in containers under the N log generation directories in the computing cluster.

[0107] It should be noted that the receiving unit 80, generating unit 81, and extracting unit 82 described above correspond to steps S201 to S203 in Example 1. The examples and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above units can also be part of a device and can be run in the computer terminal 10 provided in Example 1.

[0108] Example 3

[0109] The embodiment of the present application can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal or an electronic device.

[0110] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0111] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the log processing method: receiving a log query request, extracting the log generation time and log tag of the target log file from the log query request, wherein the log tag is used to indicate the file type of the target log file; generating a query condition based on the log generation time and the log tag, and obtaining the storage path of the target log file from the query index associated with the object storage system based on the query condition to obtain the target storage path; extracting the target log file from the object storage system based on the target storage path, and sending the target log file to the client.

[0112] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the log processing method: obtain M target log generation directories in the computing cluster, extract log file names from the M target log generation directories, and obtain M groups of log file names, wherein each target log generation directory refers to an updated log generation directory in the computing cluster, and M is a positive integer; obtain M groups of log files according to the M groups of log file names, and store the M groups of log files in the object storage system.

[0113] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the log processing method: obtain N log collection components in the computing cluster, and obtain the log generation directory corresponding to each log collection component to obtain N log generation directories, where N is greater than or equal to M and N is a positive integer; control each log collection component to detect each log generation directory in the computing cluster; if the N log generation directories include M updated log generation directories, determine the M updated log generation directories as M target log generation directories.

[0114] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the log processing method: obtaining the initial log file from the computing cluster according to the name of each log file to obtain M groups of initial log files; verifying each initial log file in each group of initial log files, and if the M groups of initial log files pass the verification, performing field filtering on each initial log file to obtain M groups of log files.

[0115] Optionally, the computer terminal may execute the program code of the following steps in the log processing method: compressing M groups of log files to obtain M groups of compressed log files; obtaining a log tag in each compressed log file to obtain M groups of log tags, wherein each log tag is used to indicate the file type of each compressed log file; obtaining the file storage type of the object storage system, classifying the M groups of log tags according to the file storage type, and obtaining a classification result; and storing the compressed log files associated with the M groups of log tags in the object storage system based on the classification result.

[0116] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the log processing method: calling the application programming interface associated with the object storage system, and uploading M groups of compressed log files to the first database of the object storage system based on the application programming interface; obtaining the storage information of each compressed log file, wherein each storage information at least includes the storage path, log generation time, compression information and log tag of each compressed log file stored in the first database; establishing a query index of M groups of compressed log files based on the storage information of each compressed log file to obtain a target query index, wherein the target query index includes the storage information of each compressed log file; and storing the target query index to the second database of the object storage system.

[0117] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the log processing method: when the data processing program is running, obtain the files generated when the data processing program is running at a preset time interval to obtain Y log files, where Y is a positive integer; obtain the job type associated with each log file, and divide each log file into N log generation directories according to each job type, where each log generation directory includes the log file names of multiple log files of at least one job type; store the Y log files in containers under the N log generation directories in the computing cluster.

[0118] Optionally, Figure 9 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 9 As shown, the electronic device may include: one or more ( Figure 9 Only one is shown) processor 902, memory 904, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0119] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the log processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned log processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0120] The processor may call the information and application programs stored in the memory through the transmission device to execute the above steps in the above log processing method.

[0121] An embodiment of the present application provides a log processing solution. By receiving a log query request, extracting the log generation time and log tag of a target log file from the log query request, wherein the log tag is used to indicate the file type of the target log file; generating a query condition based on the log generation time and the log tag, and obtaining the storage path of the target log file from a query index associated with an object storage system based on the query condition to obtain the target storage path; extracting the target log file from the object storage system based on the target storage path, and sending the target log file to a client, the method solves the technical problem of low query efficiency when obtaining target log files from a computing cluster in the related art. By extracting the log generation time and log tag of the target log file from the log query request, generating a query condition based on the log generation time and the log tag, determining the target storage path using the query condition, and obtaining the target log file based on the target storage path, the method achieves the technical effect of improving the efficiency of querying target log files from a computing cluster.

[0122] It can be understood by those skilled in the art that Figure 9 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), or a PAD. Figure 9 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 9 Different configurations shown.

[0123] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0124] Example 4

[0125] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the log processing method provided in the first embodiment.

[0126] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0127] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: receiving a log query request, extracting the log generation time and log tag of the target log file from the log query request, wherein the log tag is used to indicate the file type of the target log file; generating a query condition based on the log generation time and the log tag, and obtaining the storage path of the target log file from the query index associated with the object storage system based on the query condition to obtain the target storage path; extracting the target log file from the object storage system based on the target storage path, and sending the target log file to the client.

[0128] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the log processing method.

[0129] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0130] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0132] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0135] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A log processing method, characterized in that: include: receiving a log query request, and extracting a log generation time and a log tag of a target log file from the log query request, wherein the log tag is used to indicate a file type of the target log file; Generate a query condition according to the log generation time and the log tag, and obtain the storage path of the target log file from the query index associated with the object storage system according to the query condition to obtain the target storage path; The target log file is extracted from the object storage system based on the target storage path, and the target log file is sent to the client.

2. The method according to claim 1, characterized in that Before receiving the log query request, the method further includes: Obtain M target log generation directories in the computing cluster, extract log file names from the M target log generation directories, and obtain M groups of log file names, where each target log generation directory refers to a log generation directory with an update in the computing cluster, and M is a positive integer; Acquire M groups of log files according to the names of the M groups of log files, and store the M groups of log files in the object storage system.

3. The method according to claim 2, characterized in that Obtaining M target log generation directories includes: Obtain N log collection components in the computing cluster, and obtain the log generation directory corresponding to each log collection component to obtain N log generation directories, where N is greater than or equal to M and is a positive integer; Controlling each log collection component to detect each log generation directory in the computing cluster; If the N log generation directories include M updated log generation directories, the M updated log generation directories are determined as the M target log generation directories.

4. The method according to claim 2, characterized in that Acquiring M groups of log files according to the names of the M groups of log files includes: Obtaining an initial log file from the computing cluster according to the name of each log file to obtain M groups of initial log files; Each initial log file in each group of initial log files is verified. If the M groups of initial log files pass the verification, field filtering is performed on each initial log file to obtain the M groups of log files.

5. The method according to claim 2, characterized in that Storing the M groups of log files in the object storage system includes: Compressing the M groups of log files to obtain M groups of compressed log files; Obtain a log tag in each compressed log file to obtain M groups of log tags, where each log tag is used to indicate the file type of each compressed log file; Obtaining a file storage type of the object storage system, and classifying the M groups of log tags according to the file storage type to obtain a classification result; Based on the classification result, the compressed log files associated with the M groups of log tags are stored in the object storage system.

6. The method according to claim 5, characterized in that The object storage system is associated with a first database and a second database, and storing the compressed log files associated with the M groups of log tags in the object storage system based on the classification result includes: calling an application programming interface associated with the object storage system, and uploading the M groups of compressed log files to the first database of the object storage system based on the application programming interface; Obtaining storage information of each compressed log file, wherein each storage information includes at least a storage path of each compressed log file stored in the first database, log generation time, compression information, and a log tag; Establishing a query index for the M groups of compressed log files according to the storage information of each compressed log file to obtain a target query index, wherein the target query index includes the storage information of each compressed log file; The target query index is stored in the second database of the object storage system.

7. The method according to claim 2, characterized in that Before obtaining the M target log generation directories, the method further includes: When the data processing program is running, files generated by the data processing program are obtained at preset time intervals to obtain Y log files, where Y is a positive integer; Obtaining a job type associated with each log file, and dividing each log file into N log generation directories according to each job type, wherein each log generation directory includes log file names of multiple log files of at least one job type; The Y log files are stored in containers under the N log generation directories in the computing cluster.

8. A log processing system, characterized in that: include: a log collection module, configured to obtain M target log generation directories in the computing cluster, extract log file names from the M target log generation directories to obtain M groups of log file names, and obtain M groups of log files based on the M groups of log file names, where M is a positive integer; a log archiving module, configured to compress the M groups of log files to obtain M groups of compressed log files, and store the M groups of compressed log files in an object storage system; The log analysis module is used to receive a log query request, extract the log generation time and log tag of the target log file from the log query request, generate query conditions based on the log generation time and the log tag, obtain the storage path of the target log file from the query index associated with the object storage system based on the query conditions, obtain the target storage path, extract the target log file from the object storage system based on the target storage path, and send the target log file to the client.

9. An electronic device, characterized in that: include: a memory storing an executable program; A processor is used to run the program, wherein the program executes the log processing method according to any one of claims 1 to 7 when running.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the log processing method according to any one of claims 1 to 7 are implemented.

Images