Data processing method and device, computer equipment and computer readable storage medium

Abstract

The application provides a data processing method and device, computer equipment and a computer readable storage medium, which can be applied to cloud technology, artificial intelligence, wisdom platform, log analysis and various scenes, wherein the method comprises the following steps: obtaining a to-be-processed task, the to-be-processed task comprising a business server identifier; sending the to-be-processed task and a log processing program corresponding to the to-be-processed task to a business server corresponding to the business server identifier, the log processing program being used for executing the to-be-processed task according to log data stored in the business server; receiving a processing result sent by the log processing program on the business server after the to-be-processed task is executed, outputting the processing result, and sending a destruction instruction to the log processing program on the business server, the destruction instruction being used for destroying the log processing program on the business server. The application can avoid centralized collection of log data, improve log retrieval efficiency, and reduce data storage cost.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data processing method, a data processing apparatus, a computer device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development and application of computer technology, more and more scenarios require the use of data processing techniques for data collection, such as retrieving log data for log analysis. Currently, a manual retrieval method is used, where operators periodically copy all log files from the business server. Upon receiving a log retrieval request, all copied log data is manually retrieved and stored uniformly. This method is inefficient in log retrieval and has high data storage costs. Therefore, how to improve the efficiency of log retrieval and reduce data storage costs has become a current research hotspot. Summary of the Invention

[0003] This application provides a data processing method, apparatus, computer equipment, and computer-readable storage medium, which can improve the efficiency of log retrieval and reduce labor costs and data storage costs.

[0004] Firstly, this application provides a data processing method, which includes: Obtain tasks to be processed, including business server identifiers. The above-mentioned pending tasks and the corresponding log processing programs are sent to the business server corresponding to the above-mentioned business server identifier. The above-mentioned log processing programs are used to execute the above-mentioned pending tasks based on the log data stored in the above-mentioned business server. The system receives the processing result of the pending task after its completion from the log processing program on the aforementioned business server, outputs the processing result, and sends a destruction command to the log processing program on the aforementioned business server. The destruction command is used to destroy the log processing program on the aforementioned business server.

[0005] Secondly, this application provides a data processing apparatus, the apparatus comprising: The acquisition module is used to acquire tasks to be processed, including the business server identifier. The sending module is used to send the above-mentioned task to be processed and the log processing program corresponding to the above-mentioned task to the business server corresponding to the above-mentioned business server identifier. The log processing program is used to execute the above-mentioned task to be processed based on the log data stored in the above-mentioned business server. The processing module is used to receive the processing result of the pending task after it has been completed, sent by the log processing program on the business server, output the processing result, and send a destruction command to the log processing program on the business server. The destruction command is used to destroy the log processing program on the business server.

[0006] Thirdly, this application provides a computer device, including: a memory and a processor, wherein the memory stores a data processing program, and when the data processing program is executed by the processor, it is used to implement the data processing method described above.

[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions that are executed by a processor to implement the data processing method described above.

[0008] Fifthly, this application provides a computer program product, which includes a computer program or computer instructions, which are executed by a processor to implement the data processing method described above.

[0009] This application automates log processing by sending pending tasks to the business server and a log processing program capable of executing those tasks. After the log processing program completes the task, the application receives the processing result from the log processing program on the business server. Compared to manual data collection, this application automates log processing through maintenance equipment, avoiding manual log data collection, improving log retrieval efficiency, and reducing storage costs by storing only the processing result on the maintenance equipment instead of the full log data. When the completion of a pending task is detected, a destruction command can be sent to the log processing program on the business server to destroy the log processing program, improving the resource utilization of the business server. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating a data processing method provided in an exemplary embodiment of this application; Figure 3AThis is a schematic diagram of the interface of a log retrieval and log analysis system provided in an exemplary embodiment of this application; Figure 3B This is a schematic diagram of the interface of another log retrieval and log analysis system provided in an exemplary embodiment of this application; Figure 3C This is a schematic diagram of the interface of another log retrieval and log analysis system provided in an exemplary embodiment of this application; Figure 4 This is a flowchart illustrating another data processing method provided in an exemplary embodiment of this application; Figure 5A This is a schematic diagram of the architecture of a log retrieval and log analysis system provided in an exemplary embodiment of this application; Figure 5B This is a timing diagram of a log retrieval and log analysis system provided in an exemplary embodiment of this application; Figure 6 This is a schematic block diagram of a data processing apparatus provided in an exemplary embodiment of this application; Figure 7 This is a schematic block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0013] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.

[0014] Cloud computing refers to the delivery and usage model of Internet Technology (IT) infrastructure, meaning obtaining necessary resources through a network in an on-demand and easily scalable manner. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services through a network in an on-demand and easily scalable manner. These services can be IT and software related, Internet-related, or other services. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the development of the Internet, real-time data streams, the diversification of connected devices, and the demands of retrieval services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel and distributed computing, the emergence of cloud computing will fundamentally revolutionize the entire Internet model and enterprise management model. The solutions provided in this application involve the above-mentioned cloud computing technologies, using cloud computing technology for log retrieval and analysis. This application is applicable to log retrieval and analysis in scenarios such as public cloud, private cloud, and hybrid cloud, and can avoid the costs of human deployment and resource storage. The following will describe private cloud, public cloud, and hybrid cloud.

[0015] A private cloud is created within a firewall, housing cloud infrastructure and hardware / software resources for various departments within an organization or enterprise to share resources within a data center. Besides hardware resources, creating a private cloud typically involves cloud equipment (IaaS, Infrastructure as a Service) software. Private cloud computing also comprises three layers: cloud hardware, cloud platform, and cloud services. The difference is that cloud hardware consists of the user's own personal computer or server, rather than the cloud computing vendor's data center. Cloud computing vendors build data centers to provide public cloud services to millions of users, thus requiring hundreds of thousands or even millions of servers. For individuals, private cloud computing serves only family and friends; for enterprises, it serves only their own employees, customers, and suppliers. Therefore, the personal computers or servers of individuals or enterprises are sufficient to provide cloud services.

[0016] Public cloud typically refers to cloud services provided by third-party providers to users. Public clouds are generally accessible via the Internet and may be free or inexpensive. The core attribute of a public cloud is shared resource services. Many instances of this type of cloud exist, providing services across today's open public networks.

[0017] Hybrid cloud, which combines public cloud and private cloud, has become a major model and development direction of cloud computing in recent years. Private cloud primarily targets enterprise users; for security reasons, enterprises prefer to store data in private clouds, but at the same time, they also want access to the computing resources of public clouds. Hybrid cloud is increasingly adopted in this context, combining and matching public and private clouds to achieve optimal results. This personalized solution achieves both cost-effectiveness and security.

[0018] Currently, log retrieval is performed manually. This involves the data retriever accessing log files on different business servers (which can represent a user's host) and storing the data uniformly. This method is inefficient. Furthermore, if log data analysis is required, the large volume of retrieved log data needs to be categorized and stored, resulting in high labor and data storage costs for manual log retrieval.

[0019] To address the shortcomings of the aforementioned log retrieval methods, this application first proposes a method for deploying a robot (agent) for log retrieval. Specifically, this method consists of an agent-side and a server-side. On the agent-side, data generation and scheduled reporting are performed. In the data generation phase, by setting collection anchor points, event logs are automatically generated according to a specified format when trigger conditions are met. In the scheduled reporting phase, the agent, considering the server's network conditions, compresses a certain amount of data before periodically reporting it. On the server-side, log analysis, log storage, and intelligent learning are performed. In the log analysis phase, upon receiving real-time data, the server quickly performs single-log analysis; if problems are found, it reports an intrusion to the system and issues a separate alarm. In the log storage phase, the data is sent to an embedded storage system (ES) for storage, and the relevant real-time reporting interface also directly reports new data to the relevant platform. In the intelligent learning phase, the system can periodically learn the day's logs and generate relevant analysis reports through multi-log correlation analysis.

[0020] The above method enables unified storage and intelligent analysis of logs. However, it requires pre-configuration and deployment of agents, resulting in high resource consumption, high development and maintenance costs, and high complexity. Furthermore, applying this method to a private cloud environment without on-site engineers and automated deployment poses a risk of infeasibility, further increasing maintenance costs.

[0021] The log retrieval method described above, which is based on the deployment of robots (agents), is a non-demand-driven approach. That is, the agent generates data and reports it on a regular basis. During the log collection phase, all log data is stored in the database, and during the log analysis phase, the required log data is retrieved from the database for log analysis.

[0022] Based on the above, this application optimizes and improves it, proposing a demand-driven method, which obtains target log data according to the log collection task, processes the target log data directly, and generates processing results, thus avoiding the full collection and storage of logs.

[0023] This application proposes a log retrieval and analysis system, including: business server groups, log topics, and orchestration rules, and provides log interface display capabilities. The business server groups describe the business server source, IP address, username, password, and other business server information for log retrieval. Log topics define attributes such as log paths under these business server groups. Orchestration rules provide cascading analysis and filtering capabilities for logs, enabling correlation between logs. Based on the technical solution provided in this application, remote program distribution and execution are achieved through the Secure Shell (SSH) protocol, eliminating the need for log collection probes, log collection agents, and unified log storage and analysis. This reduces deployment and resource storage costs, and can be applied to various scenarios such as distributed systems and private cloud environments for rapid troubleshooting. It is suitable for log retrieval and analysis in large-scale distributed systems and scenarios with diverse log types.

[0024] It is understood that in the specific embodiments of this application, log data and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, the collection, use and processing of related data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0025] This application will be specifically described through the following embodiments.

[0026] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an exemplary embodiment of this application. For example... Figure 1As shown, the data processing system may specifically include a terminal device 101, a first server 102, and a second server 103. The terminal device 101, the first server 102, and the second server 103 are connected via a network, such as a wireless network. The first server 102 may refer to the object that configures the tasks to be processed and the log processing program in this application, such as the backend (master end); the terminal device 101 may refer to the object that the first server 102 obtains configuration parameters in this application, such as the frontend (web end); and the second server 103 may refer to the object that receives the tasks to be processed and the log processing program in this application, such as the business server (host end). Based on the data processing method proposed in this application, terminal device 101 can obtain configuration parameters and request retrieval from first server 102. First server 102 obtains configuration parameters from terminal device 101 and creates a log data retrieval task, and sends the task ID corresponding to the retrieval task to terminal device 101 so that terminal device 101 can query progress information, etc. First server 102 sends the log analysis and processing program to second server 103 to execute the retrieval task. The log analysis and processing program on second server 103 returns the real-time retrieval progress (i.e., progress information) or retrieval results to first server 102. Terminal device 101 can query the current progress of task ID through task ID. That is, when first server 102 receives a query request from terminal device 101 for task ID, it displays the data on the display interface. In addition, first server 102 can also perform management operations on retrieval progress and log data.

[0027] In one embodiment, terminal device 101 may be a device for operating objects to edit data and perform retrieval tasks; first server 102 may be a server for providing interface services, data interaction, data persistence processing and data management; terminal device 103 may be a business server group that receives log analysis and log processing programs (the business server group may be one business server or multiple business servers).

[0028] Terminal device 101 (or terminal device 103) is also referred to as terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication device, user agent, or user device. Terminal device can be a smart home appliance, a handheld device with wireless communication capabilities (such as a smartphone or tablet), a computing device (such as a personal computer (PC), an in-vehicle terminal, a smart voice interaction device, a wearable device, or other smart device, but is not limited to these).

[0029] The first server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0030] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. For example, in this application, the terminal device 101 includes, in addition to, Figure 1 In addition to the three devices shown, it may also include more than three devices; similarly, the first server 102 may include, in addition to, Figure 1 In addition to the single server shown, the system can also consist of multiple servers (i.e., a server cluster). Terminal device 101 and terminal device 103 can be the same terminal device (i.e., the architecture of the entire data processing system includes terminal device 101 and the first server 102). That is, data configuration is performed by terminal device 101, and the first server 102 sends a log analysis and processing program to terminal device 101 based on the received configuration data, where the retrieval task is executed. Alternatively, data configuration can be performed on the first server 102, and the log analysis and processing program can be sent to terminal device 101 based on the received configuration data, where the retrieval task is executed. As those skilled in the art will recognize, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0031] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an exemplary embodiment of this application. Taking the application of this method to an operation and maintenance device (the terminal device 101 and server 102 mentioned above can be regarded as a whole as the execution subject of this application embodiment, i.e., the operation and maintenance device, which will be described in the following text) as an example, the method may include the following steps: S201. Obtain the task to be processed, which includes the business server identifier.

[0032] In this embodiment, the task to be processed is a log data operation task to be performed, including a log data retrieval task and a log data analysis task. The task to be processed includes a business server identifier. The business server is the execution object of the task to be processed; that is, the task to be processed is executed on the log data stored in the business server. The business server identifier can distinguish different business servers, and the task to be processed is executed based on the business server address corresponding to the business server identifier. There can be one or more business server identifiers.

[0033] In one embodiment, the task to be processed is generated based on the received configuration data, i.e., the configuration data is pre-set. When the task to be processed is a log data retrieval task, the pre-setting allows for the targeted definition of information such as the business server to be processed, the log path in the business server, and the execution time, thereby improving the accuracy and efficiency of log retrieval. When the task to be processed is a log data analysis task, the pre-setting allows for the targeted definition of the device information to be analyzed in the logs of the business server to be processed. The device information to be analyzed can be multiple, thereby improving the accuracy and efficiency of log analysis.

[0034] In one embodiment, when the task to be processed is a log data retrieval task, the task to be processed includes the target log storage path. Based on this, the above method for obtaining the task to be processed can be implemented according to the following steps.

[0035] (1) Display the search interface, which includes a topic search control.

[0036] In one embodiment, the search interface can be a web interface, which can display multiple controls, including topic search controls, orchestration search controls, and management controls. The management controls may include business server configuration controls, log topic configuration controls, and orchestration configuration controls. This embodiment first introduces the topic search controls and orchestration search controls. The management controls, business server configuration controls, log topic configuration controls, and orchestration configuration controls will be introduced in subsequent embodiments and will not be repeated in this embodiment.

[0037] (2) When the topic search control is triggered, one or more log topics are displayed on the search interface.

[0038] When the topic search control is triggered, i.e., a trigger operation is received for the topic search control, the maintenance device displays one or more log topics. The method for displaying one or more log topics can be various, such as generating an option floating window, generating an option list, or generating a new interface. Displaying one or more log topics in these ways is used to detect operator trigger operations on one or more log topics. After detecting an operator trigger operation on one or more log topics, the maintenance device can use the log topic corresponding to the operator's trigger operation as the target log topic. The one or more log topics can be pre-configured; the configuration method will be described in detail in subsequent embodiments and will not be repeated in this embodiment.

[0039] (3) When the target log topic is triggered, a log data retrieval task is generated based on the target log topic. The target log topic includes the business server identifier and the target log storage path.

[0040] In this embodiment, a log topic can be understood as a set of associated data, each set of associated data including a log topic name, a business server identifier, and multiple log storage paths. Log files are stored under each log storage path. A log data retrieval task is generated based on the target log topic. The essence of this log data retrieval task is to perform a log retrieval on the log files under the multiple log storage paths corresponding to the business server identifier of the target log topic. The target log topic is one of one or more log topics displayed on the retrieval interface.

[0041] Please see Figure 3A , Figure 3A This is a schematic diagram of a search interface provided in an embodiment of this application. The diagram includes multiple controls (e.g., Figure 3A The 301 section includes a subject search control (i.e., Figure 3A The controls include those for "Topic Search," orchestration search controls, and management controls. The management controls include log topic controls, business server group controls, and orchestration rule controls. When the topic search control is triggered (displayed as an arrow pointing to the topic search control in a 301 redirect), one or more log topics (such as...) are displayed on the search interface. Figure 3A As shown in section 302, section 302 includes an expandable list of options, which can be triggered to display one or more log topics. When the target log topic is triggered (e.g., Figure 3A As shown in 302, when the log topic yunapi-access is triggered (displayed in 302), a log data retrieval task can be generated based on the target log topic.

[0042] Figure 3AThe parts 303 to 305 will be described in subsequent embodiments, and will not be repeated in this embodiment.

[0043] In one embodiment, the retrieval interface further includes an execution order setting control. When there are multiple target log topics, the above-described method for generating log data retrieval tasks based on target log topics can be implemented according to the following steps.

[0044] (1) When the execution order setting control is triggered, the target execution order is determined according to the execution order setting control.

[0045] In this embodiment, the execution order setting control is used by the operation and maintenance device to set the execution order of multiple target log topics, and to set them as the target execution order. For example, the multiple target log topics include log topic a, log topic b, and log topic c. When the execution order setting control is triggered, the operation and maintenance device receives data (e.g., receives the execution order of a, b, and c) in the execution order setting control and sets it as the target execution order.

[0046] (2) Generate log data retrieval tasks based on the target execution order and multiple target log topics. The log data retrieval tasks include the target execution order, which is used to instruct the log processing program to perform log retrieval according to the target execution order.

[0047] In one embodiment, the log processing program may perform log retrieval in the order of execution of the targets included in the data retrieval task (e.g., the log processing program performs log retrieval in the order of execution of received abc).

[0048] In one embodiment, the search interface further includes an orchestration search control, which is used to obtain a target orchestration from multiple orchestrations. The method for obtaining tasks to be processed based on orchestration can be implemented according to the following steps.

[0049] (1) When the arrangement search control is triggered, one or more arrangements are displayed on the search interface.

[0050] When the orchestration retrieval control is triggered, i.e., a trigger operation is received for the orchestration retrieval control, the maintenance equipment displays one or more orchestrations. The method for displaying one or more orchestrations can be various, such as generating an option floating window, generating an option list, or generating a new interface. Displaying one or more orchestrations in these ways is used to detect the operator's trigger operation on one or more orchestrations. After detecting the operator's trigger operation on one or more orchestrations, the maintenance equipment can use the orchestration corresponding to the operator's trigger operation as the target orchestration. The one or more orchestrations can be pre-configured; the configuration method will be described in detail in subsequent embodiments and will not be repeated in this embodiment.

[0051] (2) When the target orchestration is triggered, a log data retrieval task is generated based on the target orchestration. The target orchestration includes multiple target log topics, and the multiple target log topics include the business server identifier and the target log storage path.

[0052] In this embodiment, an orchestration can be understood as a set of associated data. Each set of associated data includes an orchestration name, a list of log topics (including multiple log topics), and the original execution order of the multiple log topics in the log topic list. Each log topic includes a log topic name, a business server identifier, and multiple log storage paths. Log files are stored under the log storage paths. A log data retrieval task is generated based on the target orchestration. The essence of this log data retrieval task is to perform log retrieval on the log files under the multiple log storage paths corresponding to the business server identifiers of the multiple target log topics corresponding to the target orchestration. Here, the target orchestration is one of one or more orchestrations displayed on the retrieval interface.

[0053] In this embodiment, the operation and maintenance equipment combines multiple target log topics to generate a target orchestration, and then generates a log data retrieval task based on the target orchestration. This realizes the association between logs, provides cascading analysis and filtering capabilities for logs, and also improves the freedom and scalability of log retrieval.

[0054] In one embodiment, the retrieval interface further includes an execution order setting control, and the target orchestration includes the original execution order. The method described above for generating log data retrieval tasks based on the target orchestration can be implemented according to the following steps.

[0055] (1) When the execution order setting control is triggered, the target execution order is determined according to the execution order setting control.

[0056] In this embodiment, the execution order setting control is used by the operation and maintenance equipment to set the execution order of multiple log topics included in the target orchestration, and to set them as the target execution order. For example, the target orchestration includes log topic a, log topic b, log topic c, and log topic d. When the execution order setting control is triggered, it receives data (e.g., the execution order of bcad) and sets it as the target execution order.

[0057] (2) Adjust the original execution order in the target orchestration to the target execution order, and generate a log data retrieval task according to the adjusted target orchestration. The log data retrieval task includes the target execution order, which is used to instruct the log processing program to perform log retrieval according to the target execution order.

[0058] In this embodiment, the original execution order of the target orchestration can be generated when configuring multiple log topics in the target orchestration (e.g., the original execution order of the target orchestration is generated according to the order in which multiple log topics in the target orchestration are configured). The target execution order can be customized by the operator based on business conditions (e.g., adjusting the log retrieval order according to the content to be analyzed). The operation and maintenance equipment instructs the log processing program to perform log retrieval according to the target execution order, thereby improving the accuracy of log retrieval.

[0059] It should be noted that the target execution order can be either sequential or parallel. Sequential execution means that multiple execution entities (e.g., log topic a, log topic b, and log topic c) execute sequentially according to the target execution order (e.g., in the order of a, b, c). For example, log topic a is executed first, then log topic b, and finally log topic c. Sequential execution is well-suited for scenarios requiring hierarchical retrieval, such as using the results of the previous retrieval as input for the next.

[0060] Parallel execution order refers to the parallel execution of multiple execution entities (e.g., log topic a, log topic b, and log topic c) based on a multi-task approach (e.g., executing log topic a, log topic b, and log topic c in parallel). Parallel execution order is well-suited for fast retrieval scenarios, such as retrieving data that satisfies any of multiple conditions.

[0061] When retrieving tasks based on log topics or orchestration, the target execution order is determined. When the target execution order is sequential, multiple target log topics include a first log topic and a second log topic, with the first log topic being executed before the second log topic. The processing result includes the execution result of the first log topic and the difference between the execution results of the first and second log topics. The difference execution result is the result of the log processing program filtering the execution result of the second log topic based on the execution result of the first log topic.

[0062] In this embodiment, the first log topic and the second log topic are two adjacent log topics, and the execution order of the first log topic precedes that of the second log topic. Performing log retrieval based on the first log topic yields the execution result of the first log topic, and performing log retrieval based on the second log topic yields the execution result of the second log topic. The difference between the execution results of the first and second log topics refers to the execution result after the log processing program filters the execution result of the second log topic based on the execution result of the first log topic. In other words, it is the execution result obtained by filtering the execution result of the second log topic within the execution result of the first log topic. This method avoids duplicate collection of the same log data, improving computational efficiency and resource utilization.

[0063] In one embodiment, the execution result of the first log topic includes log data A, log data B, log data C, and log data D, and the execution result of the second log topic includes log data A, log data C, log data D, and log data E. The difference execution result between the execution results of the first and second log topics is the result of filtering out duplicate log data C and D from the execution result of the second log topic, resulting in log data A and log data E. Therefore, the processing result is obtained by adding the execution result of the first log topic and the difference execution result between the execution results of the first and second log topics; that is, the processing result is log data A, log data B, log data C, log data D, and log data E.

[0064] This application provides a user-friendly, intuitive log retrieval and analysis interface. "User-friendly" means that the interface is displayed through a web interface or other visual methods, rather than a console, greatly improving the user experience and reducing the learning curve for operating the log retrieval and analysis system proposed in this application. In the process of retrieving tasks based on log topics, one or more log topics are configured through the log topic configuration interface; similarly, in the process of retrieving tasks based on orchestration, one or more orchestrations are also configured through the orchestration configuration interface.

[0065] Please see Figure 3B , Figure 3Bis a schematic diagram of an interface for configuring log topics in a log retrieval and log analysis system provided by an embodiment of the present application, which includes a plurality of log topics, each log topic corresponds to a log topic name, a service server identifier, a plurality of log storage paths and configuration time, and the configuration time is the time when the operator completes the configuration of the log topic. For example, the figure includes a log topic named yunapi-acess, the corresponding service server identifier of which is tec-k8s-node (the service server with the service server identifier tec-k8s-node is configured in a service server group), and the corresponding log storage path thereof is / data / k8s / log / tce / tcloud-tcenter-yunapi3-access...... the configuration time of this log topic is 01:01:01, January 1, 2022. The log topic configuration interface is also provided with an editing control for each log topic, which facilitates the operator to perform operations such as viewing, modifying and deleting on the configured log topics; meanwhile, the log topic configuration interface is also provided with an adding control for the operator to add a new log topic. After all log topics are configured, the plurality of configured log topics together form a log topic library, and the information of the log topic library displayed on the log topic configuration interface includes a log topic library name (for example, tce360), a library version (for example, 10), and a last modification time (for example, 01:01:01, January 1, 2022). The log topic configuration interface is also provided with a topic library import control, which facilitates utilization of an already created log topic library and improves the configuration efficiency of log topics.

[0066] Please refer to Figure 3C , Figure 3Cis a schematic diagram of an interface for orchestration rule configuration in a log retrieval and log analysis system provided by an embodiment of the present application. The interface comprises a plurality of orchestrations, each orchestration corresponds to an orchestration name, an orchestration description (describing specific information of the orchestration), a log topic list (comprising log topic names corresponding to a plurality of log topics, wherein the log topic names have a corresponding relationship with the log topic names comprised in a log topic configuration interface, that is, the log topic names comprised in the log topic list exist in the log topic configuration interface, and relevant information of corresponding log topics is acquired by looking up the log topic names in the log topic configuration interface), and configuration time, which is the time when an operator completes the configuration of the orchestration. For example, the figure comprises an orchestration named cvm-diag, the corresponding orchestration description thereof is "one-click troubleshooting requirement for CVM", and the corresponding log topic list thereof is "access-log.cvm-cgw-log.xxxx" (a plurality of log topics comprised in the log topic list are separated by ".", that is, the log topics comprised in the log topic list are access-log, cvm-cgw-log and xxxx), and the configuration time of the orchestration is 01:01:01, January 1, 2022. The orchestration configuration interface is further provided with an editing control for each orchestration, which is convenient for an operator to perform operations such as viewing, modifying and deleting on the configured orchestrations; meanwhile, the orchestration configuration interface is further provided with an adding control for an operator to add a new orchestration. After all orchestrations are configured, the plurality of configured orchestrations together form an orchestration library, and the information of the orchestration library displayed on the orchestration configuration interface comprises an orchestration library name (e.g., tce), a library version (e.g., 10), and a last modification time (e.g., 01:01:01, January 1, 2022). The orchestration configuration interface is further provided with an orchestration library import control, which is convenient for operation and maintenance equipment to perform log retrieval by using a created orchestration library and improves the configuration efficiency of orchestrations.

[0067] S202, sending a to-be-processed task and a log processing program corresponding to the to-be-processed task to a service server corresponding to a service server identifier, wherein the log processing program is configured to execute the to-be-processed task according to log data stored in the service server.

[0068] In the embodiment of the present application, the log processing program corresponding to the to-be-processed task is a non-resident program delivered to the service server corresponding to the service server identifier, and the log processing program can execute the to-be-processed task (e.g., a log data retrieval task or a log data analysis task) according to the log data stored in the service server.

[0069] Wherein, the functions and interaction manners of the log processing program will be described in detail in subsequent embodiments, and will not be repeated herein in this embodiment.

[0070] S203. Receive the processing result of the pending task sent by the log processing program on the business server after the task has been completed, output the processing result, and send a destruction command to the log processing program on the business server. The destruction command is used to destroy the log processing program on the business server.

[0071] In this embodiment, the log processing program is a non-resident program. During execution, it returns the processing results and displays them on the search interface. Once the task to be processed is completed, it can be destroyed. Using this log processing program for log data retrieval and analysis eliminates the need for centralized storage and analysis, improving efficiency, reducing resource storage costs, and eliminating the need for deploying log collectors or log analyzers, thus lowering deployment costs.

[0072] It should be noted that there can be multiple pending tasks. The maintenance equipment issues a log processing program, which then obtains the target pending task from among the multiple pending tasks and executes the corresponding operation based on the target pending task. According to this method, the log processing program executes all pending tasks sequentially. When the maintenance equipment detects that the log processing program has returned all processing results, it sends a destruction command to the log processing program on the business server. The destruction command is used to destroy the log processing program on the business server.

[0073] Please see Figure 3A , Figure 3A The 305 error message in the image shows the processing results interface after the pending tasks sent by the log processing program have been completed. It includes an identifier bar (containing three items: time, log data, and operation) and multiple processing result data entries. Each result data entry includes the time and log data. The time indicates the return time of the log data entry, and the log data displays the relevant content, including the business server address, log file, and message. The message is the content of the log file. The processing result interface also provides a file view control and a file information control for each processing result, allowing the operator to easily view the corresponding log data. When the file view control is triggered, a new data display interface is generated to show the log data. When the file information control is triggered, the file attributes of the corresponding log file (such as file size, creation time, version information, etc.) are displayed. Multiple processing results can be displayed in a paginated format. (Li Ru) Figure 3A The page number indicator control in the lower right corner can be clicked to jump to the corresponding page number to view the processing result.

[0074] In one embodiment, the processing result sent by the log processing program after the completion of the pending task includes a processing result, which includes time (e.g., 2022-01-01 01:01:01), log data (e.g., the business server address is "xx.xx.xx.xx", the log file is " / data / test / log", and the message is "[2021-07-07.11:23:47.056892]INFO xx.xx.xx.com / cloud_ops / log_ops...").

[0075] This application automates log processing by issuing pending tasks to a business server and providing a log processing program capable of executing those tasks. After the log processing program completes its task, the application receives the processing result from the business server's log processing program. Compared to manual collection, this achieves automated log processing. Furthermore, the maintenance equipment only stores the processing result, eliminating the need to store the full log data, thus reducing storage costs and labor costs. Upon detecting the completion of a pending task, a destruction command can be sent to the log processing program on the business server to destroy the program, improving the resource utilization of the business server. This method of issuing log processing programs to execute pending tasks improves log retrieval efficiency. The pending tasks need to be pre-defined; they can be log data retrieval or log data analysis tasks. Pre-setting allows for the specific definition of the business server to be processed, the log path within the business server, the execution time, and the device information to be analyzed, thereby improving the accuracy and efficiency of log retrieval and analysis.

[0076] When the task to be processed is a log data retrieval task, the retrieval interface allows configuration of the task. Log retrieval can be based on log topic or orchestration. A log topic corresponds to a log topic name, a business server identifier, and multiple log storage paths. An orchestration corresponds to an orchestration name, a list of log topics (including multiple log topics), and the original execution order of the log topics in the list. The specific execution order for log topic-based or orchestration-based retrieval can be set via an execution order setting control. Through these highly customizable retrieval settings, the association between logs is realized, enabling cascading analysis and filtering capabilities, and improving the freedom and scalability of log retrieval. This application also allows importing a topic library (or orchestration library) during log topic configuration (or orchestration configuration), facilitating the use of pre-created log topic libraries (or orchestration libraries) and improving the configuration efficiency of log topics (or orchestrations). This application also proposes obtaining processing results based on the differences in execution results from multiple log topics, avoiding duplicate collection of the same log data and improving computational efficiency and resource utilization. This application provides a white-screen log retrieval and analysis interface, which greatly improves the user experience and reduces the learning cost of operating the log retrieval and log analysis system proposed in this application.

[0077] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data processing method provided in an exemplary embodiment of this application. Taking the application of this method to maintenance equipment as an example, the method may include the following steps: S401. Obtain the task to be processed, which includes the business server identifier.

[0078] In this embodiment, the task to be processed can be generated based on the displayed search interface. For a detailed description of step S401 in the preceding embodiments, please refer to the relevant description of step S201; this embodiment will not repeat it here. In one embodiment, the search interface further includes a filter control, and the method for filtering logs using the filter control is as follows.

[0079] When the maintenance equipment obtains the filtering parameters through the filtering control, it sends the filtering parameters to the log processing program on the business server. The filtering parameters are used to instruct the log processing program to perform log data retrieval tasks according to the filtering parameters. The filtering parameters include keywords and environment attributes. The environment attributes include one or more of the test environment attributes and development environment attributes.

[0080] In this embodiment, the maintenance equipment sends filtering parameters to the log processing program. These filtering parameters instruct the program to perform a log data retrieval task based on them; that is, the log processing program performs log retrieval according to the filtering parameters. It should be noted that log retrieval based on these filtering parameters can occur at the start of the log retrieval process (e.g., retrieving log data using only the filtering parameters) or during the log retrieval process (e.g., after retrieving log data based on a target log topic, the filtering parameters are used again to filter out log data that meets the criteria based on the processing results of the target log topic). This improves the efficiency of log retrieval.

[0081] like Figure 3A As shown, Figure 3A The 303 documentation illustrates a user interface for log retrieval based on filtering parameters. It showcases several functional controls, including time-period filtering, keyword filtering, filename filtering, and business server identifier filtering. Additionally, it includes prompt messages (e.g., "Please enter your filtering criteria"). When the maintenance equipment detects a trigger operation of a target functional control, it displays the corresponding input window (which can also be a content selection window; for example, the content selection window for the business server identifier filtering control includes all business server identifiers in the processing results. By retrieving the trigger operation of the target business server identifier from all business server identifiers, the processing results corresponding to the target business server identifier are displayed). When input information is received, and when a submission control is triggered, the processing results are filtered based on the input information, and the filtered results are displayed.

[0082] In one embodiment, the operation and maintenance equipment may send the task to be processed, the log processing program corresponding to the task to be processed, and the filtering parameters to the business server corresponding to the business server identifier, perform log retrieval based on the task to be processed and the filtering parameters, and then receive the processing result after the task to be processed is completed from the log processing program on the business server, and output the processing result.

[0083] In one embodiment, the maintenance equipment may send the tasks to be processed and the corresponding log processing program to the business server identified by the business server identifier, so that the log processing program can execute the tasks. Simultaneously, it can also send other tasks to the log processing program for execution. In this case, there is only one log processing program on the business server, which is used to execute multiple tasks. After any one of the multiple tasks is completed, the system can receive the processing result from the log processing program on the business server and output the result.

[0084] S402. Send the task to be processed to the business server corresponding to the business server identifier, so that the business server can calculate the first message digest of the task to be processed.

[0085] A message digest, also known as a digital digest, is a fixed-length value that uniquely corresponds to a message or text. It can be generated by applying a one-way hash function to the message. If the message is altered en route, its corresponding message digest will also change. The receiver can determine whether the message has been modified by comparing the newly generated digest with the original digest. This application uses message digests for data verification, ensuring data integrity.

[0086] In one embodiment, the message digest can be calculated using a one-way hash function to generate a 128-bit ciphertext from the plaintext to be encrypted. This ciphertext is the message digest proposed in this application. The message digest can also be called a digital fingerprint. It has a fixed length, and different plaintexts will always produce different ciphertexts, while the same plaintext will always produce the same message digest. Based on this, the message digest can be used to verify whether the plaintext has been tampered with.

[0087] S403. Obtain the first message digest sent by the business server and calculate the second message digest of the task to be processed.

[0088] In this embodiment, after the business server calculates the first message digest of the task to be processed, the maintenance equipment can obtain the first message digest sent by the business server and compare it with the second message digest to verify the integrity of the data. The second message digest serves as a comparison standard and can be obtained by the sender of the log processing program (e.g., the maintenance server) calculating the message digest of the task to be processed.

[0089] S404. If the first message digest and the second message digest are the same, the log processing program corresponding to the task to be processed is sent to the business server corresponding to the business server identifier. The log processing program is used to execute the task to be processed based on the log data stored in the business server.

[0090] In this embodiment, if the first message digest and the second message digest are the same, it means that the data of the task to be processed corresponding to the first message digest is exactly the same as the data of the task to be processed corresponding to the second message digest. In this case, the maintenance device can send the log processing program corresponding to the task to be processed to the business server corresponding to the business server identifier, so as to execute the task to be processed based on the log data stored in the business server. If the first message digest and the second message digest are different, it means that the data of the task to be processed corresponding to the first message digest is not exactly the same as the data of the task to be processed corresponding to the second message digest, and the data may have changed. In this case, the task to be processed on the business server is incomplete. Based on this, the maintenance device can generate a prompt message, which is used to instruct the sender to resend the task to be processed.

[0091] In one embodiment, the first message digest and the second message digest can be calculated using any one of md2, md4, md5, SHA1, SHA256, SHA384 and SHA512.

[0092] In one embodiment, after the log data retrieval task is completed, the maintenance equipment will obtain the processing result (i.e., the retrieved log data). Since the log data retrieval task is a phased task, it is completed within a certain time period based on the amount of log data retrieved. To improve the user experience of the log retrieval and log analysis system and allow operators to understand the data retrieval progress in real time, this application proposes to periodically obtain the log data retrieval progress information, and its implementation method is as follows.

[0093] (1) During the process of the log processing program on the business server executing the log data retrieval task, a progress acquisition request is sent to the log processing program at regular intervals so that the log processing program can obtain the progress information of the task to be processed.

[0094] (2) Receive progress information sent by the log processing program on the business server and display the progress information in the form of an image.

[0095] In this embodiment, the progress information of the task to be processed refers to the ratio between the number of completed subtasks and the total number of subtasks to be completed. The progress information can be presented in the form of different types of statistical charts, such as progress bars, line charts, bar charts, pie charts, etc.

[0096] like Figure 3AAs shown in Figure 303, the search interface displays a progress bar indicating the search progress, including a progress display box (such as the progress box in the figure) and a progress value (such as 50%). It also displays a details control. When the maintenance device receives a trigger operation from the details control, it displays detailed progress information (such as all log file data that has been retrieved, log file data that has not been retrieved, etc.). The search interface also displays the total number of tasks, that is, the total number of subtasks that need to be processed (such as 10,000). By displaying the total number of tasks, the operator can understand the approximate workload of the tasks to be processed, thus improving the user experience.

[0097] like Figure 3A As shown in Figure 304, a 304 error displays the number of hits for each time interval within a preset time range after searching based on filtering conditions. For example, when the maintenance device detects a filtering condition (e.g., the filter condition is " / host" for filename filtering) and detects a trigger operation on the submit control, the current time (e.g., 10:30) is obtained. If the time interval is 1 hour, the number of hits based on the filtering condition within the preset time range (e.g., 10:30~11:30) is displayed. It should be noted that the preset time range can be preset, i.e., the statistical duration can be set (e.g., if the statistical duration is 2 hours, then the preset time range is 10:30~12:30), or... Figure 3A Set the number of bars in the bar chart of 304 (for example, set 50 bars, divide the preset time range into 50 parts, and count the hit count for each part).

[0098] In this embodiment, the task to be processed specifies the scope to be searched (i.e., log data under different log storage paths). At the start of the log data retrieval task, the maintenance equipment can first count the total number of searches, and then count the number of searches completed during the log data retrieval process. Progress information is determined based on the total number of searches and the number of searches completed. The total number of searches can be the total number of files searched or the total number of bytes searched, and the corresponding number of searches completed can be the number of files searched or the number of bytes searched. When the number of business servers is one, progress information can be calculated based on this method.

[0099] In one embodiment, when there are multiple service servers, each service server corresponds to a unit progress information. The progress information is the minimum unit progress information, the maximum unit progress information, or the average unit progress information among the multiple unit progress information corresponding to the multiple service servers.

[0100] The above can be understood as follows: each business server is a subtask, and each subtask includes sub-progress information. The progress information displayed on the search interface needs to represent the overall progress of multiple subtasks corresponding to multiple business servers. Therefore, the minimum unit progress information, the maximum unit progress information, or the average unit progress information among the multiple unit progress information corresponding to multiple business servers can be used as the overall progress (i.e., progress information).

[0101] For example, when the business servers include server 1, server 2 and server 3, the progress information corresponding to server 1 is 10% (that is, the ratio of the number of retrieved items to the total number of retrieved items is 1:10), the progress information corresponding to server 2 is 30%, and the progress information corresponding to server 3 is 20%. Then, by using the smallest unit of progress information as the progress information, the progress information corresponding to server 1 (e.g., 10%) can be used as the progress information of multiple business servers.

[0102] S405. Receive the processing result of the pending task sent by the log processing program on the business server after its execution is completed, output the processing result, and send a destruction command to the log processing program on the business server. The destruction command is used to destroy the log processing program on the business server.

[0103] The specific implementation of step S405 is described in the relevant description of step S203 in the foregoing embodiments, and will not be repeated here.

[0104] The log retrieval and analysis method proposed in this application can not only perform log data retrieval tasks, but also log data analysis tasks (i.e., the task to be processed is a log data analysis task). When the task to be processed is a log data analysis task, and the log processing program integrates an analysis model, the implementation of the above steps for obtaining the task to be processed is as follows.

[0105] The analysis configuration interface is displayed. When the maintenance equipment obtains the information of the device to be analyzed through the analysis configuration interface, it generates a log data analysis task based on the information of the device to be analyzed. The information of the device to be analyzed is used by the log processing program to analyze the log data of the device to be analyzed by calling the analysis model. The information of the device to be analyzed includes one or more of the following: network address, login time, logged-in device, and device operation.

[0106] The device information to be analyzed can be the device information of multiple devices included in a single business server, or the device information of multiple devices included in multiple business servers. Its specific scope can be configured according to business requirements. The functions, implementation methods, and interface layout of the analysis configuration interface can be referred to the relevant descriptions of the search interface in the preceding embodiments, and will not be repeated in this embodiment.

[0107] In one embodiment, when abnormal information appears in the device information to be analyzed, it indicates that the entity corresponding to the abnormal information in the business server is at risk. This allows for a risk warning to the operator performing the log data analysis task, facilitating early risk avoidance and improving data analysis efficiency based on the risk warning. The risk warning can be implemented by generating an anomaly report, the specific implementation of which is as follows.

[0108] An anomaly report is generated based on the processing results. The anomaly report includes information about the abnormal devices to be analyzed from the device information to be analyzed.

[0109] The abnormal information in the device information to be analyzed can include abnormal network addresses (such as network addresses in the blacklist or network addresses outside the country), abnormal login times (such as a login operation at 4:00 AM, indicating that the login may be an abnormal user login and requires risk analysis), abnormal login devices (such as new devices other than commonly used devices), and abnormal device operations (such as multiple consecutive identical device operations). In addition to the abnormal network addresses, abnormal login times, abnormal login devices, and abnormal device operations mentioned above, the anomaly report can also include the corresponding related information. For example, when displaying the abnormal login time item in the anomaly report, in addition to showing the abnormal login time, it can also show the device model, device login address, and device operation (in this case, the device operation is a login operation). By displaying an anomaly and its corresponding related information, the operation and maintenance equipment facilitates log data analysis, improves analysis efficiency, and enhances the user experience.

[0110] Based on steps S201-S203 and steps S401-S405 of this application, embodiments of this application propose an implementation method for a log retrieval and log analysis system. Please refer to... Figure 5A , Figure 5A This is an architecture diagram of the log retrieval and log analysis system provided in this application embodiment, including a web client, a master client, and a client (in... Figure 5A In this context, "front-end" refers to the web client, "back-end" refers to the master client, "business server" refers to the host, and "business server side" refers to the client. The client exists within the business server side. Figure 5AThe backend includes master1 (i.e., backend 1) and master2 (i.e., backend 2), and the client side includes client1 (i.e., business server group 1) and client2 (i.e., business server group 2). The data in master1 and master2 is identical. master1 and master2 can be considered a master-slave relationship; for example, master1 acts as the master server, and master2 acts as the slave server. The slave server master2 can serve as a backup of the master server master1, used to restore the master server master1 in special circumstances (e.g., if the master server master1 fails). When master1 and master2 are identical, mutual synchronization can be configured (e.g., ...). Figure 5B (as shown in step 7) and cache synchronization (such as...) Figure 5B (As shown in step 8) Data synchronization is performed between master1 and master2 to facilitate data restoration to the master server. Since the client can include multiple business server groups (e.g., business server 1 and business server 2), the master can send non-resident log processing programs (i.e., the log processing programs corresponding to business server 1 and business server 2 in the diagram) to each business server in each business server group. The interaction method between the master and each business server group is consistent, so the following explanation will take the interaction between master1 and the first business server group (i.e., business server group 1) as an example.

[0111] The web client, master client, and client client will be described in detail below.

[0112] First, the web-based client is introduced. It provides a user-friendly (white-screen) log retrieval and analysis system, enhancing the overall user experience. The web client primarily handles operations related to log data editing, log retrieval, and log analysis. The data model can include business server groups, log topics, and orchestration rules, defined as follows: a business server group indicates a set of target business servers for log retrieval; a log topic associates the business server group with a set of log storage paths; and an orchestration rule associates a set of log topics. It should be noted that the web client and the master client can be considered as a whole, serving as the operation and maintenance device (or operation and maintenance terminal) in this embodiment.

[0113] Log data retrieval tasks mainly include topic retrieval and orchestration retrieval, defined as follows: Topic retrieval is a retrieval task performed on a single log topic, essentially retrieving log data from a set of files across a group of hosts (which can be understood as performing a log data retrieval task based on a target log topic within a log topic); Orchestration retrieval is an orchestration retrieval task performed on multiple log topics (which can be understood as performing a log data retrieval task based on a target orchestration rule within an orchestration rule). Orchestration retrieval can further include parallel retrieval and serial retrieval methods, as detailed below. Parallel retrieval involves executing multiple log topic retrievals in parallel. After selecting a target orchestration rule, log data retrieval tasks can be performed separately for each log topic within the target orchestration rule, and the retrieval results for each log topic in the target orchestration rule are summed to obtain the final result. Serial retrieval means using the result of the previous topic retrieval as the filtering condition for the next topic retrieval. In other words, it is based on the execution result of the first log topic among two adjacent target log topics, and the difference execution result between the execution result of the first log topic and the execution result of the second log topic. The difference execution result is the execution result after the log processing program filters the execution result of the second log topic based on the execution result of the first log topic.

[0114] It should be noted that the web client can interact with the master client by calling the RESTful API (Representational StateTransfer).

[0115] Next, we'll introduce the master side. The master side has a three-tier architecture, including: an API layer that provides services to the outside world (i.e., ... Figure 5A The interface layer in the middle, and the client-server layer (i.e., the interface layer in the middle) that interacts with the client to exchange data. Figure 5A The service layer in the middle), and the action-manager layer located between the two layers mentioned above (i.e., Figure 5A The action manager in the context of [the application / system]. The API layer can simultaneously provide two sets of interface services: RESTful API and gRPC (an open-source, high-performance RPC framework for remote procedure calls); the client-server layer uses a Secure Shell (SSH) channel (i.e., [other protocol]). Figure 5AThe Security Shell Protocol layer in the middle issues retrieval-related tasks and receives the return data of the retrieval tasks through the gRPC interface; the Manager is mainly responsible for two things: first, persisting the basic data model (such as business server groups, log topics, topic orchestration) (e.g., storing it in a YAML file); second, caching and managing the process and result data of distributed retrieval tasks (e.g., merging data, statistics, timed cleanup, timed backup, etc.).

[0116] Next, let's introduce the client. The client is a non-resident program that is sent from the master to the host via an SSH channel. During its execution, it actively submits data to the master via a gRPC channel. The client mainly includes the following functions.

[0117] (1) Log retrieval: Target logs are retrieved by filtering parameters, which may include time period (necessary), keywords, business server identifier, file name, etc.

[0118] (2) Submit search progress: Actively report the search progress to the master during the search process in the log.

[0119] (3) Time period hit count statistics: Count the number of hits in the search results within a time period.

[0120] (4) Log retrieval pagination: Returns log retrieval results in pagination.

[0121] (5) View file information: View the file information where the selected target log is located.

[0122] (6) Log context browsing: View the context information of the selected target log.

[0123] (7) MD5 verification: ensures the validity of the client program and improves the speed of client program distribution.

[0124] (8) CPU control: Limit the client's CPU resource usage on the host.

[0125] Please see Figure 5AA complete log retrieval operation can include the following interactive steps. First, the web client interacts with the internal DNS (i.e., step 1.1 domain name resolution and step 1.2 domain name response). After obtaining the domain name, the web client sends a retrieval request to the master server (i.e., step 2.1 sending the retrieval request and step 2.2 returning the request result). The master server includes api-server, action-manager, ssh-lib (i.e., the SSH component), and client-server. First, the api-server creates a log data retrieval task for the action-manager. The api-server sends the client and the command to start the retrieval to the ssh-lib control, which then issues the client and starts the log data retrieval task. The client can include multiple business servers (e.g., business server 1 and business server 2). The master server sends a non-resident log processing program to each business server (i.e., the log processing programs corresponding to business server 1 and business server 2 in the diagram). The interaction between the master and each business server is consistent. Therefore, the interaction between the master and the first business server (i.e., host1) will be used as an example. The ssh-lib control in the master first sends the client to host1 and starts the log data retrieval task. During the execution of the log processing program on host1, the master receives the progress information and processing results returned by host1 through gRPC.

[0126] Please see Figure 5B , Figure 5B This is a sequence diagram of the log retrieval and log analysis system provided in this application embodiment, including a web client, a master client, and a client client. The client client resides on the business server. A complete retrieval request can be divided into two parts: the interaction process between the web client and the master client, and the interaction process between the host client and the master client. The following will take a log topic retrieval task as an example to further describe the interaction methods of the web client, master client, and client client.

[0127] The first part, the interaction process between the web client and the master client, will be introduced below. The web client and the master client communicate via HTTP, therefore a RESTful protocol is used. A complete log retrieval involves three core steps between the web client and the master client: creating an asynchronous retrieval task, obtaining the asynchronous retrieval progress, and obtaining the asynchronous retrieval results. The specific protocol details for these three steps are as follows.

[0128] The first step is to create an asynchronous search task. This involves the web client submitting a topic search request to the master client, which includes filtering parameters (such as...). Figure 5B As shown in step 1), the master creates a search based on the search request (e.g., ...). Figure 5B As shown in step 2), and the retrieval packet sent back to the web client, the retrieval packet includes the search-id (i.e., the task identifier, such as...). Figure 5B (See step 3).

[0129] Specifically, the web client initiates an asynchronous retrieval request task to the master server via a RESTful API. The master server creates a retrieval request object and returns asynchronous task information to the web client. The following are the protocol details for creating an asynchronous retrieval task.

[0130]

[0131] Next, we'll introduce the second step: obtaining the asynchronous search progress. After obtaining the search-id, the web client requests the current search progress from the master client using the search-id (e.g., ...). Figure 5B As shown in step 4), the master returns the retrieval progress (e.g., ...). Figure 5B (See step 5 in the middle).

[0132] Specifically, after creating the asynchronous search task, the web client receives an asynchronous search task ID. The master client then synchronously initiates search tasks on both the master and host sides. This task is often time-consuming (e.g., several seconds or minutes). To ensure a smooth search experience on the web client, the search task progress can be retrieved and a progress bar displayed periodically (e.g., every 3 seconds). The following are the protocol details for retrieving asynchronous search progress.

[0133]

[0134] The third step involves retrieving asynchronous search results. Specifically, the web application continuously monitors the progress of the asynchronous search task and checks if the progress is 100%. Once the task is detected as complete, the web application retrieves the search results from the master server using the search-id (e.g., ...). Figure 5B (See step 6 in the middle); then the master end returns the search results (such as...) Figure 5B (See step 7).

[0135] The following are the protocol details for obtaining asynchronous search results.

[0136]

[0137] The second part, the interaction process between the client and master, will be introduced below. The host and master communicate in the background, using the gRPC protocol for efficient communication. When the master receives a search task, it parses the list of business servers to be searched and sends the log analysis and processing client via the SSH channel. The client then starts the search task via the SSH channel. After starting the task, the client performs the following steps: the master sends the log analysis client to the host, the client obtains the search task, the client reports the search task progress in real time, and the client submits the search results to the master. These four steps will be described in detail below.

[0138] It should be noted that after the master creates the retrieval, it first sends an MD5 verification request (i.e., message digest verification) to the host via the SSH (Secure Shell) channel. Figure 5B (As shown in step 2.1); the client performs an MD5 self-check (e.g.) Figure 5B (as shown in step 2.2), and return the MD5 value via the SSH channel (e.g.) Figure 5B (As shown in step 2.3). The master end determines whether to send a client by comparing whether the MD5 values ​​are consistent. If the MD5 values ​​are consistent, the client is sent; if the MD5 values ​​are inconsistent, the client is not sent. The aforementioned client refers to the log processing program sent to the business server for log retrieval.

[0139] The first step involves the master sending the log analysis client to the host. When the master receives a search task, it resolves the list of business servers to be searched, sends the log analysis and processing client (with the search-id as a parameter) via the SSH channel, and starts the client to execute the search task (e.g., ...). Figure 5B (See step 2.4).

[0140] The second step involves the client acquiring the search task. After the client is started, it requests the search task ID and parameters such as CPU and memory management via the SSH channel. In other words, it requests specific search parameters (e.g., search-id) from the master server. Figure 5B (See step 2.5); the master end will use the gRPC protocol (i.e. Figure 5BRemote procedure call (RPC) sends retrieval parameters (such as...) Figure 5B (As shown in step 2.6); After obtaining the search parameters, the client will retrieve the search task information based on the search task ID and execute the search task (e.g., ...). Figure 5B (See step 2.7). The following are the details of the client obtaining the retrieval task protocol.

[0141]

[0142] Next, we'll introduce the third step: the client reporting the retrieval task progress in real time. The client needs time to perform log analysis. To improve the retrieval experience, it needs to report the analysis progress in real time. This means the client continuously submits retrieval progress updates to the master during the retrieval task execution process (e.g., ...). Figure 5B As shown in step 2.7.1, for the master side, this means periodically obtaining the client's progress information during the execution of the retrieval task. In addition, the master side can also manage the retrieval progress (e.g., Figure 5B (See step 2.7.2). The following are the protocol details for the client to report the progress of the retrieval task in real time.

[0143]

[0144] The fourth step involves the client submitting the search results to the master. After completing the search, the client submits the results to the master (e.g., ...). Figure 5B As shown in step 2.7.3), the master then merges the results from each host. In addition, the master can also manage log data (such as...). Figure 5B (See step 2.7.4). The following are the protocol details for the client to submit search results to the master.

[0145]

[0146] This application implements the log retrieval and analysis system proposed above and conducts log retrieval and analysis tests. The tests show that the proposed log retrieval and analysis system is suitable for log analysis in scenarios such as private clouds and public clouds. It can be deployed without manual intervention and provides a user-friendly, intuitive log retrieval and analysis interface. This system avoids centralized collection of log data for analysis, exhibits strong scalability, and is also suitable for log retrieval and analysis tasks in large-scale distributed systems. Using this solution, the starting point for retrieval can be located from a single 2.36G log file within 50ms, and 1 million target log entries can be found from 14 log files in just 10.75 seconds.

[0147] This application allows for retrieval based not only on log topics or arrangement but also on filtering parameters. These parameters can be used independently as retrieval criteria or combined with log topics and arrangement to improve accuracy and efficiency. Before sending the log processing program corresponding to the task to the business server, this application first sends the task for message digest verification. This message digest-based verification ensures data integrity. After sending the log data retrieval task to the business server, a progress request is periodically sent to the log processing program to obtain its progress information. Upon receiving the returned progress information, it is displayed graphically, providing a clear and intuitive understanding of the retrieval progress and improving user experience. When the task is a log data analysis task, the log processing program integrates an analysis model. By configuring the device information to be analyzed through the analysis configuration interface, the log processing program on the business server can be guided to analyze the log data based on this information, improving the efficiency of log analysis. When abnormal information is obtained, it indicates that the entity corresponding to the abnormal information in the business server is at risk. This allows for risk alerts to operators performing log data analysis tasks, facilitating early risk avoidance and enabling data analysis based on the risk alerts, thereby improving data analysis efficiency. This application also proposes a detailed architecture and interaction scheme for a log retrieval and log analysis system, and provides detailed explanations of data from various functional protocols, providing a basis for implementing the log retrieval and log analysis system of this application.

[0148] Please see Figure 6 , Figure 6 This is a schematic block diagram of a data processing apparatus provided in an embodiment of this application. Specifically, the data processing apparatus may include: The acquisition module 601 is used to acquire tasks to be processed, including the business server identifier. The sending module 602 is used to send the above-mentioned task to be processed and the log processing program corresponding to the above-mentioned task to the business server corresponding to the above-mentioned business server identifier. The log processing program is used to execute the above-mentioned task to be processed based on the log data stored in the above-mentioned business server. The processing module 603 is used to receive the processing result of the pending task after it has been completed, sent by the log processing program on the business server, output the processing result, and send a destruction command to the log processing program on the business server. The destruction command is used to destroy the log processing program on the business server.

[0149] Optionally, when the task to be processed is a log data retrieval task, the task to be processed includes the target log storage path, and the acquisition module 601, when acquiring the task to be processed, is specifically used for: The search interface is displayed, which includes a topic search control. When the aforementioned topic search control is triggered, one or more log topics will be displayed on the aforementioned search interface; When the target log topic is triggered, the log data retrieval task is generated based on the target log topic, which includes the business server identifier and the target log storage path.

[0150] Optionally, the search interface also includes a search arrangement control. When the acquisition module 601 is used, it is specifically used for: Optionally, the acquisition module 601 described above is also used for: When the above-mentioned arrangement and search control is triggered, one or more arrangements are displayed on the above-mentioned search interface; When the target orchestration is triggered, the log data retrieval task is generated according to the target orchestration. The target orchestration includes multiple target log topics, and the multiple target log topics include the business server identifier and the target log storage path.

[0151] Optionally, the above one or more log topics are configured through the log topic configuration interface, and the above one or more orchestrations are configured through the orchestration configuration interface.

[0152] Optionally, the above-mentioned retrieval interface also includes an execution order setting control, the above-mentioned target orchestration includes the original execution order, and the above-mentioned acquisition module 601, when used to generate the above-mentioned log data retrieval task according to the above-mentioned target orchestration, is specifically used for: When the above execution order setting control is triggered, the target execution order is determined according to the above execution order setting control; The original execution order in the above target orchestration is adjusted to the above target execution order. The above log data retrieval task is generated according to the adjusted target orchestration. The above log data retrieval task includes the above target execution order, which is used to instruct the above log processing program to perform log retrieval according to the above target execution order.

[0153] Optionally, the search interface also includes an execution order setting control. When there are multiple target log topics, the acquisition module 601, when generating the log data search task based on the target log topics, specifically performs the following: When the above execution order setting control is triggered, the target execution order is determined according to the above execution order setting control; The log data retrieval task is generated based on the above target execution order and the above multiple target log topics. The log data retrieval task includes the above target execution order, which is used to instruct the above log processing program to perform log retrieval in accordance with the above target execution order.

[0154] Optionally, when the above target execution order is a serial execution order, the above multiple target log topics include a first log topic and a second log topic, and the execution order of the first log topic is before that of the second log topic. The above processing result includes the execution result of the first log topic and the difference execution result between the execution result of the first log topic and the execution result of the second log topic. The difference execution result is the execution result after the log processing program filters the execution result of the second log topic based on the execution result of the first log topic.

[0155] Optionally, the search interface may also include a filter control, and the acquisition module 601 may further be used for: When the filtering parameters are obtained through the above-mentioned filtering control, the filtering parameters are sent to the log processing program on the above-mentioned business server. The filtering parameters are used to instruct the log processing program to perform the above-mentioned log data retrieval task according to the filtering parameters. The filtering parameters include keywords and environment attributes. The environment attributes include one or more of the test environment attributes and development environment attributes.

[0156] Optionally, the above-mentioned processing module 603 is also used for: During the process of the log data retrieval task being performed by the log processing program on the aforementioned business server, a progress acquisition request is periodically sent to the aforementioned log processing program so that the aforementioned log processing program can obtain the progress information of the aforementioned task to be processed. Receive the progress information sent by the log processing program on the aforementioned business server and display the progress information in the form of an image.

[0157] Optionally, when there are multiple business servers, each of the multiple business servers corresponds to a unit progress information, which is the minimum unit progress information, the maximum unit progress information, or the average unit progress information among the multiple unit progress information corresponding to the multiple business servers.

[0158] Optionally, when the sending module 602 is used to send the task to be processed and the log processing program corresponding to the task to be processed to the business server corresponding to the business server identifier, it is specifically used for: The above-mentioned task to be processed is sent to the business server corresponding to the above-mentioned business server identifier, so that the above-mentioned business server can calculate the first message digest of the above-mentioned task to be processed. Obtain the first message digest sent by the aforementioned business server, and calculate the second message digest of the aforementioned task to be processed; If the first message digest and the second message digest are the same, the log processing program corresponding to the task to be processed will be sent to the business server corresponding to the business server identifier.

[0159] Optionally, when the task to be processed is a log data analysis task, the log processing program integrates an analysis model, and the acquisition module 601, when used to acquire the task to be processed, is specifically used for: The analysis configuration interface is displayed. When the information of the device to be analyzed is obtained through the analysis configuration interface, the log data analysis task is generated based on the information of the device to be analyzed. The information of the device to be analyzed is used by the log processing program to analyze the log data of the device to be analyzed by calling the analysis model. The information of the device to be analyzed includes one or more of the following: network address, login time, login device, and device operation.

[0160] The aforementioned processing module 603 is also used for: An anomaly report is generated based on the above processing results. The anomaly report includes the abnormal device information in the device information to be analyzed.

[0161] It should be noted that the functions of each functional module of the data processing device in this application embodiment can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0162] Please see Figure 7 , Figure 7This is a schematic block diagram of a computer device according to an embodiment of this application. As shown in the figure, the smart terminal in this embodiment may include: a processor 701, a storage device 702, and a network interface 703. The processor 701, storage device 702, and network interface 703 can interact with each other.

[0163] The aforementioned storage device 702 may include volatile memory, such as random-access memory (RAM); the storage device 702 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the aforementioned storage device 702 may also include a combination of the above types of memory.

[0164] The processor 701 described above may be a central processing unit (CPU). In one embodiment, the processor 701 may also be a graphics processing unit (GPU). Alternatively, the processor 701 may be a combination of a CPU and a GPU. In one embodiment, the storage device 702 is used to store program instructions, and the processor 701 can invoke these program instructions to perform the following operations: Obtain tasks to be processed, including business server identifiers. The above-mentioned pending tasks and the corresponding log processing programs are sent to the business server corresponding to the above-mentioned business server identifier. The above-mentioned log processing programs are used to execute the above-mentioned pending tasks based on the log data stored in the above-mentioned business server. The system receives the processing result of the pending task after its completion from the log processing program on the aforementioned business server, outputs the processing result, and sends a destruction command to the log processing program on the aforementioned business server. The destruction command is used to destroy the log processing program on the aforementioned business server.

[0165] Optionally, when the task to be processed is a log data retrieval task, the task to be processed includes a target log storage path, and the processor 701, when used to obtain the task to be processed, is specifically used for: The search interface is displayed, which includes a topic search control. When the aforementioned topic search control is triggered, one or more log topics will be displayed on the aforementioned search interface; When the target log topic is triggered, the log data retrieval task is generated based on the target log topic, which includes the business server identifier and the target log storage path.

[0166] Optionally, the search interface also includes a search arrangement control, and the processor 701, when used, is specifically used for: Optionally, the processor 701 described above is also used for: When the above-mentioned arrangement and search control is triggered, one or more arrangements are displayed on the above-mentioned search interface; When the target orchestration is triggered, the log data retrieval task is generated according to the target orchestration. The target orchestration includes multiple target log topics, and the multiple target log topics include the business server identifier and the target log storage path.

[0167] Optionally, the above one or more log topics are configured through the log topic configuration interface, and the above one or more orchestrations are configured through the orchestration configuration interface.

[0168] Optionally, the above-mentioned retrieval interface also includes an execution order setting control. The above-mentioned target orchestration includes the original execution order. When the processor 701 generates the above-mentioned log data retrieval task according to the above-mentioned target orchestration, it is specifically used for: When the above execution order setting control is triggered, the target execution order is determined according to the above execution order setting control; The original execution order in the above target orchestration is adjusted to the above target execution order. The above log data retrieval task is generated according to the adjusted target orchestration. The above log data retrieval task includes the above target execution order, which is used to instruct the above log processing program to perform log retrieval according to the above target execution order.

[0169] Optionally, the above search interface also includes an execution order setting control. When there are multiple target log topics, the processor 701, when generating the log data retrieval task based on the target log topics, specifically uses the following: When the above execution order setting control is triggered, the target execution order is determined according to the above execution order setting control; The log data retrieval task is generated based on the above target execution order and the above multiple target log topics. The log data retrieval task includes the above target execution order, which is used to instruct the above log processing program to perform log retrieval in accordance with the above target execution order.

[0170] Optionally, when the above target execution order is a serial execution order, the above multiple target log topics include a first log topic and a second log topic, and the execution order of the first log topic is before that of the second log topic. The above processing result includes the execution result of the first log topic and the difference execution result between the execution result of the first log topic and the execution result of the second log topic. The difference execution result is the execution result after the log processing program filters the execution result of the second log topic based on the execution result of the first log topic.

[0171] Optionally, the search interface may also include a filter control, and the processor 701 may further be used for: When the filtering parameters are obtained through the above-mentioned filtering control, the filtering parameters are sent to the log processing program on the above-mentioned business server. The filtering parameters are used to instruct the log processing program to perform the above-mentioned log data retrieval task according to the filtering parameters. The filtering parameters include keywords and environment attributes. The environment attributes include one or more of the test environment attributes and development environment attributes.

[0172] Optionally, the processor 701 described above is also used for: During the process of the log data retrieval task being performed by the log processing program on the aforementioned business server, a progress acquisition request is periodically sent to the aforementioned log processing program so that the aforementioned log processing program can obtain the progress information of the aforementioned task to be processed. Receive the progress information sent by the log processing program on the aforementioned business server and display the progress information in the form of an image.

[0173] Optionally, when there are multiple business servers, each of the multiple business servers corresponds to a unit progress information, which is the minimum unit progress information, the maximum unit progress information, or the average unit progress information among the multiple unit progress information corresponding to the multiple business servers.

[0174] Optionally, when the processor 701 is used to send the task to be processed and the log processing program corresponding to the task to be processed to the business server corresponding to the business server identifier, it is specifically used for: The above-mentioned task to be processed is sent to the business server corresponding to the above-mentioned business server identifier, so that the above-mentioned business server can calculate the first message digest of the above-mentioned task to be processed. Obtain the first message digest sent by the aforementioned business server, and calculate the second message digest of the aforementioned task to be processed; If the first message digest and the second message digest are the same, the log processing program corresponding to the task to be processed will be sent to the business server corresponding to the business server identifier.

[0175] Optionally, when the task to be processed is a log data analysis task, the log processing program integrates an analysis model, and the processor 701, when used to acquire the task to be processed, is specifically used for: The analysis configuration interface is displayed. When the information of the device to be analyzed is obtained through the analysis configuration interface, the log data analysis task is generated based on the information of the device to be analyzed. The information of the device to be analyzed is used by the log processing program to analyze the log data of the device to be analyzed by calling the analysis model. The information of the device to be analyzed includes one or more of the following: network address, login time, login device, and device operation.

[0176] The aforementioned processor 701 is also used for: An anomaly report is generated based on the above processing results. The anomaly report includes the abnormal device information in the device information to be analyzed.

[0177] In specific implementations, the processor 701, storage device 702, and network interface 703 described in the embodiments of this application can execute the embodiments of this application. Figure 2 or Figure 4 The implementation methods described in the relevant embodiments of the provided data processing method can also be used to execute the embodiments of this application. Figure 6 The implementation methods described in the relevant embodiments of the provided data processing device will not be repeated here.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0179] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium storing a computer program executed by the aforementioned data processing device, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the aforementioned... Figure 2 , Figure 4The methods described in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same methods will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0180] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned... Figure 2 , Figure 4 The methods described in the corresponding embodiments will not be repeated here.

[0181] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0182] The above-disclosed embodiments are merely some of the embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application still fall within the scope of the invention.

Claims

1. A data processing method, characterized in that, The method includes: Obtain tasks to be processed, including business server identifiers; The task to be processed and the log processing program corresponding to the task to be processed are sent to the business server corresponding to the business server identifier. The log processing program is used to execute the task to be processed based on the log data stored in the business server. The system receives the processing result of the pending task after its completion from the log processing program on the business server, outputs the processing result, and sends a destruction command to the log processing program on the business server. The destruction command is used to destroy the log processing program on the business server. In the case where the task to be processed is a log data retrieval task, when the log data retrieval task includes multiple target log topics, there is a target execution order among the multiple target log topics. When the target execution order is a serial execution order, the multiple target log topics include a first log topic and a second log topic, and the execution order of the first log topic is before that of the second log topic. The processing result includes the execution result of the first log topic and the difference execution result between the execution result of the first log topic and the execution result of the second log topic. The difference execution result is the execution result after the log processing program filters the execution result of the second log topic based on the execution result of the first log topic.

2. The method according to claim 1, characterized in that, When the task to be processed is a log data retrieval task, the task to be processed includes a target log storage path, and obtaining the task to be processed includes: Display a search interface, which includes a topic search control; When the topic search control is triggered, one or more log topics are displayed on the search interface; When a target log topic is triggered, a log data retrieval task is generated based on the target log topic, which includes the business server identifier and the target log storage path.

3. The method according to claim 2, characterized in that, The search interface also includes search arrangement controls, and the method further includes: When the arrangement search control is triggered, one or more arrangements are displayed on the search interface; When a target orchestration is triggered, the log data retrieval task is generated based on the target orchestration. The target orchestration includes multiple target log topics, and the multiple target log topics include the business server identifier and the target log storage path.

4. The method according to claim 3, characterized in that, The one or more log topics are configured through the log topic configuration interface, and the one or more orchestrations are configured through the orchestration configuration interface.

5. The method according to claim 3, characterized in that, The search interface also includes an execution order setting control, the target orchestration includes the original execution order, and the step of generating the log data search task based on the target orchestration includes: When the execution order setting control is triggered, the target execution order is determined according to the execution order setting control; The original execution order in the target orchestration is adjusted to the target execution order, and the log data retrieval task is generated according to the adjusted target orchestration. The log data retrieval task includes the target execution order, which is used to instruct the log processing program to perform log retrieval according to the target execution order.

6. The method according to claim 2, characterized in that, The search interface also includes an execution order setting control. When there are multiple target log topics, generating the log data search task based on the target log topics includes: When the execution order setting control is triggered, the target execution order is determined according to the execution order setting control; The log data retrieval task is generated based on the target execution order and the plurality of target log topics. The log data retrieval task includes the target execution order, which is used to instruct the log processing program to perform log retrieval in accordance with the target execution order.

7. The method according to any one of claims 2 to 6, characterized in that, The search interface also includes a filter control, and the method further includes: When the filtering parameters are obtained through the filtering control, the filtering parameters are sent to the log processing program on the business server. The filtering parameters are used to instruct the log processing program to perform the log data retrieval task according to the filtering parameters. The filtering parameters include keywords and environment attributes. The environment attributes include one or more of test environment attributes and development environment attributes.

8. The method according to any one of claims 2 to 6, characterized in that, The method further includes: During the process of the log data retrieval task being executed by the log processing program on the business server, a progress acquisition request is periodically sent to the log processing program so that the log processing program can obtain the progress information of the task to be processed. The system receives the progress information sent by the log processing program on the business server and displays the progress information in the form of an image.

9. The method according to claim 8, characterized in that, When there are multiple service servers, each of the multiple service servers corresponds to a unit progress information. The progress information is the minimum unit progress information, the maximum unit progress information, or the average unit progress information among the multiple unit progress information corresponding to the multiple service servers.

10. The method according to claim 1, characterized in that, The step of sending the task to be processed and the log processing program corresponding to the task to the business server corresponding to the business server identifier includes: The task to be processed is sent to the business server corresponding to the business server identifier, so that the business server can calculate the first message digest of the task to be processed. Obtain the first message digest sent by the business server, and calculate the second message digest of the task to be processed; If the first message digest and the second message digest are the same, then the log processing program corresponding to the task to be processed is sent to the business server corresponding to the business server identifier.

11. The method according to claim 1, characterized in that, When the task to be processed is a log data analysis task, the log processing program integrates an analysis model, and the process of obtaining the task to be processed includes: The analysis configuration interface is displayed. When the device information to be analyzed is obtained through the analysis configuration interface, the log data analysis task is generated based on the device information to be analyzed. The device information to be analyzed is used by the log processing program to analyze the log data of the device information to be analyzed by calling the analysis model. The device information to be analyzed includes one or more of the following: network address, login time, login device, and device operation. The method further includes: An anomaly report is generated based on the processing results. The anomaly report includes the abnormal device information in the device information to be analyzed.

12. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire tasks to be processed, including business server identifiers. The sending module is used to send the task to be processed and the log processing program corresponding to the task to be processed to the business server corresponding to the business server identifier. The log processing program is used to execute the task to be processed according to the log data stored in the business server. The processing module is used to receive the processing result of the task to be processed after it has been completed, sent by the log processing program on the business server, output the processing result, and send a destruction command to the log processing program on the business server, the destruction command being used to destroy the log processing program on the business server. In the case where the task to be processed is a log data retrieval task, when the log data retrieval task includes multiple target log topics, there is a target execution order among the multiple target log topics. When the target execution order is a serial execution order, the multiple target log topics include a first log topic and a second log topic, and the execution order of the first log topic is before that of the second log topic. The processing result includes the execution result of the first log topic and the difference execution result between the execution result of the first log topic and the execution result of the second log topic. The difference execution result is the execution result after the log processing program filters the execution result of the second log topic based on the execution result of the first log topic.

13. A computer device, characterized in that, include: A memory and a processor, wherein the memory stores a data processing program, which, when executed by the processor, is used to implement the data processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which are executed by a processor to implement the data processing method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-11.

Images