Log processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a log processing method and device, electronic equipment and a storage medium, and is applied to the technical field of information, the method comprises the steps that a first notification message of a log processing module in a first server is received, and the first notification message is used for representing whether the log processing module completes processing of logs in a storage space or not; when it is determined that the log processing module does not complete processing of the logs in the storage space based on the first notification message, the first server is controlled to be offline, the storage space is mounted to a second server, so that the second server continues to process the logs in the storage space, the second server replaces the first server, and the log processing module does not complete processing of the logs in the storage space. And the cost required by the second server for log processing is lower than that required by the first server for log processing. Through the log processing method and device, the electronic equipment and the storage medium provided by the embodiment of the invention, the cost of log processing can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular, to a log processing method, apparatus, electronic device, and storage medium. Background Art

[0002] In a K8s (Kubernetes) cluster, generally, business logs are written into a specified directory of a host machine through a local mounting method, and a log collection agent on the host machine reads, analyzes, and uploads the files in the directory. However, the processing speed of the agent is limited, and when there are a large number of business logs, there will be obvious delays in the agent. In a cloud environment, when some host machines in the K8s cluster are recycled due to low utilization, for example, during some periods with less business volume, the utilization rate of the K8s cluster is low, and at this time, some host machines in the K8s cluster need to be recycled. The logs on these host machines may not have been processed yet; usually, the overall specifications of K8s host machines are large, expensive, and the models are not dedicated to the log processing scenario, and the agent cannot utilize all resources such as CPU (Central Processing Unit) and memory, and there will be more cost waste if waiting for the log processing to be completed. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a log processing method, apparatus, electronic device, and storage medium to reduce the cost of log processing. The specific technical solutions are as follows:

[0004] In the first aspect of the present invention, a log processing method is provided, including:

[0005] Receiving a first notification message from a log processing module in a first server, where the first notification message is used to indicate whether the log processing module has completed processing the logs in the storage space. The first notification message is sent after the log processing module receives a second notification message, and the second notification message is used to inform the log processing module that the first server is to be taken offline, so that the log processing module detects whether it has completed processing the logs in the storage space after receiving the second notification message. The storage space is used to store the logs generated by the service;

[0006] When it is determined based on the first notification message that the log processing module has not completed processing the logs in the storage space, controlling the first server to go offline and mounting the storage space to a second server, so that the second server continues to process the logs in the storage space, where the second server replaces the first server, and the cost required for the second server to process logs is lower than the cost required for the first server to process logs.

[0007] Optionally, mounting the storage space to the second server includes:

[0008] Determine whether there is an idle second server in the resource pool, where the resource pool is used to store second servers;

[0009] If there is an idle second server, use the idle second server as the target second server and mount the storage space to the target second server, so that the target second server continues to process the logs in the storage space.

[0010] Optionally, mounting the storage space to the second server includes:

[0011] Determine whether there is an idle second server in the resource pool, where the resource pool is used to store second servers;

[0012] If there is no idle second server, create a new second server, use the new second server as the target second server, and mount the storage space to the target second server, so that the target second server continues to process the logs in the storage space.

[0013] Optionally, if there is no idle second server, creating a new second server includes:

[0014] If there is no idle second server, determine whether the scale of the current resource pool has reached the upper limit of the resource pool scale;

[0015] If the scale of the current resource pool has not reached the upper limit of the resource pool scale, create a new second server.

[0016] Optionally, if the scale of the current resource pool has reached the upper limit of the resource pool scale, save the attribute information of the storage space to a queue. The queue is used to mount the storage space to an idle second server according to the attribute information of the storage space in the queue after there is an idle second server. The attribute information of the storage space represents the own characteristics of the storage space.

[0017] Optionally, after mounting the storage space to the target second server, the method further includes:

[0018] When it is determined that the target second server has completed processing the logs in the storage space, recycle the storage space and mark the target second server as idle.

[0019] Optionally, mounting the storage space to the second server includes:

[0020] Save the attribute information of the storage space to a queue, so that an idle second server can obtain the attribute information of the storage space from the queue and mount the storage space to the second server based on the attribute information of the storage space.

[0021] Optionally, the queue is used to save the attribute information of multiple storage spaces;

[0022] After saving the attribute information of the storage space to the queue, the method further includes:

[0023] If it is determined that the idle second server has not obtained the attribute information of the storage space to be processed from the queue within a preset time range, recycle the idle second server.

[0024] In a second aspect of the implementation of the present invention, a log processing device is provided, including:

[0025] A receiving module, configured to receive a first notification message from a log processing module in a first server, where the first notification message is used to indicate whether the log processing module has completed processing of logs in a storage space, and the first notification message is sent after the log processing module receives a second notification message, and the second notification message is used to inform the log processing module that the first server is to go offline, so that the log processing module detects whether it has completed processing of the logs in the storage space after receiving the second notification message, and the storage space is used to store logs generated by services;

[0026] A re-mounting module, configured to, when it is determined based on the first notification message that the log processing module has not completed processing of the logs in the storage space, control the first server to go offline and mount the storage space to a second server, so that the second server continues to process the logs in the storage space, where the second server replaces the first server, and the cost required for the second server to perform log processing is lower than the cost required for the first server to perform log processing.

[0027] In yet another aspect of the implementation of the present invention, an electronic device is further provided, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0028] The memory is used to store a computer program;

[0029] The processor is configured to, when executing the program stored on the memory, implement the log processing method described in any one of the above.

[0030] In yet another aspect of the implementation of the present invention, there is also provided a computer-readable storage medium storing a computer program therein, and when the computer program is executed by a processor, the above-mentioned log processing method is implemented.

[0031] In yet another aspect of the implementation of the present invention, there is also provided a computer program product containing instructions, which when run on a computer, causes the computer to execute the above-mentioned log processing method.

[0032] In an embodiment of the present invention, a first notification message from a log processing module in a first server is received. The first notification message is used to indicate whether the log processing module has completed processing the logs in the storage space. The first notification message is sent after the log processing module receives a second notification message. The second notification message is used to inform the log processing module that the first server is to be taken offline, so that the log processing module detects whether it has completed processing the logs in the storage space when it receives the second notification message. The storage space is used to store the logs generated by the service; when it is determined based on the first notification message that the log processing module has not completed processing the logs in the storage space, the first server is controlled to go offline, and the storage space is mounted to a second server, so that the second server can continue to process the logs in the storage space. In this way, when the first server is to be taken offline, if its log processing module has not completed processing the logs in the storage space, the first server is controlled to go offline and the second server with a lower cost for log processing is used to replace the first server with a higher cost for log processing. The second server processes the logs in the storage space, and there is no need for the first server with a higher cost to wait only for processing logs, thus reducing the cost of log processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art.

[0034] Figure 1 It is a flowchart of a log processing method provided by an embodiment of the present invention;

[0035] Figure 2 It is another flowchart of a log processing method provided by an embodiment of the present invention;

[0036] Figure 3 It is a schematic diagram of the process of the service normally using the first server provided by an embodiment of the present invention;

[0037] Figure 4 It is a schematic diagram of the process of the service no longer using the first server provided by an embodiment of the present invention;

[0038] Figure 5An example flowchart of the log processing method provided by the embodiments of the present invention;

[0039] Figure 6 A schematic structural diagram of the log processing device provided by the embodiments of the present invention;

[0040] Figure 7 A schematic structural diagram of the electronic device provided by the embodiments of the present invention. Detailed implementation manners

[0041] Next, the technical solutions in the embodiments of the present invention will be described with reference to the accompanying drawings in the embodiments of the present invention.

[0042] The embodiments of the present invention provide a log processing method, as Figure 1 shown, which may include:

[0043] S101. Receive a first notification message from a log processing module in a first server. The first notification message is used to indicate whether the log processing module has completed processing the logs in the storage space. The first notification message is sent after the log processing module receives a second notification message. The second notification message is used to inform the log processing module that the first server is to be taken offline, so that the log processing module detects whether it has completed processing the logs in the storage space after receiving the second notification message. The storage space is used to store the logs generated by the service;

[0044] S102. When it is determined based on the first notification message that the log processing module has not completed processing the logs in the storage space, control the first server to go offline and mount the storage space to a second server, so that the second server continues to process the logs in the storage space. Here, the second server replaces the first server, and the cost required for the second server to process the logs is lower than the cost required for the first server to process the logs.

[0045] In an embodiment of the present invention, a first notification message from a log processing module in a first server is received. The first notification message is used to indicate whether the log processing module has completed processing the logs in the storage space. The first notification message is sent after the log processing module receives a second notification message. The second notification message is used to inform the log processing module that the first server is to be taken offline, so that the log processing module detects whether it has completed processing the logs in the storage space when it receives the second notification message. The storage space is used to store the logs generated by the service. When it is determined based on the first notification message that the log processing module has not completed processing the logs in the storage space, the first server is controlled to go offline, and the storage space is mounted to a second server, so that the second server continues to process the logs in the storage space. In this way, when the first server is to be taken offline, if its log processing module has not completed processing the logs in the storage space, the first server is controlled to go offline and the second server with a lower cost for log processing is used to replace the first server with a higher cost for log processing. The second server processes the logs in the storage space, and there is no need for the first server with a higher cost to wait only for processing the logs. In this way, the cost of log processing is reduced.

[0046] The execution subject of the log processing method provided in the embodiment of the present invention may be an electronic device, or it can be understood that the log processing method provided in the embodiment of the present invention can be applied to an electronic device. Among them, the electronic device may be a server, a controller, etc. In one implementable manner, the controller may specifically be a log controller.

[0047] In an embodiment of the present invention, the first server may also be understood as the host on which the above-mentioned log processing module runs. The above-mentioned log processing module refers to the module in the first server for log processing. For example, it may be a log processing tool. In one example, the log processing module is a log collection agent on the host in K8s.

[0048] The second server is another server different from the first server, and the cost required for the second server to perform log processing is lower than the cost required for the first server to perform log processing.

[0049] Specifically, it can be understood that the second server is suitable for log processing, and the performance, price, etc. of the second server are lower than those of the first server.

[0050] Simply understood, the first server is a high-specification server (which can also be understood as a general-purpose server with higher performance and higher price), and the second server is a low-specification server (a server with lower performance and low cost). The second server has the ability to perform log processing.

[0051] When the execution entity detects that a certain first server in the system where the first server is located is about to go offline, it sends a notification message (the above-mentioned second notification message) to the log processing module of the first server to notify the log processing module of this event, that is, the event that the first server is to be taken offline. When the log processing module receives this notification message, that is, receives this event, it detects whether it has completed the processing of the logs in the storage space and sends a first notification message to the execution entity (indicating whether the log processing module has completed the processing of the logs in the storage space). After receiving this first notification message, the execution entity can determine whether the log processing module has completed the processing of the logs in the storage space. If it is determined that the log processing module has not completed the processing of the logs in the storage space, it controls the first server to go offline and mounts the storage space to the second server. In this way, the second server continues to process the logs in the storage space. That is, the second server replaces the above-mentioned first server and continues to process the logs in the storage space. Specifically, it continues to process the logs that the above-mentioned log processing module has not completed.

[0052] The processing of the logs in the storage space that the log processing module of the first server has not completed, and the processing by the second server of the logs in the storage space can include reading, analyzing, uploading, etc.

[0053] For example, in K8s, business logs are written into a specified directory of the host machine (such as the first server) through the local mounting method, and the log collection agent on the host machine reads, analyzes, and uploads the files in the directory. In the actual application process, the first server stores the logs written by the business into the storage space, that is, the above-mentioned specified directory is in the storage space, and the log collection agent on the host machine reads, analyzes, and uploads the files in the directory. The reading, analyzing, and uploading of the files in the directory here are also the processing of the logs in the storage space.

[0054] Among them, the second server's continued processing of the logs in the storage space can specifically include the processing of the logs in the storage space that the log processing module has not completed. The logs here can include reading, analyzing, uploading, etc. That is, for the logs written by the business into the storage space, during the process of reading, analyzing, uploading, etc. of them, when the log processing module of the first server has not completed the above-mentioned processing of the logs in the storage space, the second server continues to perform the above-mentioned processing on the unfinished part.

[0055] The storage space is the storage space for storing business logs. The business writes logs into the storage space. Specifically, a directory can be specified in the storage space, and the business can write logs into the specified directory. Furthermore, logs can be read from the storage space, or the logs in the storage space can be analyzed, or the logs in the storage space can be uploaded to other devices as needed. Here is only an example of processing the logs in the storage space. The specific processing method can be determined according to actual requirements, etc. The specific processing method can be only one, or can also include multiple ones, such as reading, analyzing, uploading, etc.

[0056] The storage space can specifically be a storage block. In one implementation, the storage space can be understood as a removable storage device, specifically, it can be understood that the storage space can be detachably installed on different devices. In one example, the storage space is specifically a cloud disk.

[0057] The processing progress of the log processing module of the first server on the storage space can be recorded in the storage space, such as marking the processed logs or the division between the processed logs and the unprocessed logs. Thus, when the storage space is mounted to the second server, the second server can continue to process the unprocessed logs according to the processing progress recorded in the storage space. Or, the log processing module of the first server records its processing progress on the logs in the storage space, and when the execution entity mounts the storage space to the second server, it can send the processing progress recorded by the log processing module of the first server to the second server at the same time. Thus, the second server can continue to process the unprocessed logs according to the received processing progress.

[0058] Among them, the execution entity controls the first server to go offline. In one implementation, the execution entity notifies the node controller, and the node controller realizes making the first server go offline. Specifically, the execution entity can send a go-offline instruction to the node controller, and the go-offline instruction carries the identification information of the first server. The node controller receives the go-offline instruction and reclaims the first server, making the first server go offline. The node controller can be understood as a module or device dedicated to reclaiming nodes, etc.

[0059] In an optional embodiment, as Figure 2 shown, mounting the storage space to the second server in S102 can include:

[0060] Judge whether there is an idle second server in the resource pool, where the resource pool is used to save second servers; if there is an idle second server, use the idle second server as the target second server, and mount the storage space to the target second server so that the target second server continues to process the logs in the storage space.

[0061] The initial number of the second servers in the resource pool is 0, and it can also be reduced to 0 when there are no tasks to be processed.

[0062] After determining whether there is an idle second server in the resource pool, if there is no idle second server, a new second server is created, the new second server is used as the target second server, and the storage space is mounted to the target second server so that the target second server can continue to process the logs in the storage space.

[0063] In one case, if it is determined that there is no idle second server, at this time, first determine whether the scale of the current resource pool reaches the upper limit of the resource pool scale; if the scale of the current resource pool does not reach the upper limit of the resource pool scale, then create a new second server;

[0064] Among them, the upper limit of the resource pool scale is also a preset threshold for restricting the number of second servers in the resource pool.

[0065] Determining whether the scale of the current resource pool reaches the upper limit of the resource pool scale specifically means determining whether the number of second servers in the current resource pool reaches the preset threshold.

[0066] This preset threshold can be determined according to actual requirements, experience, etc., such as 10, 15, 20, etc.

[0067] If the scale of the current resource pool has reached the upper limit of the resource pool scale, the attribute information of the storage space is saved to the queue.

[0068] The queue is used to mount the storage space to the idle second server according to the attribute information of the storage space in the queue after there is an idle second server.

[0069] The attribute information of the storage space represents the self-characteristics of the storage space. Among them, the self-characteristics of the storage space can include characteristics in multiple dimensions of the storage space, such as identification, capacity, task volume, etc. The identification can be the ID of the storage space, the capacity can be the capacity of the storage space to store data, the task volume can be the number of logs stored in the storage space, and the number of logs mentioned here can be the number of logs that the log processing module of the first server has not completed, that is, the number of logs that the second server needs to continue to process.

[0070] In the embodiments of the present invention, only when the above-mentioned steps have no spare capacity to process (that is, there is no idle second server in the resource pool and the scale of the current resource pool has reached the upper limit of the resource pool scale), the attribute information of the storage space is put into the queue to wait for the idle node to process. In this case, the execution entity controls the scheduling of the second servers in the entire resource pool.

[0071] After mounting the storage space to the target second server as described above, the following steps may further be included:

[0072] When it is determined that the target second server has completed the processing of the logs in the storage space, the storage space is recycled, and the target second server is marked as idle.

[0073] In this way, on the one hand, the storage space where the log processing has been completed is recycled, which can save storage resources; on the other hand, by marking the idle state of the idle nodes (second servers), the execution entity can quickly and conveniently detect the idle nodes to mount the unfinished tasks to the idle nodes, improving the rate of the entire system's log processing.

[0074] In another alternative embodiment, mounting the storage space to the second server in S102 may include:

[0075] Save the attribute information of the storage space to the queue, so that the idle second server can obtain the attribute information of the storage space from the queue and mount the storage space to the second server based on the attribute information of the storage space.

[0076] Compared with Figure 2 In the embodiment shown, the attribute information of the storage space is saved to the queue when it meets the conditions that there is no idle second server in the resource pool and the scale of the current resource pool has reached the upper limit of the resource pool scale. The embodiment of the present invention provides another processing method, that is, during the process of mounting the storage space to the second server, when there is a task that the logs of the storage space need to be continued to be processed by the second server, the attribute information of the storage space is directly saved to the queue. That is, the steps of judging whether there is an idle second server in the resource pool and whether the scale of the current resource pool has reached the upper limit of the resource pool scale are not executed. When there is a task that the logs of the storage space need to be continued to be processed by the second server, the attribute information of the storage space is saved to the queue.

[0077] The queue is used to save the attribute information of multiple storage spaces.

[0078] In this case, it can be understood that the execution entity only needs to save the attribute information of the storage space to the queue when there is a task that the logs of the storage space need to be continued to be processed by the second server. It is the second server that actively obtains the attribute information of the storage space from the queue and mounts the storage space to the second server based on the attribute information of the storage space. That is, the second server obtains the task to execute from the queue after it becomes idle itself.

[0079] In one implementable way, after saving the attribute information of the storage space to the queue, it can also be determined that the idle second server has not obtained the attribute information of the storage space to be processed from the queue within the preset time range, and then the idle second server is recycled.

[0080] The preset time range can be determined according to actual requirements, experience, etc.

[0081] In a specific embodiment, the execution subject of the embodiment of the present invention is a log controller, which is applied to a K8s cluster. The storage space is specifically a cloud disk.

[0082] In the embodiment of the present invention, after the service is scheduled away and the log writing stops, the cloud disk used to store the logs is remounted to a low-specification, inexpensive server suitable for log collection.

[0083] As Figure 3 shown, when the service normally uses the server (the server here is the above-mentioned first server, that is, the high-specification server), service X and service Y respectively write their respective service logs into the cloud disk, and the cloud disk is used to store the logs. At this time, the cloud disk is mounted on the first server, and the log collection tool in the first server reads the logs from the cloud disk.

[0084] In one case, it may be necessary to recycle the first server. For example, K8s needs to scale down and take the first server offline. At this time, if the log collection tool of the first server has not completed the processing of the logs in the cloud disk, the log processing method provided by the embodiment of the present invention can be adopted. As Figure 4 shown, mount the above cloud disk to the above second server (i.e., the low-specification server) to complete the subsequent log collection. At the same time, recycle the above high-specification server. At this time, the high-specification server can also be understood as the original server.

[0085] When the log controller detects that K8s is about to take a certain host offline, the log controller notifies the log collection agent of this event, and the log collection agent detects whether it has completed the collection of the local logs (the logs mounted in the cloud disk of the first server).

[0086] If the log collection agent has completed the above log collection, it notifies the log controller, and the log controller notifies the node controller to normally recycle the node and destroy the first server and the cloud disk;

[0087] If the log collection agent has not completed the above log collection, it notifies the log controller, and the log controller notifies the node controller to only destroy the first server and retain the cloud disk. The log controller puts the cloud disk information into the queue. The cloud disk information is the attribute information of the above storage space, which here refers to the attribute information of the cloud disk, and specifically may include the ID of the cloud disk, or may also include the number of unprocessed logs in the cloud disk, etc.

[0088] Among them, the log controller creates min(queue length, maximum concurrency) small-scale log processing servers suitable for log collection (i.e., the above-mentioned low-specification servers) as needed, obtains the cloud disk information to be processed from the queue, releases and destroys the cloud disk after processing; min can be understood as the number of low-specification processing servers. In one example, the queue length for storing the attribute information of the cloud disk can also be min; at the same time, min can also be understood as the maximum number of processors allowed to run simultaneously.

[0089] When the low-specification log processing server is idle, it obtains the cloud disk information to be processed from the queue. If it fails to obtain the information after exceeding an idle time threshold, the low-specification log processing server itself will be destroyed.

[0090] In a public cloud, the server and the cloud disk can be managed separately. When the server is recycled, the cloud disk can be retained, and the same cloud disk can be mounted to different servers successively.

[0091] The embodiments of the present invention can be applied to a K8s server cluster with ultra-large scale within a company and enabling elastic scaling of the number of server nodes to reduce the cost of the log collection closing work. The log processing method provided by the embodiments of the present invention supports the application log processing of the K8s cluster with elastic scaling.

[0092] After the service is scheduled away and the log writing stops, the cloud disk used to save the log is promptly remounted to a low-specification, inexpensive server suitable for log collection, and the original high-specification server is recycled. Avoid occupying high-specification and expensive general-purpose servers for a long time to complete the log collection closing work.

[0093] In a specific example, as Figure 5 shown, the following steps may be included:

[0094] 1. Start;

[0095] 2. Service expansion;

[0096] 3. Create a high-specification server;

[0097] 4. Create a cloud disk;

[0098] 5. Mount the cloud disk to the high-specification server;

[0099] 6. Service usage.

[0100] Steps 1 to 6 can be understood as the existing business environment.

[0101] 7. Service contraction;

[0102] In the actual application process, there may be a need for business downsizing. The embodiments of the present invention are processing solutions when there is a need for business downsizing and some of the above-mentioned high-specification servers need to be recycled.

[0103] 8. Recycle the high-specification server and retain the cloud disk;

[0104] 9. Determine whether there are idle machines in the resource pool;

[0105] If yes, continue to step 10; if no, execute step 15.

[0106] The resource pool here can also be referred to as a low-specification server resource pool, and the idle machine is the above-mentioned idle second server.

[0107] 10. Mount the cloud disk to the idle node and continue data collection;

[0108] The idle node is the above-mentioned idle machine.

[0109] 11. After data collection is completed, recycle the cloud disk and mark the node as idle;

[0110] 12. Determine whether the queue is empty;

[0111] If yes, execute step 13; if no, execute step 14.

[0112] The queue here can also be referred to as an idle node waiting queue.

[0113] 13. Recycle the data collection node;

[0114] The data collection node is the above-mentioned second server for log processing.

[0115] 14. Take a cloud disk from the queue, mount it and collect logs;

[0116] 15. Determine whether the resource pool scale upper limit is reached;

[0117] If no, execute step 16; if yes, execute step 17;

[0118] 16. Create a new node;

[0119] After executing step 16, return to step 10.

[0120] 17. Enter the queue.

[0121] In one case, if it is determined that there is no idle second server, first determine whether the scale of the current resource pool has reached the upper limit of the resource pool scale; if the scale of the current resource pool has not reached the upper limit of the resource pool scale, create a new second server; and if the scale of the current resource pool has reached the upper limit of the resource pool scale, save the attribute information of the storage space to the queue. That is the implementation process of step 17 above.

[0122] In this case, the log controller controls the scheduling of the second servers in the entire resource pool. Only when the foregoing steps have no remaining capacity to handle (that is, there is no idle second server in the resource pool and the scale of the current resource pool has reached the upper limit of the resource pool scale), the attribute information of the storage space is put into the queue to wait for the idle node to process.

[0123] In another case, during the process of mounting the storage space to the second server, when there is a task that requires the second server to continue processing the log of the storage space, directly save the attribute information of the storage space to the queue.

[0124] In this case, it can be understood that the log controller only needs to save the attribute information of the storage space to the queue when there is a task that requires the second server to continue processing the log of the storage space. It is the second server that actively obtains the attribute information of the storage space from the queue and mounts the storage space to the second server based on the attribute information of the storage space. That is, the second server obtains tasks from the queue for execution after it becomes idle itself.

[0125] When in the scenario of recycling some host machines in the K8s cluster, if the log collection agent on the host machine has not completed the processing such as reading, analyzing, and uploading the logs in the storage space, waiting for the log collection agent on the host machine to complete the log processing before recycling the host machine will result in high - specification servers being only used for log processing, making the log processing cost relatively high. For example, before business scaling down, the first server is also used to implement other business functions, and the above - mentioned log processing process is only a part of its functions. In actual use, it is possible that due to a decrease in business volume, etc., there is no need for too many servers to implement business functions. At this time, it may be necessary to take some first servers offline. If the first servers that originally implemented both business functions and log processing are not taken offline first and are only used for log processing, it will also cause resource waste. The embodiment of the present invention enables the second server to continue the unfinished log processing process, that is, to perform the log finishing work, and can also reduce resource waste.

[0126] The embodiment of the present invention saves the cost of processing this part of the unfinished logs, and can also be understood as reducing the cost of the log collection finishing work. It can also be understood that in the embodiment of the present invention, the free combination ability of public cloud servers and cloud disks is utilized to reduce the cost of the log collection finishing work.

[0127] Corresponding to the above log processing method, an embodiment of the present invention further provides a log processing apparatus, as Figure 6 shown, including:

[0128] A receiving module 601, configured to receive a first notification message from a log processing module in a first server, where the first notification message is used to indicate whether the log processing module has completed processing of the logs in the storage space. The first notification message is sent after the log processing module receives a second notification message, and the second notification message is used to inform the log processing module that the first server is to be taken offline, so that the log processing module detects whether it has completed processing of the logs in the storage space after receiving the second notification message. The storage space is used to store the logs generated by the service;

[0129] A re-mounting module 602, configured to, when it is determined based on the first notification message that the log processing module has not completed processing of the logs in the storage space, control the first server to go offline and mount the storage space to a second server, so that the second server continues to process the logs in the storage space, where the second server replaces the first server, and the cost required for the second server to perform log processing is lower than the cost required for the first server to perform log processing.

[0130] Optionally, the re-mounting module 602 is specifically configured to determine whether there is an idle second server in the resource pool, where the resource pool is used to save the second servers; if there is an idle second server, use the idle second server as the target second server and mount the storage space to the target second server, so that the target second server continues to process the logs in the storage space.

[0131] Optionally, the re-mounting module 602 is specifically configured to determine whether there is an idle second server in the resource pool, where the resource pool is used to save the second servers; if there is no idle second server, create a new second server, use the new second server as the target second server, and mount the storage space to the target second server, so that the target second server continues to process the logs in the storage space.

[0132] Optionally, the re-mounting module 602 is specifically configured to, if there is no idle second server, determine whether the scale of the current resource pool reaches the upper limit of the resource pool scale; if the scale of the current resource pool does not reach the upper limit of the resource pool scale, create a new second server.

[0133] Optionally, the re-mounting module 602 is specifically configured to, if the scale of the current resource pool has reached the upper limit of the resource pool scale, save the attribute information of the storage space to a queue, where the queue is used to mount the storage space to an idle second server according to the attribute information of the storage space in the queue after there is an idle second server, and the attribute information of the storage space represents the own characteristics of the storage space.

[0134] Optionally, the apparatus further includes:

[0135] A first processing module, configured to, after mounting the storage space to the target second server, when it is determined that the target second server has completed processing the logs in the storage space, recycle the storage space and mark the target second server as idle.

[0136] Optionally, the re-mounting module 602 is specifically configured to save the attribute information of the storage space to a queue, so that an idle second server can obtain the attribute information of the storage space from the queue and mount the storage space to the second server based on the attribute information of the storage space.

[0137] Optionally, the queue is used to save the attribute information of multiple storage spaces;

[0138] The apparatus further includes:

[0139] A second processing module, configured to, after saving the attribute information of the storage space to the queue, determine that an idle second server has not obtained the attribute information of the storage space to be processed from the queue within a preset time range, and recycle the idle second server.

[0140] An embodiment of the present invention further provides an electronic device, as Figure 7 shown, including a processor 701, a communication interface 702, a memory 703, and a communication bus 704. Among them, the processor 701, the communication interface 702, and the memory 703 complete mutual communication through the communication bus 704.

[0141] The memory 703 is used to store a computer program;

[0142] The processor 701, when executing the program stored on the memory 703, implements the method steps of the above-mentioned log processing method.

[0143] The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is used in the figure to represent it, but it does not mean that there is only one bus or one type of bus.

[0144] The communication interface is used for communication between the above terminal and other devices.

[0145] The memory may include a Random Access Memory (RAM), or may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0146] The aforementioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0147] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the log processing method described in any one of the above embodiments is implemented.

[0148] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute the log processing method described in any one of the above embodiments.

[0149] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0150] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0151] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the apparatus, electronic device, storage medium, and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the partial description of the method embodiments for the relevant parts.

[0152] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A log processing method, characterized in that, Including: Receiving a first notification message from a log processing module in a first server, where the first notification message is used to indicate whether the log processing module has completed processing of logs in a storage space. The first notification message is sent after the log processing module receives a second notification message, and the second notification message is used to inform the log processing module that the first server is to be taken offline, so that the log processing module detects whether it has completed processing of the logs in the storage space when receiving the second notification message. The storage space is used to store logs generated by services; When it is determined based on the first notification message that the log processing module has not completed processing of the logs in the storage space, controlling the first server to go offline and mounting the storage space to a second server, so that the second server continues to process the logs in the storage space, where the second server replaces the first server, and the cost required for the second server to perform log processing is lower than the cost required for the first server to perform log processing.

2. The method according to claim 1, characterized in that, The mounting the storage space to the second server includes: Judging whether there is an idle second server in a resource pool, where the resource pool is used to save second servers; If there is an idle second server, using the idle second server as the target second server and mounting the storage space to the target second server, so that the target second server continues to process the logs in the storage space.

3. The method according to claim 1, wherein The mounting the storage space to the second server includes: Judging whether there is an idle second server in a resource pool, where the resource pool is used to save second servers; If there is no idle second server, creating a new second server, using the new second server as the target second server and mounting the storage space to the target second server, so that the target second server continues to process the logs in the storage space.

4. The method according to claim 3, characterized in that, The creating a new second server if there is no idle second server includes: If there is no idle second server, judging whether the scale of the current resource pool has reached the upper limit of the resource pool scale; If the scale of the current resource pool has not reached the upper limit of the resource pool scale, creating a new second server.

5. The method according to claim 4, wherein If the scale of the current resource pool has reached the upper limit of the resource pool scale, saving the attribute information of the storage space to a queue, where the queue is used to mount the storage space to an idle second server according to the attribute information of the storage space in the queue after there is an idle second server. The attribute information of the storage space represents the self-characteristics of the storage space.

6. The method according to any one of claims 2 to 5, characterized in that, After mounting the storage space to the target second server, the method further includes: When it is determined that the target second server has completed processing of the logs in the storage space, recovering the storage space and marking the target second server as idle.

7. The method according to claim 1, characterized in that, The mounting the storage space to the second server includes: Save the attribute information of the storage space to a queue, so that an idle second server can obtain the attribute information of the storage space from the queue and mount the storage space to the second server based on the attribute information of the storage space.

8. The method according to claim 7, wherein The queue is used to save the attribute information of multiple storage spaces; After saving the attribute information of the storage space to the queue, the method further includes: If it is determined that the idle second server has not obtained the attribute information of the storage space to be processed from the queue within a preset time range, recycle the idle second server.

9. A log processing device, characterized in that, Includes: A receiving module, configured to receive a first notification message from a log processing module in a first server, where the first notification message is used to indicate whether the log processing module has completed processing of the logs in the storage space. The first notification message is sent after the log processing module receives a second notification message, and the second notification message is used to inform the log processing module that the first server is to go offline, so that the log processing module detects whether it has completed processing of the logs in the storage space after receiving the second notification message. The storage space is used to store the logs generated by the service; A re-mounting module, configured to, when it is determined based on the first notification message that the log processing module has not completed processing of the logs in the storage space, control the first server to go offline and mount the storage space to a second server, so that the second server continues to process the logs in the storage space, where the second server replaces the first server, and the cost required for the second server to process the logs is lower than the cost required for the first server to process the logs.

10. An electronic device, characterized in that, Includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor, when executing the program stored on the memory, implements the method steps described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method steps described in any one of claims 1-8.

Images