Log data processing methods, devices, systems, equipment and media

Abstract

This disclosure provides a log data processing method, apparatus, system, device, medium, and program product, relating to the field of computer technology, and particularly to cloud computing and function computing technologies. The specific implementation scheme is as follows: the method is applied to a machine, where multiple instances run. The method includes: acquiring metadata, wherein the metadata includes resource scaling parameters and a code package, the resource scaling parameters being used to measure the timing of scaling up or down, and the code package including log data processing rules; running the code package through any target instance among the multiple instances, wherein the machine has a runtime deployed, the runtime providing a framework for running the log data processing program, and the target instance running within the framework provided by the runtime; monitoring the runtime's operating metrics, and scaling up or down the machine based on the operating metrics and resource scaling parameters. This disclosure enables flexible scaling up or down by monitoring the runtime's operating metrics.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to cloud computing and function computing technologies, specifically to a log data processing method, apparatus, system, device, medium, and program product. Background Technology

[0002] As the number of service access requests increases, the volume of log collection and processing becomes increasingly large. Current online service log processing typically involves logging logs and installing a log collection agent on the machine, which then reports and stores the logs.

[0003] When service traffic is stable, the above method allows engineers to accurately predict the infrastructure environment based on traffic volume and thus deploy machines and components in advance. However, it cannot meet current log processing needs in scenarios with significant peak and trough traffic. Summary of the Invention

[0004] This disclosure provides a log data processing method, apparatus, system, device, medium, and program product.

[0005] According to one aspect of this disclosure, a log data processing method is provided, applied to a machine in which multiple instances are running, the method comprising:

[0006] Obtain metadata, which includes resource scaling parameters and code packages. The resource scaling parameters are used to measure when to scale up or down, and the code packages include log data processing rules.

[0007] The code package is run by any target instance among the plurality of instances, wherein the machine is deployed with a runtime that provides a framework for running the log data processing program, and the target instance runs within the framework provided by the runtime;

[0008] Monitor the runtime performance metrics and scale the machine up or down based on the performance metrics and resource scaling parameters.

[0009] According to another aspect of this disclosure, a log data processing method is provided, applied to a central control service device, the central control service device being connected to multiple machines respectively, the method comprising:

[0010] Monitor the update status of metadata, wherein the metadata includes resource scaling parameters and code packages. The resource scaling parameters are used to measure the timing of scaling up or down. The code packages include log data processing rules, which are configured by the user based on the language of the selected runtime. The runtime is deployed on the machine to provide a framework for running the log data processing program.

[0011] The updated metadata is sent to the machine.

[0012] According to another aspect of this disclosure, a log data processing apparatus is provided, configured in a machine in which multiple instances are running, the apparatus comprising:

[0013] The metadata acquisition module is used to acquire metadata, which includes resource scaling parameters and code packages. The resource scaling parameters are used to measure the timing of scaling up or down, and the code packages include log data processing rules.

[0014] A code package execution module is used to run the code package through any target instance among the plurality of instances, wherein the machine is deployed with a runtime, the runtime is used to provide a framework for running the log data processing program, and the target instance runs under the framework provided by the runtime;

[0015] The scaling module is used to monitor the runtime metrics and scale the machine up or down based on the runtime metrics and the resource scaling parameters.

[0016] According to another aspect of this disclosure, a log data processing apparatus is provided, configured in a central control service device, the central control service device being connected to multiple machines respectively, the apparatus comprising:

[0017] The monitoring module is used to monitor the update status of metadata, wherein the metadata includes resource scaling parameters and code packages. The resource scaling parameters are used to measure the timing of scaling up or down. The code packages include log data processing rules, which are configured by the user based on the language of the selected runtime. The runtime is deployed on the machine and is used to provide a framework for running the log data processing program.

[0018] The distribution module is used to distribute updated metadata to the machine.

[0019] According to another aspect of this disclosure, a log data processing system is provided, comprising:

[0020] Multiple machines, each running multiple instances, each machine having a runtime deployed thereon, the runtime providing a framework for running log data processing programs, and any target instance for processing log data running within the framework provided by the runtime;

[0021] The central control service device is connected to the multiple machines respectively, and is used to monitor the update status of metadata and send the updated metadata to the machines. The metadata includes resource scaling parameters and code packages. The resource scaling parameters are used to measure the timing of scaling up or down. The code packages include log data processing rules, which are configured by the user based on the selected runtime language.

[0022] The machine is also used to run the code package through the target instance, monitor the runtime metrics, and scale the machine up or down based on the runtime metrics and the resource scaling parameters.

[0023] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0024] At least one processor; and

[0025] A memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the log data processing method for a machine and the log data processing method for a central control service device as described in any embodiment of this disclosure.

[0027] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to execute the log data processing method for a machine and the log data processing method for a central control service device as described in any embodiment of this disclosure.

[0028] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the log data processing method for a machine and the log data processing method for a central control service device as described in any embodiment of this disclosure.

[0029] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0030] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0031] Figure 1 This is a schematic diagram of a log data processing method according to an embodiment of the present disclosure;

[0032] Figure 2 This is a schematic diagram of a log data processing method according to an embodiment of the present disclosure;

[0033] Figure 3 This is a schematic diagram of a log data processing method according to an embodiment of the present disclosure;

[0034] Figure 4 This is a schematic diagram of a log data processing method according to an embodiment of the present disclosure;

[0035] Figure 5 This is a schematic diagram of a log data processing method according to an embodiment of the present disclosure;

[0036] Figure 6 This is a schematic diagram of the structure of a log data processing system according to an embodiment of the present disclosure;

[0037] Figure 7 This is a schematic diagram of the structure of another log data processing system according to an embodiment of the present disclosure;

[0038] Figure 8 This is a schematic diagram of the structure of a log data processing apparatus according to an embodiment of the present disclosure;

[0039] Figure 9 This is a schematic diagram of the structure of another log data processing apparatus according to an embodiment of the present disclosure;

[0040] Figure 10 This is a block diagram of an electronic device used to implement the log data processing method of the embodiments of this disclosure. Detailed Implementation

[0041] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0042] Figure 1 This is a flowchart illustrating a log data processing method according to an embodiment of this disclosure. This embodiment is applicable to situations involving the collection and processing of logs generated by services, and relates to the field of computer technology, particularly cloud computing and function computing technologies. The method is applied to a machine running multiple instances. The method can be executed by a log data processing device configured on the machine, implemented in software and / or hardware. The machine can be, for example, any electronic device, including a physical machine, or an electronic device hosting virtual machines or containers. Figure 1As shown, the method specifically includes the following:

[0043] S101. Obtain metadata, which includes resource scaling parameters and code packages. Resource scaling parameters are used to measure when to scale up or down, and code packages include log data processing rules.

[0044] S102. Run the code package through any target instance among multiple instances, wherein the machine is deployed with a runtime, the runtime is used to provide a framework for running the log data processing program, and the target instance runs under the framework provided by the runtime.

[0045] S103. Monitor the operating indicators during operation, and expand or shrink the machine according to the operating indicators and resource scaling parameters.

[0046] Specifically, log data processing rules refer to how logs are collected and processed, which are configured by the user according to business needs, and this disclosure does not impose any restrictions on them.

[0047] The machine deploys a runtime environment, which provides a framework for running log data processing programs. This disclosure supports runtime environments in any language, such as Java, Python, Go, C#, and shell. Once the machine obtains metadata, it runs a code package through any target instance from among multiple instances. The target instance then collects and processes logs based on the log data processing rules within the code package, and this target instance runs within the framework provided by the runtime environment. Therefore, log filtering and computation can be performed locally on the machine in a timely manner, improving log processing efficiency.

[0048] It should be noted that since this disclosure supports runtime environments in any language, users can configure log data processing rules based on their preferred language and generate corresponding code packages, allowing the target instance to run the code packages within the runtime framework. This significantly reduces the technical barrier for users and improves work efficiency.

[0049] Furthermore, the machine can monitor runtime performance metrics and scale up or down based on these metrics and resource scaling parameters. These resource scaling parameters are used to determine the timing of scaling up or down. For example, they can be scaling up or down conditions set for runtime performance metrics. When the runtime performance metrics meet the conditions corresponding to the resource scaling parameters, scaling up or down is performed. In one implementation, the performance metric is, for example, the number of logs processed by a single process within a unit during runtime. Of course, in other implementations, the runtime performance metrics can be other parameters. This disclosure does not limit the performance metrics or the conditions configured to determine the timing of scaling up or down.

[0050] It's important to note that existing log processing methods, when service traffic is stable, allow engineers to accurately predict infrastructure needs based on traffic volume and deploy machines and components in advance. However, in scenarios with significant peak and trough traffic fluctuations, configuring the infrastructure based on peak traffic estimates leads to substantial resource waste during off-peak periods. Conversely, deploying based on trough traffic estimates results in insufficient log collection during peak periods, hindering timely monitoring of service status and detection of online service issues. In other words, existing technologies cannot flexibly scale up or down based on traffic volume.

[0051] The technical solution of this disclosure, by allowing the target instance processing log data to run within a runtime framework during machine deployment, enables flexible scaling up or down by monitoring runtime metrics. This solves the problem of dynamic log collection in systems with significant peak and trough traffic, avoiding resource waste during off-peak hours and delays caused by insufficient peak resources. Furthermore, this disclosure requires only that users configure log data processing rules and generate code packages using their preferred programming language, significantly lowering the barrier to entry and improving work efficiency. Simultaneously, by running the code package on the target instance for log data processing, this disclosure allows for timely filtering and calculation of logs locally on the machine, further enhancing log processing efficiency.

[0052] Figure 2 This is a flowchart illustrating a log data processing method according to an embodiment of the present disclosure. This embodiment is further optimized based on the above embodiments. Figure 2 As shown, the method specifically includes the following:

[0053] S201. Obtain metadata, which includes resource scaling parameters and code packages. Resource scaling parameters are used to measure when to expand or shrink the scale, and the code packages include log data processing rules.

[0054] S202. Run the code package through any target instance among multiple instances, where the machine has a runtime deployed, the runtime is used to provide a framework for running the log data processing program, and the target instance runs under the framework provided by the runtime.

[0055] The machine includes a virtual machine, a physical machine, or a container. That is, this disclosure can be deployed and run on a virtual machine, a physical machine, or a container to collect and process log data.

[0056] S203. Monitor runtime metrics, including the amount of logs processed by a single process within a preset unit of time during runtime, and resource scaling parameters including expansion ratio, shrinkage ratio, and the maximum and minimum values ​​of the total number of instances used to process log data.

[0057] S204. Calculate the ratio between the amount of logs processed by a single process within a preset unit time and the set saturation value. If the ratio is greater than the expansion ratio, the machine is expanded. If the ratio is less than the shrinkage ratio and the number of currently running processes is greater than 1, the machine is shrunk.

[0058] The machine is pre-installed with a management engine (Manager Collect Agent). This engine monitors runtime metrics and scales the machine up or down based on these metrics and resource scaling parameters. The machine also obtains metadata through the management engine. The management engine needs to be pre-installed on the machine; in this embodiment, it can be installed on a physical machine via a shell or deployed to a container using Kubernetes (k8s). The management engine operates as a single, persistent process, managing the runtime and dynamically scaling up or down based on runtime metrics.

[0059] If the ratio is greater than the expansion ratio, it indicates that the amount of logs processed by a single process within a preset unit time is close to the set saturation value, suggesting that the current traffic is high and therefore the machine needs to be expanded. If the ratio is less than the shrinking ratio, it indicates that the amount of logs processed by a single process within a preset unit time is far from the set saturation value, suggesting that the current traffic is low and resources are being wasted. In this case, if the number of currently running processes is greater than 1, the machine will be shrunk. Thus, dynamic expansion and shrinking are achieved by monitoring runtime metrics.

[0060] S205. Whether the total number of currently running instances in the monitoring machine exceeds the maximum value or is less than the minimum value.

[0061] If instances running other services are deployed on the machine, too many instances for processing log data will affect those other services, while too few will impact the efficiency of log data processing. Therefore, the resource scaling parameters in this embodiment include a maximum and a minimum total number of instances for processing log data. The minimum number of log processing processes is ensured by monitoring the total number of currently running instances on the machine and comparing it to the maximum and minimum values, thus preventing too many log processing processes from affecting the functionality of other services on the distributed instances.

[0062] The technical solution of this disclosure addresses the problem of dynamic log collection in systems with significant peak and trough traffic by allowing the target instance processing log data to run within a runtime framework during machine deployment. It achieves flexible scaling up or down by monitoring runtime metrics, thus avoiding resource waste during off-peak hours and latency issues caused by insufficient peak resources. Furthermore, this disclosure also monitors the total number of currently running instances on the machine to ensure a minimum number of log processing processes and prevent excessive log processing processes from impacting other service functions on the mixed-deployment instances.

[0063] Figure 3 This is a flowchart illustrating a log data processing method according to an embodiment of the present disclosure. This embodiment is further optimized based on the above embodiments. Figure 3 As shown, the method specifically includes the following:

[0064] S301. Obtain metadata, which includes resource scaling parameters and code packages. Resource scaling parameters are used to measure when to expand or shrink the scale, and the code packages include log data processing rules.

[0065] S302. Run the code package through any target instance among multiple instances, wherein the machine is deployed with a runtime, the runtime is used to provide a framework for running the log data processing program, and the target instance runs under the framework provided by the runtime.

[0066] S303. Monitor runtime performance metrics and adjust machine capacity based on these metrics and resource scaling parameters.

[0067] S304. Collect health data of the management engine through the health monitoring client, wherein the health data is used as the basis for controlling the management engine.

[0068] The machine comes pre-installed with a management engine and a health monitoring client. The management engine monitors runtime metrics and scales the machine up or down based on these metrics and resource scaling parameters. The health monitoring client manages the management engine. Specifically, the health monitoring client collects health data from the management engine, such as its heart rate, which can then be used as a basis for controlling the management engine, such as controlling its start, stop, and update operations.

[0069] The technical solution of this disclosure addresses the issue of dynamic log collection in traffic systems with significant peak and trough periods by monitoring runtime metrics of the management engine. This avoids resource waste during off-peak hours and latency issues caused by insufficient peak resources. Furthermore, to resolve the issue of abnormal exits of the management engine on a single machine, this disclosure uses a health monitoring client to collect the management engine's health data. This health data is then used as the basis for controlling the management engine, eliminating the need for manual online intervention, automating health data collection, ensuring the stability of the management engine, and guaranteeing high availability across the entire network.

[0070] Figure 4 This is a flowchart illustrating a log data processing method according to an embodiment of the present disclosure. This embodiment is applicable to situations involving the collection and processing of logs generated by services, and relates to the field of computer technology, particularly cloud computing and function computing technologies. The method is applied to a central control service device connected to multiple machines. The method can be executed by a log data processing device configured on the central control service device, which is implemented in software and / or hardware. The central control service device can be, for example, any type of electronic device.

[0071] S401. Monitor the update status of metadata, which includes resource scaling parameters and code packages. Resource scaling parameters are used to measure when to expand or shrink the scale. The code packages include log data processing rules. The log data processing rules are configured by the user based on the language of the selected runtime. The runtime is deployed on the machine to provide a framework for running the log data processing program.

[0072] S402. Send the updated metadata to the machine.

[0073] Specifically, in this embodiment, a console can be provided to the user, where the user can select the runtime language and configure log data processing rules based on that language. The console can also generate and store code packages based on the user's configuration information.

[0074] The central control service device can monitor the update status of metadata via the console. If updated metadata is detected, it will be distributed to the machines. A runtime is pre-deployed on the machines, providing a framework for running log data processing programs. Thus, once the machine obtains the code package, it can run the code package through any target instance to collect and process log data. Furthermore, since the target instance runs within the framework provided by the runtime, the machine can also monitor runtime metrics and adjust its size based on these metrics and resource scaling parameters.

[0075] The technical solution of this disclosure involves a central control service device monitoring and distributing updated metadata to the machine. Then, by deploying and running the system on the machine, the target instance processing the log data runs the code package from the metadata within the runtime framework. This not only enables log filtering and calculation operations locally on the machine, improving log processing efficiency, but also allows for flexible scaling up or down by monitoring runtime metrics. This solves the problem of dynamic log collection under systems with significant peak and trough traffic, avoiding resource waste during off-peak hours and delays caused by insufficient peak resources. Furthermore, this disclosure requires only that the user configure log data processing rules and generate the code package using their preferred programming language on the monitoring console, significantly lowering the user's barrier to entry and improving work efficiency.

[0076] Figure 5 This is a flowchart illustrating a log data processing method according to an embodiment of the present disclosure. This embodiment is further optimized based on the above embodiments. Figure 5 As shown, the method specifically includes the following:

[0077] S501, monitor the update status of metadata, which includes resource scaling parameters and code packages. Resource scaling parameters are used to measure when to expand or shrink the scale. The code packages include log data processing rules, which are configured by the user based on the language of the selected runtime. The runtime is deployed on the machine to provide a framework for running log data processing programs.

[0078] S502: Send the updated metadata to the machine.

[0079] S503. Receive the monitoring results from the receiving machine regarding whether the total number of currently running instances exceeds the maximum value or is less than the minimum value, and issue an alarm based on the monitoring results.

[0080] S504 receives and displays runtime metrics monitored by the machine through the management engine, as well as health data collected by the machine through the health monitoring client from the management engine.

[0081] The machine comes pre-installed with a management engine and a health monitoring client. The management engine monitors runtime metrics and scales the machine up or down based on these metrics and resource scaling parameters. The health monitoring client collects health data from the management engine. Specifically, runtime metrics include the amount of logs processed by a single process within a preset unit of time, and resource scaling parameters include the scaling-up ratio, scaling-down ratio, and the maximum and minimum total number of instances used to process log data.

[0082] The management engine within the machine monitors and calculates the ratio between the amount of logs processed by a single process within a preset unit of time and a set saturation value. If the ratio is greater than the expansion ratio, the machine is expanded; if the ratio is less than the shrinkage ratio, and the number of currently running processes is greater than one, the machine is shrunk. The management engine also monitors whether the total number of currently running instances on the machine exceeds the maximum value or is less than the minimum value, and reports the monitoring results to the central control service device, which then issues an alarm based on the monitoring results.

[0083] In addition, the central control service equipment can also receive and display the machine's runtime metrics monitored by the management engine, as well as the management engine's health data collected by the machine through the health monitoring client.

[0084] The technical solution of this disclosure involves a central control service device monitoring and distributing updated metadata to the machine. Then, by deploying and running the system on the machine, the target instance processing log data runs the code package from the metadata within the runtime framework. This not only enables log filtering and calculation operations locally on the machine, improving log processing efficiency, but also allows for flexible scaling up or down by monitoring runtime metrics. This solves the problem of dynamic log collection under systems with significant peak and trough traffic, avoiding resource waste during off-peak hours and delays caused by insufficient peak resources. Furthermore, this disclosure can monitor the total number of currently running instances and issue alarms based on the monitoring results to ensure a minimum number of log processing processes and prevent excessive log processing processes from affecting other service functions on the mixed-deployment instances.

[0085] Figure 6 This is a schematic diagram of a log data processing system according to an embodiment of the present disclosure. This embodiment is applicable to the collection and processing of logs generated by services, and relates to the field of computer technology, particularly cloud computing and function computing technologies. The system includes:

[0086] Multiple machines (only four machines are shown in the figure), multiple instances (not shown in the figure) running on the machines, and the machines are deployed with runtimes, which provide a framework for running log data processing programs. Any target instance used to process log data runs within the framework provided by the runtime.

[0087] The central control service device 601 is connected to multiple machines (machines 1-4) to monitor the update status of metadata and send the updated metadata to the machines. The metadata includes resource scaling parameters and code packages. The code packages include log data processing rules, which are configured by the user based on the runtime language.

[0088] The machine is pre-installed with a management engine (not shown in the figure). The machine is also used to run the code package through the target instance and monitor the runtime metrics through the management engine. The machine is then scaled up or down based on the runtime metrics and resource scaling parameters.

[0089] The machine also comes pre-installed with a health monitoring client (not shown in the figure), which is used to collect health data from the management engine. Correspondingly, the system also includes a health monitoring module 602, which controls the start, stop, and update operations of the management engine based on the health data collected by the health monitoring client. This health data includes heart rate.

[0090] In addition, the system includes a console 603 and a log storage module 604. The console 603 provides users with the option to select the runtime language and generates code packages based on the log data processing rules configured by the user according to the selected runtime language. The log storage module 604 stores the log data collected and processed by the machine while running the target instance.

[0091] Figure 7 This is a schematic diagram of another log data processing system according to an embodiment of this disclosure. As shown in the figure, the user selects a familiar language to run the system via the console, configures log data processing rules based on that language, and generates a code package. Regarding the code package, the user can edit it online, then compile, package, and store it on the console, but is not limited to developing it locally in an IDE (Integrated Development Environment) and then uploading it. In addition, resource scaling parameters need to be configured, including the expansion ratio, shrinkage ratio, and the maximum and minimum values ​​of the total number of instances used to process log data. The resource scaling parameters and the code package together constitute metadata and are stored. The central control service device monitors whether there are updated metadata in the data storage and distributes the updated metadata to each machine.

[0092] The machine comes pre-installed with a management engine and a health monitoring client. The management engine monitors runtime metrics and scales the machine up or down based on these metrics and resource scaling parameters. The health monitoring client collects health data from the management engine to manage it.

[0093] The machine receives metadata through the management engine and runs code packages on any target instance from multiple instances. Based on user-configured log data processing rules, logs can be collected and processed locally on the machine. The processing results are then stored in storage, message queues, or file systems as needed.

[0094] The machine is deployed with a runtime environment, which provides a framework for running log data processing programs. The target instance runs within this framework. Therefore, the machine can also monitor runtime metrics through the management engine, and scale the machine up or down based on these metrics and resource scaling parameters.

[0095] To address the issue of abnormal shutdowns of the standalone management engine, the machine collects health data from the management engine via a health monitoring client and sends this data to a health monitoring device. This health monitoring device operates in bypass mode, using the health data as the basis for controlling the management engine. This eliminates the need for manual online intervention, automates health data collection, ensures the stability of the management engine, and guarantees high availability throughout the entire process. It should also be noted that if the health monitoring client is installed first, the management engine can be downloaded through the client, eliminating the need for a separate installation.

[0096] The central control service device in the system can receive monitoring results from the machine to determine whether the total number of currently running instances exceeds the maximum value or is less than the minimum value, make judgments based on the monitoring results, and issue alarms in the auxiliary module.

[0097] In addition, the central control service equipment can also receive and display in the auxiliary module the operating indicators monitored by the machine through the management engine, as well as the health data collected by the machine through the health detection client from the management engine.

[0098] As can be seen, the technical solution disclosed herein solves the problem of dynamic log collection under traffic systems with significant peaks and troughs, avoiding resource waste during off-peak hours and delays caused by insufficient resources during peak hours. Furthermore, this disclosure introduces a function computing approach, providing a function framework and runtime. Users only need to write business-related logic using their preferred programming language, significantly lowering the barrier to entry and improving work efficiency. Moreover, this disclosure, through single-machine resident process management, allows for flexible scaling of the runtime instance collector according to the scaling ratio. In addition, health management services ensure the stability of the management engine itself, guaranteeing high availability throughout the entire chain. Finally, the system implemented based on this disclosure can be deployed and used on physical machines, virtual machines, and containers, making it widely applicable.

[0099] Figure 8 This is a schematic diagram of a log data processing apparatus according to an embodiment of the present disclosure. This embodiment is applicable to the collection and processing of logs generated by services, and relates to the field of computer technology, particularly cloud computing and function computing technologies. The apparatus is configured on a machine, and multiple instances run on the machine. This apparatus can implement the log data processing method applied to a machine as described in any embodiment of the present disclosure. Figure 8 As shown, the device 800 specifically includes:

[0100] Metadata acquisition module 801 is used to acquire metadata, wherein the metadata includes resource scaling parameters and code packages, the resource scaling parameters are used to measure the timing of scaling up or down, and the code packages include log data processing rules;

[0101] The code package execution module 802 is used to run the code package through any target instance among the plurality of instances, wherein the machine is deployed with a runtime, the runtime is used to provide a framework for running the log data processing program, and the target instance runs under the framework provided by the runtime;

[0102] The scaling module 803 is used to monitor the running indicators of the machine and to scale the machine up or down according to the running indicators and the resource scaling parameters.

[0103] Optionally, the operational metrics include the amount of logs processed by a single process within a preset unit of time during runtime; the resource scaling parameters include expansion ratio and shrinkage ratio.

[0104] Optionally, the expansion / shrinkage module 803 includes:

[0105] The calculation unit is used to calculate the ratio between the amount of logs processed by a single process during the runtime and a set saturation value.

[0106] An expansion unit, if the ratio is greater than the expansion ratio, then expands the capacity of the machine;

[0107] A shrinking unit is used to shrink the machine if the ratio is less than the shrinking ratio and the number of processes currently running at runtime is greater than 1.

[0108] Optionally, the resource scaling parameters further include a maximum and a minimum value for the total number of instances used to process log data; correspondingly, the device further includes:

[0109] The total number of instances monitoring module is used to monitor whether the total number of currently running instances in the machine exceeds the maximum value or is less than the minimum value.

[0110] Optionally, a management engine is pre-installed in the machine; correspondingly, the scaling module 803 is specifically used for:

[0111] The management engine monitors the runtime metrics and scales up or down the machine based on the runtime metrics and resource scaling parameters.

[0112] Optionally, the machine also has a health monitoring client pre-installed; correspondingly, the device further includes:

[0113] A health monitoring module is used to collect health data of the management engine through the health monitoring client, wherein the health data is used as the basis for controlling the management engine.

[0114] Optionally, the health data includes heart rate; controlling the management engine refers to controlling the start, stop, and update operations of the management engine.

[0115] Optionally, the machine may include a virtual machine, a physical machine, or a container.

[0116] The above-described product can execute the log data processing method for machines provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0117] Figure 9 This is a schematic diagram of a log data processing apparatus according to an embodiment of the present disclosure. This embodiment is applicable to the collection and processing of logs generated by services, and relates to the field of computer technology, particularly cloud computing and function computing technologies. The apparatus is configured on a central control service device, which is connected to multiple machines respectively. This apparatus can implement the log data processing method applied to the central control service module as described in any embodiment of the present disclosure. Figure 9 As shown, the device 900 specifically includes:

[0118] The monitoring module 901 is used to monitor the update status of metadata, wherein the metadata includes resource scaling parameters and code packages. The resource scaling parameters are used to measure the timing of scaling up or down. The code packages include log data processing rules, which are configured by the user based on the language of the selected runtime. The runtime is deployed on the machine and is used to provide a framework for running the log data processing program.

[0119] The distribution module 902 is used to distribute updated metadata to the machine.

[0120] Optionally, the resource scaling parameters include a maximum and a minimum value for the total number of instances used to process log data; correspondingly, the apparatus further includes:

[0121] The alarm module is used to receive the monitoring results from the machine regarding whether the total number of currently running instances exceeds the maximum value or is less than the minimum value; and to issue an alarm based on the monitoring results.

[0122] Optionally, the machine is pre-installed with a management engine and a health monitoring client. The management engine is used to monitor the runtime metrics and scale the machine up or down based on the runtime metrics and the resource scaling parameters. The health monitoring client is used to collect health data from the management engine.

[0123] Optionally, the device further includes:

[0124] The receiving module is used to receive and display the runtime metrics monitored by the machine through the management engine, as well as the health data of the management engine collected by the machine through the health detection client.

[0125] The above-described product can execute the log data processing method for the central control service module provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0126] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this technical solution comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0127] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0128] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0129] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0130] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0131] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as a log data processing method applied to a machine. For example, in some embodiments, the log data processing method applied to a machine may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the log data processing method applied to a machine described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured by any other suitable means (e.g., by means of firmware) to perform log data processing methods applied to the machine.

[0132] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0137] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem that addresses the management difficulties and weak business scalability inherent in traditional physical hosting and VPS services. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0138] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0139] Cloud computing refers to a technology system that enables access to a shared pool of physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.

[0140] Furthermore, according to embodiments of this disclosure, this disclosure also provides another electronic device, another readable storage medium, and another computer program product for performing one or more steps of the log data processing method applied to a central control service device as described in any embodiment of this disclosure. Its specific structure and program code can be found in... Figure 10 The content of the illustrated embodiments will not be repeated here.

[0141] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0142] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A log data processing method applied to a machine, wherein a plurality of instances are running in the machine, and the method comprises: obtaining metadata, wherein the metadata comprises resource scaling parameters and a code package, the resource scaling parameters are used to measure the timing of scaling up or down, and the code package comprises log data processing rules; running the code package through any target instance in the plurality of instances, wherein the machine is deployed with a runtime, the runtime is used to provide a framework for running log data processing programs, and the target instance is running under the framework provided by the runtime; monitoring a running index of the runtime, and scaling up or down the machine according to the running index and the resource scaling parameters; wherein the running index comprises the amount of logs processed by a single process of the runtime in a preset unit of time, and the resource scaling parameters comprise a scaling up ratio and a scaling down ratio; wherein the scaling up or down of the machine according to the running index and the resource scaling parameters comprises: calculating the ratio between the amount of logs processed by a single process of the runtime in a preset unit of time and a set saturation value; scaling up the machine if the ratio is greater than the scaling up ratio; scaling down the machine if the ratio is less than the scaling down ratio and the number of currently running processes of the runtime is greater than 1.

2. The method of claim 1, wherein, The resource scaling parameters further comprise a maximum value and a minimum value of the total number of instances used to process log data, and the method further comprises: monitoring whether the total number of currently running instances in the machine exceeds the maximum value or is less than the minimum value.

3. The method of claim 1, wherein, The machine is pre-installed with a management engine, and the monitoring of the running index of the runtime and the scaling up or down of the machine according to the running index and the resource scaling parameters comprises: monitoring the running index of the runtime through the management engine, and scaling up or down the machine according to the running index and the resource scaling parameters.

4. The method of claim 3, wherein, The machine is further pre-installed with a health detection client, and the method further comprises: collecting health data of the management engine through the health detection client, wherein the health data is used as a basis for controlling the management engine.

5. The method of claim 4, wherein, The health data comprises a heartbeat, and the control of the management engine refers to the control of start-stop and update operations of the management engine.

6. The method of claim 1, wherein, The machine comprises a virtual machine, a physical machine or a container. 7.A log data processing method applied to a central control service device, wherein the central control service device is connected with a plurality of machines respectively, and the method comprises: listening to the update state of metadata, wherein the metadata comprises resource scaling parameters and a code package, the resource scaling parameters are used to measure the timing of scaling up or down, the code package comprises log data processing rules, and the log data processing rules are configured by a user based on the language of a selected runtime, wherein the runtime is deployed in the machine and is used to provide a framework for running log data processing programs; issuing the updated metadata to the machine. The machine is used for monitoring an operation index of the runtime, and performing capacity expansion or capacity reduction on the machine according to the operation index and the resource scaling parameter; the operation index includes a log quantity processed by a single process of the runtime within a preset unit time; and the resource scaling parameter includes a capacity expansion ratio and a capacity reduction ratio. The capacity expansion or capacity reduction on the machine according to the operation index and the resource scaling parameter includes: calculating a ratio between the log quantity processed by the single process of the runtime within the preset unit time and a set saturation value; if the ratio is greater than the capacity expansion ratio, performing capacity expansion on the machine; and if the ratio is less than the capacity reduction ratio and a current process number of the runtime is greater than 1, performing capacity reduction on the machine.

8. The method of claim 7, wherein, The resource scaling parameter includes a maximum value and a minimum value of a total number of instances for processing log data; and the method further includes: receiving a monitoring result of whether a total number of currently running instances of the machine exceeds the maximum value or is less than the minimum value; performing alarm according to the monitoring result.

9. The method of claim 7, wherein, The machine is pre-installed with a management engine and a health detection client; the management engine is used for monitoring an operation index of the runtime, and performing capacity expansion or capacity reduction on the machine according to the operation index and the resource scaling parameter; and the health detection client is used for collecting health data of the management engine.

10. The method according to claim 9, further comprising: receiving and displaying the operation index of the runtime monitored by the management engine of the machine and the health data of the management engine collected by the health detection client of the machine.

11. A log data processing apparatus configured in a machine, wherein a plurality of instances are running in the machine, and the apparatus comprises: a metadata acquisition module configured to acquire metadata, wherein the metadata includes a resource scaling parameter and a code package; the resource scaling parameter is used for measuring a capacity expansion or capacity reduction opportunity; and the code package includes a log data processing rule; a code package running module configured to run the code package through any target instance in the plurality of instances, wherein the machine is deployed with a runtime, the runtime is used for providing a framework for running a log data processing program, and the target instance is running under the framework provided by the runtime; a capacity expansion and reduction module configured to monitor an operation index of the runtime, and perform capacity expansion or capacity reduction on the machine according to the operation index and the resource scaling parameter; the operation index includes a log quantity processed by a single process of the runtime within a preset unit time; and the resource scaling parameter includes a capacity expansion ratio and a capacity reduction ratio; the capacity expansion and reduction module includes: a calculation unit configured to calculate a ratio between the log quantity processed by the single process of the runtime within the preset unit time and a set saturation value; a capacity expansion unit configured to perform capacity expansion on the machine if the ratio is greater than the capacity expansion ratio; a capacity reduction unit configured to perform capacity reduction on the machine if the ratio is less than the capacity reduction ratio and a current process number of the runtime is greater than 1. 12.A log data processing apparatus configured in a central control service device, the central control service device being connected with a plurality of machines respectively, the apparatus comprising: a listening module configured to listen to an update state of metadata, wherein the metadata comprises resource scaling parameters and code packages, the resource scaling parameters are used to measure a timing of scaling up or down, the code packages comprise log data processing rules, the log data processing rules are configured by a user based on a language of a selected runtime, and the runtime is deployed in the machine to provide a framework for running a log data processing program; a delivery module configured to deliver the updated metadata to the machine; wherein the machine is configured to monitor a running index of the runtime, and to scale up or down the machine according to the running index and the resource scaling parameters, wherein the running index comprises a log amount processed by a single process of the runtime in a preset unit of time, and the resource scaling parameters comprise a scaling up ratio and a scaling down ratio; wherein the scaling up or down of the machine according to the running index and the resource scaling parameters comprises: calculating a ratio between the log amount processed by the single process of the runtime in the preset unit of time and a set saturation value, scaling up the machine if the ratio is greater than the scaling up ratio, and scaling down the machine if the ratio is less than the scaling down ratio and a number of currently running processes of the runtime is greater than 1. 13.A log data processing system comprising: a plurality of machines, a plurality of instances running in the machines, the machines being deployed with a runtime, the runtime being used to provide a framework for running a log data processing program, and any target instance for processing log data running under the framework provided by the runtime; a central control service device connected with the plurality of machines respectively, configured to listen to an update state of metadata, and to deliver the updated metadata to the machine, wherein the metadata comprises resource scaling parameters and code packages, the resource scaling parameters are used to measure a timing of scaling up or down, the code packages comprise log data processing rules, and the log data processing rules are configured by a user based on a language of a selected runtime; the machine is further configured to run the code packages by the target instance, and to monitor a running index of the runtime, and to scale up or down the machine according to the running index and the resource scaling parameters; wherein the running index comprises a log amount processed by a single process of the runtime in a preset unit of time, and the resource scaling parameters comprise a scaling up ratio and a scaling down ratio; wherein the scaling up or down of the machine according to the running index and the resource scaling parameters comprises: calculating a ratio between the log amount processed by the single process of the runtime in the preset unit of time and a set saturation value; scaling up the machine if the ratio is greater than the scaling up ratio; scaling down the machine if the ratio is less than the scaling down ratio and a number of currently running processes of the runtime is greater than 1.

14. The system of claim 13, wherein, The machine is pre-installed with a management engine for monitoring running indexes of the runtime and performing expansion or contraction of the machine according to the running indexes and the resource scaling parameters.

15. The system of claim 14, wherein the machine is further pre-installed with a health detection client for collecting health data of the management engine; accordingly, the system further comprises: a health detection module for controlling start-stop and update operations of the management engine based on the health data of the management engine collected by the health detection client, wherein the health data comprises heartbeat.

16. The system of claim 13, further comprising: a console for providing a user with a selection of languages of the runtime and generating the code package according to log data processing rules configured by the user based on the selected languages of the runtime; a log storage module for storing log data collected and processed by the machine by running the target instance.

17. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the log data processing method applied to a machine according to any one of claims 1-6, and the log data processing method applied to a central control service device according to any one of claims 7-10. The computer instructions are used to enable the computer to perform the log data processing method applied to a machine according to any one of claims 1-6, and the log data processing method applied to a central control service device according to any one of claims 7-10.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, 19. A computer program product comprising computer programs / instructions, which, when executed by a processor, implement the log data processing method applied to a machine according to any one of claims 1-6, and the log data processing method applied to a central control service device according to any one of claims 7-10. ​

Images