A selector determination method, device, apparatus and computer readable storage medium

Abstract

The application discloses a selector determination method and device, equipment and computer readable storage medium, apply to log collection technical field, including: obtaining the resource information corresponding to the current log distribution task and corresponding data type;Wherein, the resource information includes the resource information of the event temporary buffer area and the host resource information;According to the resource information and the data type, the selector information is predicted;Wherein, the selector information includes the selector type and the selector configuration information;According to the selector type and the selector configuration information, the selector corresponding to the next log distribution task is determined.Compared with the current manual configuration selector, the application predicts the selector type and the corresponding selector parameters required by the next log distribution task according to the resource information corresponding to the current log distribution task and the data type, since each selector type can correspond to multiple selector parameters, which is equivalent to the existence of many parameter different selectors, and the accuracy of the selector determination is improved.

Description

Technical Field

[0001] This invention relates to the field of log acquisition technology, and in particular to a selector determination method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] In the era of big data, Flume, as an excellent data collection tool, is applied in multiple scenarios, and collecting and distributing logs is a crucial function. In the traditional Flume data collection process, the type of Channel selector is determined manually. Two main types of Channel selectors are available, but this manual approach leads to inaccurate selection, thus affecting data processing efficiency.

[0003] It is evident that improving the accuracy of channel selector selection is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a selector determination method, apparatus, device and computer-readable storage medium, which solves the technical problem of low accuracy of channel selector selection in the prior art.

[0005] To solve the above-mentioned technical problems, the present invention provides a selector determination method, comprising:

[0006] Obtain the resource information corresponding to the current log distribution task, as well as the corresponding data type; wherein, the resource information includes the resource information of the event temporary buffer and the host resource information;

[0007] Based on the resource information and the data type, predict selector information; wherein, the selector information includes selector type and selector configuration information;

[0008] The selector for the next log distribution task is determined based on the selector type and the selector configuration information.

[0009] In some embodiments, predicting selector information based on the resource information and the data type includes:

[0010] Based on the resource information and the data type, a machine learning model is used to predict the selector type; wherein, the machine learning model is a model in which the prediction accuracy increases with the number of predictions; the selector type includes a specified channel selector and a multiplexed channel selector.

[0011] The machine learning model is used to determine the selector configuration information based on the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task; wherein, the selector configuration information includes at least one of the following: number of nodes, number of temporary caches, number of hosts, memory usage configuration of temporary caches, storage configuration of temporary caches, and caching configuration of temporary caches.

[0012] In some embodiments, after determining the selector corresponding to the next log distribution task based on the selector type and the selector configuration information, the method further includes:

[0013] Determine the task execution time for the next log distribution task based on the selector corresponding to the next log distribution task;

[0014] The task execution time of the current log distribution task is compared with the task execution time of the next log distribution task, and the machine learning model parameters of the current log distribution task are compared with the machine learning model parameters of the next log distribution task to determine the comparison results.

[0015] Adjust the parameters of the machine learning model based on the comparison results;

[0016] The adjusted machine learning model is used for prediction, and the parameters of the machine learning model are continuously adjusted until the execution time of each log distribution task remains within the set range.

[0017] In some embodiments, obtaining the resource information corresponding to the current log distribution task, and the corresponding data type, includes:

[0018] The monitoring module is used to obtain resource information written to the temporary buffer of each event;

[0019] The monitoring module is used to obtain the host resource information and the data type.

[0020] In some embodiments, predicting selector information based on the resource information and the data type includes:

[0021] Determine the host resources consumed by the current log distribution task, as well as the resources consumed by the event temporary buffer.

[0022] Based on the resource information of the event temporary buffer and the consumed resources of the event temporary buffer, as well as the host resource information and the consumed host resources, determine the remaining host resources and the remaining event temporary buffer resources;

[0023] Based on the remaining host resources, the remaining event temporary buffer resources, and the task execution type, predict the selector type;

[0024] Determine whether the execution time of the current log distribution task is greater than the execution time of the previous log distribution task;

[0025] If the execution time of the current log distribution task is longer than that of the previous log distribution task, the selector configuration information corresponding to the previous log distribution task is compared with the selector configuration information corresponding to the current log distribution task to determine the selectedor configuration information that has changed.

[0026] Configure the configuration information of the selector corresponding to the selectedor type according to the changed selectedor configuration information.

[0027] In some embodiments, the host resource information includes at least one of hard disk resource information, memory resource information, central processing unit resource information, network interface card resource information, and bandwidth resource information, and the resource information of the event temporary buffer includes at least one of task execution time, host resources occupied by the task, event temporary buffer, and inter-host communication rate.

[0028] In some embodiments, predicting selector information based on the resource information and the data type includes:

[0029] Based on the resource information and the data type, a linear regression model is used to predict the task volume and task type of the next log distribution task.

[0030] The selector information is determined based on the predicted task volume and task type.

[0031] This invention also provides a selector determining device, comprising:

[0032] The input data acquisition module is used to acquire the resource information corresponding to the current log distribution task, as well as the corresponding data type; wherein, the resource information includes the resource information of the event temporary buffer and the host resource information;

[0033] The prediction module is used to predict selector information based on the resource information and the data type; wherein the selector information includes selector type and selector configuration information.

[0034] The selector determination module is used to determine the selector corresponding to the next log distribution task based on the selector type and the selector configuration information.

[0035] This invention also provides a selector determining device, comprising:

[0036] Memory, used to store computer programs;

[0037] A processor for executing the computer program to implement the steps of the selector determination method described above.

[0038] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the selector determination method described above.

[0039] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0040] To address the aforementioned technical problems, embodiments of the present invention provide a selector determination method, which may include: obtaining resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer and host resource information; predicting selector information based on the resource information and the data type; wherein the selector information includes selector type and selector configuration information; and determining the selector corresponding to the next log distribution task based on the selector type and selector configuration information.

[0041] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: Compared with the current manual configuration of selectors, the present invention predicts selector information based on the resource information and data type corresponding to the current log distribution task. The selector information includes selector type and corresponding selector parameters. Since each selector type can correspond to multiple selector parameters, it is equivalent to having many selectors, which improves the accuracy of selector determination. Attached Figure Description

[0042] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart of a selector determination method provided in an embodiment of the present invention;

[0044] Figure 2 A flowchart illustrating a selector determination method provided in an embodiment of the present invention;

[0045] Figure 3 A framework diagram of a selector determination method provided in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of a selector determining device provided in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of a selector determining device provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0049] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0050] In the era of big data, Flume, as an excellent data collection tool, is applied in multiple scenarios, and collecting and distributing logs is a very important function. In Flume's log distribution architecture, the Source (data acquisition component) encapsulates the collected data into Events, which are then copied and distributed to various Channels (temporary event buffers) through the Channel Selector. The Sink behind the Channel (responsible for consuming data from the Channel and sending it to the corresponding destination) retrieves data from the Channel and writes the log information to the destination.

[0051] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Next, a selector determination method provided by an embodiment of the present invention will be described in detail. Figure 1 A flowchart of a selector determination method provided in an embodiment of the present invention, the method may include:

[0053] S101, obtain the resource information corresponding to the current log distribution task, as well as the corresponding data type; the resource information includes the resource information of the event temporary buffer and the host resource information.

[0054] The execution subject of this embodiment is an electronic device. This embodiment is not limited to a specific electronic device. For example, the electronic device in this embodiment can be a computer; or the electronic device in this embodiment can be a server. The log distribution task in this embodiment refers to the task of distributing logs using a selector. This embodiment does not limit specific resource information. The resource information in this embodiment refers to the resource information of the current host and the event temporary buffer (Channel), where the relationship between the event temporary buffer and the host is one-to-one. The resource information in this embodiment refers to various configuration, usage, and status data related to the computer host hardware and software; the event temporary buffer resource information in this embodiment refers to Channel resource information, which typically refers to hardware and software configuration, status, and usage data related to communication or data transmission. The event temporary buffer Channel in this embodiment is a passive storage device responsible for temporarily storing data sent by the Source. The data type in this embodiment is determined by the data characteristics of the data source itself and the data structure of the data source. This is fixed and will only change in repetitive tasks unless the business party changes the data source to be collected. This is automatically detected by the Channel selector itself.

[0055] It should be further noted that the host resource information includes at least one of the following: hard disk resource information, memory resource information, central processing unit resource information, network card resource information, and bandwidth resource information. The event temporary buffer resource information includes at least one of the following: task execution time, host resources occupied by the task, event temporary buffer, and inter-host communication rate.

[0056] It should be further explained that, based on the above embodiments, obtaining the resource information corresponding to the current log distribution task, as well as the corresponding data type, may include: using the monitoring module to obtain the resource information written to each event temporary buffer; and using the monitoring module to obtain host resource information and data type. This embodiment will monitor resource information in real time to obtain it promptly.

[0057] S102, based on resource information and data type, predict selector information; wherein, selector information includes selector type and selector configuration information.

[0058] This embodiment is not limited to predicting selector information based on resource information and data type, as long as the model can obtain the output (selector information) based on the input (resource type and data type). For example, the model in this embodiment can be GBDT (Gradient Boosting Decision Tree), which uses a decision tree as the base learner and is suitable for regression and classification tasks; or it can be XGBoost (eXtreme Gradient Boosting): an efficient implementation of GBDT that significantly improves computational speed and performance; or it can be LightGBM (Light Gradient Boosting Machine): based on leaf growth strategy and histogram technology, suitable for large datasets, fast, and with low memory usage; or it can be CatBoost (Categorical Boosting), suitable for datasets containing categorical features, which does not require additional hot encoding and reduces time costs; or the model performing prediction in this embodiment can also be a linear regression algorithm model. The selector type in this embodiment is an existing selector type. For example, the selector in this embodiment can be a Replicating Channel Selector. This selector replicates each event to all Channels storage. This selector is suitable for high availability and fault tolerance scenarios, such as redundantly storing data in multiple Channels. The reliability issue is that sufficient storage space and performance are required to handle duplicate data; otherwise, a large amount of duplicate data in a production environment may cause system overload. A Multiplexing Channel Selector routes events to different Channels based on the time header attribute (Header attributes refer to the header information in an HTTP request or response, used to transmit additional data and control messages). This is suitable for scenarios where data needs to be distributed based on conditions such as event type and source. In this scenario, database reliability cannot be guaranteed because there is no redundancy, which may lead to data loss. This embodiment does not limit the specific selector configuration information, as long as the method of replicating and distributing the selector to each Channel can be determined based on the selector configuration information. For example, the selector configuration information in this embodiment can be the number of channels and the number of hosts; or the selector configuration information in this embodiment can be the resource configuration of the channels.

[0059] It should be further noted that the above-mentioned prediction of selector information based on resource information and data type may include:

[0060] S1021, Based on resource information and data type, use a machine learning model to predict the selector type; wherein, the machine learning model is a model whose prediction accuracy increases with the number of predictions; the selector type includes a specified channel selector and a multiplexed channel selector;

[0061] S1022, using a machine learning model to determine selector configuration information based on the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task; wherein, the selector configuration information includes at least one of the following: number of nodes, number of temporary caches, number of hosts, memory usage configuration of temporary caches, storage configuration of temporary caches, and cache configuration of temporary caches.

[0062] The model in this embodiment is a machine learning model. The accuracy of this model increases with the number of predictions, until the task execution time remains within a certain range when using the predicted selector for log distribution. The number of nodes in this embodiment refers to the number of host nodes corresponding to the temporary cache area; the memory usage configuration of the temporary cache area allows for increasing the memory usage configuration of each channel; and the cache configuration refers to whether caching is enabled. This embodiment can determine if a task execution time has increased, indicating a slowdown. It then compares the input parameters with the previous one to check if there are fewer host nodes, less CPU (Central Processing Unit) resources, or excessive memory usage. This allows for adjustments such as reducing the number of channels, increasing the memory usage and storage configuration of each channel, and enabling caching, thereby modifying the channel selector configuration information. Each time a task is executed, the monitoring model output is compared with the previous execution result, and the channel selector resource configuration (number of nodes, weight allocation, resource ratio) is adjusted accordingly. This embodiment improves the accuracy of the selector configuration by providing specific selector types and configuration information.

[0063] It should be further explained that, based on any of the above embodiments, predicting selector information according to resource information and data type may include: predicting the task volume and task type of the next log distribution task using a linear regression model based on resource information and data type; and determining selector information based on the predicted task volume and task type. Before establishing the linear regression model, this embodiment needs to determine the independent variables (explanatory variables) and dependent variables (explained variables). In this scenario, the independent variables may include the task volume and task type, while the dependent variables are the task volume and task type of the next log distribution task. Determining selector information based on the predicted task volume and task type in this embodiment refers to determining the selector type and its corresponding configuration information. This embodiment considers both task volume and task type simultaneously during prediction. For example, when the task volume is large and current resource information is insufficient, a specified channel selector can be selected without caching; when the task is important in this embodiment, caching can be performed.

[0064] It should be further explained that after determining the selector corresponding to the next log distribution task based on the selector type and selector configuration information, the process may further include: determining the task execution time of the next log distribution task based on the selector corresponding to the next log distribution task; comparing the task execution time corresponding to the current log distribution task with the task execution time corresponding to the next log distribution task, and comparing the machine learning model parameters corresponding to the current log distribution task with the machine learning model parameters corresponding to the next log distribution task, and determining the comparison result; adjusting the parameters of the machine learning model according to the comparison result; determining to use the adjusted machine learning model for prediction, and continuously adjusting the parameters of the machine learning model until the task execution time corresponding to each log distribution task is maintained within a set range. This embodiment does not limit the specific set range. For example, the set range can be determined based on the shortest task execution time; or the set range in this embodiment can also be determined based on the average task execution time. This embodiment can adjust the parameters of the machine learning model while making predictions, thereby improving the accuracy of the machine learning model's predictions.

[0065] It should be further noted that, based on any of the above embodiments, the prediction of selector information based on resource information and data type may include:

[0066] S1: Determine the host resources consumed by the current log distribution task, as well as the resources consumed by the event temporary buffer.

[0067] The resource information in this embodiment refers to the resource information before the task is executed. When the resource information in this embodiment is the resource information after the log distribution task is executed, the resource information is the remaining resource information and no calculation is required. This embodiment can collect the host resources (memory, hard disk, CPU, bandwidth) consumed by the task. The execution consumption of each node is different, and the average resource consumption is calculated through variance.

[0068] S2: Based on the resource information of the event temporary buffer and the resources consumed in the event temporary buffer, as well as the host resource information and the resources consumed in the host, determine the remaining host resources and the remaining event temporary buffer resources.

[0069] S3: Predict the selector type based on the remaining host resources, the remaining event temporary buffer resources, and the type of task execution.

[0070] S4: Determine whether the execution time of the current log distribution task is greater than the execution time of the previous log distribution task.

[0071] In this embodiment, the execution time of this round of tasks (each Channel will return its own individual execution time on the host) is used to calculate the average regression time based on the variance, which is the execution time of the log distribution task.

[0072] S5: If the execution time of the current log distribution task is longer than the execution time of the previous log distribution task, compare the selector configuration information corresponding to the previous log distribution task with the selector configuration information corresponding to the current log distribution task to determine the selectedor configuration information that has changed.

[0073] S6: Configure the configuration information of the selector corresponding to the selectedor type based on the changed selector configuration information.

[0074] When determining the selector's configuration information, this embodiment considers the changed selector configuration information to determine what configuration change caused the change in task execution time, thereby accurately adjusting the selector's configuration information.

[0075] S103, determine the selector corresponding to the next log distribution task based on the selector type and selector configuration information.

[0076] This embodiment configures the relevant information of the determined selector type according to the selector configuration information to obtain the selector corresponding to the next log distribution task, so that the next log distribution task can distribute logs according to the determined selector and the Channel corresponding to the selector.

[0077] It should be further explained that, based on any of the above embodiments, in order to improve the efficiency of log processing, after determining the selector corresponding to the next log distribution task according to the selector type and selector configuration information, the method may further include: determining parsing rules based on the data source and the selector, and sending them to the destination, so that different destination sinks can parse log information with different characteristics (such as IP information, time information, etc.) according to different parsing rules. This enables differentiated parsing of log information with different characteristics, improves the scalability of the collection system, and by selectively classifying, parsing, and storing log information, rather than using different parsing rules to parse redundant data, the amount of log information processed is greatly reduced, saving the computing resources used by the system.

[0078] The selector determination method provided in this embodiment of the invention may include: S101, obtaining resource information and corresponding data type corresponding to the current log distribution task; wherein, the resource information includes resource information of the event temporary buffer and host resource information; S102, predicting selector information based on the resource information and data type; wherein, the selector information includes selector type and selector configuration information; S103, determining the selector corresponding to the next log distribution task based on the selector type and selector configuration information. Compared with the current manual configuration of selectors, this invention predicts selector information based on the resource information and data type corresponding to the current log distribution task. The selector information includes selector type and corresponding selector parameters. Since each selector type can correspond to multiple selector parameters, it is equivalent to having many selectors, thus improving the accuracy of selector determination.

[0079] This invention provides a method for automatically switching and optimizing channel selector configurations based on scenario requirements, targeting fixed channel selectors. By introducing a monitoring module, it adjusts the channel selectors in real time and establishes a gradient boosting algorithm model to achieve the scalability and learning potential of Flume's collected data, thereby improving the efficiency of subsequent data cleaning.

[0080] For a clearer understanding of this invention, please refer to the following details. Figure 2 , Figure 2 A flowchart illustrating a selector determination method provided in an embodiment of the present invention may specifically include:

[0081] S201, Obtain the target data sent by the data source.

[0082] This embodiment obtains the Event sent by the Source.

[0083] S202, Write the target data to the event temporary buffer based on the current selector.

[0084] This embodiment attempts to write an event to the Channel.

[0085] S203 uses the monitoring module to obtain resource information of the event temporary cache area, as well as host resource information corresponding to the event temporary cache area.

[0086] In this embodiment, the Channel writes resource information from the event temporary buffer to the monitoring module. Simultaneously, the Monitor module queries the Host host for its computing, storage, and other resource information. In this embodiment, host resource information may include hard drive, memory, CPU, network interface card (NIC), and bandwidth; Channel resource information may include task execution time, host resource usage by the task, and communication rate between the Channel and the host.

[0087] S204, based on the resource information of the event temporary buffer and the host's resource information, uses the gradient boosting algorithm to predict the selector type and selector configuration information.

[0088] In this embodiment, the resource information of the event temporary buffer and the host resource information are the currently remaining resource information. For example, this embodiment can predict the upcoming workload based on the channel's runtime parameters using a linear regression algorithm model, dynamically configure the channel selector's decision mode, whether to use a multiplexed channel selector, or provide suggested values ​​to increase or decrease the number of event replication nodes (the number of hosts). Each node in this embodiment uses a HA (High Availability) technology solution. This embodiment can be based on a gradient boosting algorithm, importing the task execution type and the remaining computing and storage resources to determine whether to switch channel selectors and modify the channel selector's parameters accordingly.

[0089] S205: Obtain the target selector based on the selector type and selector configuration information, and distribute the data sent from the data source based on the target selector.

[0090] For easier understanding, please refer to Figure 3 , Figure 3 This is a framework diagram of a selector determination method provided in an embodiment of the present invention. Figure 3As can be seen, the system predicts selector information based on the resource information of the Channel and the host monitored by the monitoring module, and distributes logs based on the selector determined by the selector information. In this embodiment, the Source is the component in Flume that acquires data and writes it into the Channel. It is the starting point of the data flow, responsible for collecting data from various data sources and passing it to the next component; the Channel is a passive storage device responsible for temporarily storing the data sent by the Source; the Sink is responsible for consuming the data in the Channel and sending it to the corresponding destination. The Channel Selector is responsible for distributing the data from the Source to the Channel.

[0091] Currently, there are only two fixed Channel selectors to choose from. In production environments, situations may arise where resources cannot be fully utilized or resource overflow can cause node crashes. Since the existing Channel selector configuration mode is manual, it cannot address complex business scenarios in production environments, such as when event replication is needed to ensure data security, or when event-driven Channel selectors are needed for complex data source types.

[0092] In this embodiment of the invention, based on different scenarios, the monitoring model can determine how to configure and select the channel selector for the current business scenario based on the collection of indicator items and through algorithm optimization, and automatically switch between them. Moreover, the advantage of machine learning models lies in their growth potential. As the number of collected task samples increases, the prediction of the current business scenario becomes more accurate, making the configuration of the channel selector more accurate. As the number of model parameters increases, a unique channel selector can be customized for each business scenario, optimizing the channel selector configuration to its theoretical optimal level.

[0093] The selector determining device provided in the embodiments of the present invention will be described below. The selector determining device described below and the selector determining method described above can be referred to in correspondence.

[0094] Figure 4 A schematic diagram of a selector determining device provided in an embodiment of the present invention may include:

[0095] The input data acquisition module 100 is used to acquire the resource information corresponding to the current log distribution task, as well as the corresponding data type; wherein, the resource information includes the resource information of the event temporary buffer and the host resource information;

[0096] The prediction module 200 is used to predict selector information based on the resource information and the data type; wherein the selector information includes selector type and selector configuration information.

[0097] The selector determination module 300 is used to determine the selector corresponding to the next log distribution task based on the selector type and the selector configuration information.

[0098] Furthermore, based on the above embodiments, the prediction module 200 may include:

[0099] The selector type determination unit is used to predict the selector type based on the resource information and the data type using a machine learning model; wherein the machine learning model is a model whose prediction accuracy increases with the number of predictions; the selector type includes a specified channel selector and a multiplexed channel selector.

[0100] The selector configuration information determination unit is used to determine the selector configuration information based on the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task using the machine learning model; wherein, the selector configuration information includes at least one of the following: number of nodes, number of temporary caches, number of hosts, memory usage configuration of temporary caches, storage configuration of temporary caches, and cache configuration of temporary caches.

[0101] Furthermore, based on the above embodiments, the selector determining device may further include:

[0102] The task execution time determination module is used to determine the task execution time of the next log distribution task based on the selector corresponding to the next log distribution task.

[0103] The comparison result determination module is used to compare the task execution time corresponding to the current log distribution task with the task execution time corresponding to the next log distribution task, and to compare the machine learning model parameters corresponding to the current log distribution task with the machine learning model parameters corresponding to the next log distribution task, and determine the comparison result.

[0104] The parameter adjustment module is used to adjust the parameters of the machine learning model based on the comparison results.

[0105] The machine learning model parameter adjustment module is used to determine whether to use the adjusted machine learning model for prediction, and to continuously adjust the parameters of the machine learning model until the task execution time corresponding to each log distribution task is maintained within a set range.

[0106] Furthermore, based on any of the above embodiments, the input data acquisition module 100 may include:

[0107] The resource information acquisition unit for the event temporary buffer is used to acquire the resource information written to each event temporary buffer using the monitoring module;

[0108] The host information acquisition unit is used to acquire the host resource information and the data type using the monitoring module.

[0109] Furthermore, based on the above embodiments, the prediction module 200 may include:

[0110] The consumed host resource determination unit is used to determine the host resources consumed by the current log distribution task, as well as the resources consumed by the event temporary buffer.

[0111] The remaining resource determination unit is used to determine the remaining host resources and the remaining event temporary buffer resources based on the resource information of the event temporary buffer and the resources of the consumed event temporary buffer, as well as the host resource information and the consumed host resources.

[0112] The selector type prediction unit is used to predict the selector type based on the remaining host resources, the remaining event temporary buffer resources, and the task execution type.

[0113] The execution time determination unit is used to determine whether the execution time of the current log distribution task is greater than the execution time of the previous log distribution task.

[0114] The selector configuration information determination unit is used to compare the selector configuration information corresponding to the previous log distribution task with the selector configuration information corresponding to the current log distribution task if the execution time of the current log distribution task is greater than the execution time of the previous log distribution task, and determine the changed selector configuration information.

[0115] The selector configuration unit is used to configure the configuration information of the selector corresponding to the selector type according to the changed selector configuration information.

[0116] Furthermore, based on any of the above embodiments, the host resource information includes at least one of hard disk resource information, memory resource information, central processing unit resource information, network card resource information, and bandwidth resource information, and the resource information of the event temporary buffer includes at least one of task execution time, host resources occupied by the task, event temporary buffer, and inter-host communication rate.

[0117] Furthermore, based on any of the above embodiments, the prediction module 200 may include:

[0118] The task quantity and task type determination unit is used to predict the task quantity and task type of the next log distribution task based on the resource information and the data type using a linear regression model.

[0119] The selector information determination unit is used to determine the selector information based on the predicted task volume and task type.

[0120] It should be noted that the order of the modules and units in the selector determination device can be changed without affecting the logic.

[0121] Figure 4 For a description of the features in the corresponding embodiments, please refer to Figure 4 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0122] The selector determination device provided in this embodiment of the invention may include: an input data acquisition module 100, used to acquire resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer and host resource information; a prediction module 200, used to predict selector information based on the resource information and the data type; wherein the selector information includes selector type and selector configuration information; and a selector determination module 300, used to determine the selector corresponding to the next log distribution task based on the selector type and the selector configuration information. Compared with the current manual configuration of selectors, this invention predicts selector information based on the resource information and data type corresponding to the current log distribution task. The selector information includes selector type and corresponding selector parameters. Since each selector type can correspond to multiple selector parameters, it is equivalent to having many selectors, thus improving the accuracy of selector determination.

[0123] The following describes a selector determination device provided by an embodiment of the present invention. The selector determination device described below can be referred to in correspondence with the selector determination method described above.

[0124] Figure 5 This is a schematic diagram of a selector determining device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the selector determining device includes: a memory 60 for storing computer programs;

[0125] The processor 61 is used to implement the steps of the selector determination method as described in the above embodiments when executing a computer program.

[0126] The selector device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0127] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0128] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the selector determination method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include, but is not limited to, data required during the selector determination process.

[0129] In some embodiments, the selector determining device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0130] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the selector determining device and may include more or fewer components than illustrated.

[0131] It is understood that if the selector determination method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0132] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the selector determination method described above.

[0133] The foregoing has provided a detailed description of a selector determination method, apparatus, device, and computer-readable storage medium provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0134] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0135] The present invention has provided a detailed description of a selector determination method, apparatus, device, and computer-readable storage medium. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of these embodiments are merely illustrative of the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.

Claims

1. A selector determination method, characterized in that, include: Obtain the resource information corresponding to the current log distribution task, as well as the corresponding data type; wherein, the resource information includes the resource information of the event temporary buffer and the host resource information; Based on the resource information and the data type, predict selector information; wherein, the selector information includes selector type and selector configuration information; The selector for the next log distribution task is determined based on the selector type and the selector configuration information. The selector information is predicted based on the resource information and the data type, including: Based on the resource information and the data type, a machine learning model is used to predict the selector type; wherein, the machine learning model is a gradient boosting tree model in which the prediction accuracy increases with the number of predictions; the selector type includes a specified channel selector and a multiplexed channel selector. The machine learning model is used to determine the selector configuration information based on the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task; wherein, the selector configuration information includes at least one of the following: number of nodes, number of temporary caches, number of hosts, memory usage configuration of temporary caches, storage configuration of temporary caches, and caching configuration of temporary caches.

2. The selector determination method according to claim 1, characterized in that, After determining the selector for the next log distribution task based on the selector type and the selector configuration information, the process further includes: Determine the task execution time for the next log distribution task based on the selector corresponding to the next log distribution task; The task execution time of the current log distribution task is compared with the task execution time of the next log distribution task, and the machine learning model parameters of the current log distribution task are compared with the machine learning model parameters of the next log distribution task to determine the comparison results. Adjust the parameters of the machine learning model based on the comparison results; The adjusted machine learning model is used for prediction, and the parameters of the machine learning model are continuously adjusted until the execution time of each log distribution task remains within the set range.

3. The selector determination method according to claim 1, characterized in that, The step of obtaining the resource information corresponding to the current log distribution task, and the corresponding data type, includes: The monitoring module is used to obtain resource information written to the temporary buffer of each event; The monitoring module is used to obtain the host resource information and the data type.

4. The selector determination method according to any one of claims 1 to 3, characterized in that, Based on the resource information and the data type, predict selector information, including: Determine the host resources consumed by the current log distribution task, as well as the resources consumed by the event temporary buffer. Based on the resource information of the event temporary buffer and the consumed resources of the event temporary buffer, as well as the host resource information and the consumed host resources, determine the remaining host resources and the remaining event temporary buffer resources; Based on the remaining host resources, the remaining event temporary buffer resources, and the task execution type, predict the selector type; Determine whether the execution time of the current log distribution task is greater than the execution time of the previous log distribution task; If the execution time of the current log distribution task is longer than that of the previous log distribution task, the selector configuration information corresponding to the previous log distribution task is compared with the selector configuration information corresponding to the current log distribution task to determine the selectedor configuration information that has changed. Configure the configuration information of the selector corresponding to the selectedor type according to the changed selectedor configuration information.

5. The selector determination method according to claim 1, characterized in that, The host resource information includes at least one of hard disk resource information, memory resource information, central processing unit resource information, network card resource information, and bandwidth resource information, and the resource information of the event temporary buffer includes at least one of task execution time, host resources occupied by the task, event temporary buffer, and inter-host communication rate.

6. The selector determination method according to claim 1, characterized in that, Based on the resource information and the data type, predict selector information, including: Based on the resource information and the data type, a linear regression model is used to predict the task volume and task type of the next log distribution task. The selector information is determined based on the predicted task volume and task type.

7. A selector determining device, characterized in that, include: The input data acquisition module is used to acquire the resource information corresponding to the current log distribution task, as well as the corresponding data type; wherein, the resource information includes the resource information of the event temporary buffer and the host resource information; The prediction module is used to predict selector information based on the resource information and the data type; wherein the selector information includes selector type and selector configuration information. The selector determination module is used to determine the selector corresponding to the next log distribution task based on the selector type and the selector configuration information. The prediction module includes: The selector type determination unit is used to predict the selector type based on the resource information and the data type using a machine learning model; wherein the machine learning model is a gradient boosting tree model in which the prediction accuracy increases with the number of predictions; and the selector type includes a specified channel selector and a multiplexed channel selector. The selector configuration information determination unit is used to determine the selector configuration information based on the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task using the machine learning model; wherein, the selector configuration information includes at least one of the following: number of nodes, number of temporary caches, number of hosts, memory usage configuration of temporary caches, storage configuration of temporary caches, and cache configuration of temporary caches.

8. A selector determining device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the selector determination method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the selector determination method as described in any one of claims 1 to 6.

Images