Selector determination method, device and equipment and computer readable storage medium

Abstract

The invention discloses a selector determination method, device and equipment and a computer readable storage medium, and is applied to the technical field of log collection, and the method comprises the following steps: obtaining resource information corresponding to a current log distribution task and a corresponding data type; wherein the resource information comprises resource information of the event temporary cache region and host resource information; predicting selector information according to resource information and the data type; wherein the selector information comprises a selector type and selector configuration information; and determining a selector corresponding to the next log distribution task according to the selector type and the selector configuration information. Compared with a current manual configuration selector, the method has the advantages that the selector type required by the next log distribution task and the corresponding selector parameters are predicted according to the resource information and the data type corresponding to the current log distribution task, and each selector type can correspond to a plurality of selector parameters, which is equivalent to the existence of a plurality of selectors with different parameters; and the determination accuracy of the selector is improved.

Description

Technical Field

[0001] The present invention relates to the field of log collection technology, and in particular to a selector determination method, device, equipment and computer-readable storage medium. Background Art

[0002] In the era of big data, Flume, as an excellent collection tool, is used in multiple scenarios. Collecting and distributing logs is a very important function. In the traditional Flume data collection process, the type of Channel selector is determined by manual configuration. There are mainly two types of Channel selectors. The manual method makes the selection of Channel selector type inaccurate, thus affecting the data processing efficiency.

[0003] It can be seen that how to improve the accuracy of Channel selector selection is a technical problem that technical personnel in this field need to solve urgently. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a selector determination method, device, equipment and computer-readable storage medium, which solve the technical problem of low accuracy of Channel selector selection in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a selector determination method, comprising:

[0006] Obtain resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer area and host resource information;

[0007] Predicting selector information according to the resource information and the data type; wherein the selector information includes a selector type and selector configuration information;

[0008] A selector corresponding to the next log distribution task is determined according to the selector type and the selector configuration information.

[0009] In some embodiments, predicting selector information according to the resource information and the data type includes:

[0010] According to the resource information and the data type, the selector type is predicted using a machine learning model; wherein the machine learning model is a model whose prediction accuracy increases with an increase in the number of predictions; the selector type includes a designated channel selector and a multiplexed channel selector;

[0011] The selector configuration information is determined by using the machine learning model according to the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task; wherein the selector configuration information includes at least one of the number of nodes, the number of temporary cache areas, the number of hosts, the memory usage configuration of the temporary cache area, the storage configuration of the temporary cache area, and the cache configuration of the temporary cache area.

[0012] In some embodiments, after determining the selector corresponding to the next log distribution task according to the selector type and the selector configuration information, the method further includes:

[0013] Determine a task execution time for executing the next log distribution task based on the selector corresponding to the next log distribution task;

[0014] Compare the task execution time corresponding to the current log distribution task with the task execution time corresponding to the next log distribution task, and compare the machine learning model parameters corresponding to the current log distribution task with the machine learning model parameters corresponding to the next log distribution task to determine the comparison result;

[0015] Adjusting the parameters of the machine learning model according to the comparison result;

[0016] Determine to use the adjusted machine learning model for prediction, and continuously adjust the parameters of the machine learning model until the task execution time corresponding to each log distribution task is maintained within a set range.

[0017] In some embodiments, the obtaining of resource information corresponding to the current log distribution task and the corresponding data type includes:

[0018] Use the monitoring module to obtain the resource information written into the temporary buffer area of ​​each event;

[0019] The monitoring module is used to obtain the host resource information and the data type.

[0020] In some embodiments, predicting selector information according to the resource information and the data type includes:

[0021] Determine the host resources consumed by the current log distribution task and the resources consumed by the event temporary buffer area;

[0022] Determine remaining host resources and remaining event temporary buffer resources according to the resource information of the event temporary buffer and the consumed resources of the event temporary buffer, as well as the host resource information and the consumed host resources;

[0023] Predicting the selector type according to the remaining host resources, the remaining event temporary buffer resources, and the task execution type;

[0024] Determine whether the execution time of the current log distribution task is greater than the execution time of the previous log distribution task;

[0025] If the execution time of the current log distribution task is greater than the execution time of the previous log distribution task, compare the selector configuration information corresponding to the previous log distribution task with the selector configuration information corresponding to the current log distribution task to determine the changed selector configuration information;

[0026] The configuration information of the selector corresponding to the selector type is configured according to the changed selector configuration information.

[0027] In some embodiments, the host resource information includes at least one of hard disk resource information, memory resource information, central processing unit resource information, network card resource information and bandwidth resource information, and the resource information of the event temporary buffer area includes at least one of task execution time, task occupied host resources, event temporary buffer area and communication rate between hosts.

[0028] In some embodiments, predicting selector information according to the resource information and the data type includes:

[0029] According to the resource information and the data type, a linear regression model is used to predict the task volume and task type of the next log distribution task;

[0030] The selector information is determined according to the predicted task volume and task type.

[0031] The embodiment of the present invention further provides a selector determination device, comprising:

[0032] The input data acquisition module is used to obtain the resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes the resource information of the event temporary buffer area and the host resource information;

[0033] A prediction module, used to predict selector information according to the resource information and the data type; wherein the selector information includes a selector type and selector configuration information;

[0034] A selector determination module is used to determine the selector corresponding to the next log distribution task according to the selector type and the selector configuration information.

[0035] The embodiment of the present invention further provides a selector determination device, including:

[0036] Memory for storing computer programs;

[0037] A processor is used to execute the computer program to implement the steps of the selector determination method as described above.

[0038] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the selector determination method described above are implemented.

[0039] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0040] To solve the above technical problems, an embodiment of the present invention provides a selector determination method, which may include: obtaining resource information corresponding to the current log distribution task, and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer area and host resource information; based on the resource information and the data type, predicting selector information; wherein the selector information includes a selector type and selector configuration information; determining the selector corresponding to the next log distribution task based on the selector type and the selector configuration information.

[0041] It can be seen from the above technical solution that the beneficial effect of the present invention is that compared with the current manually configured selector, the present invention predicts selector information based on the resource information and data type corresponding to the current log distribution task, and the selector information includes the selector type and the corresponding selector parameters. Since each selector type can correspond to multiple selector parameters, it is equivalent to the existence of many selectors, which improves the accuracy of selector determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flow chart of a selector determination method provided by an embodiment of the present invention;

[0044] Figure 2 An example flowchart of a selector determination method provided by an embodiment of the present invention;

[0045] Figure 3 A framework diagram of a selector determination method provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the structure of a selector determination device provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the structure of a selector determination device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0050] In the era of big data, Flume is an excellent collection tool that is used in multiple scenarios. Collecting and distributing logs is a very important function. After the Source (data acquisition component) in the Flume log distribution architecture encapsulates the collected data into Event (data event), it will copy and distribute it to each Channel (event temporary buffer) through the Channel Selector (selector). The Sink behind the Channel (Sink is responsible for consuming the data in the Channel and sending it to the corresponding destination) obtains data from the Channel, thereby writing the log information to the destination.

[0051] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0052] Next, a selector determination method provided by an embodiment of the present invention is described in detail. Figure 1 A flowchart of a selector determination method provided by an embodiment of the present invention, the method may include:

[0053] S101, obtaining resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer area and host resource information.

[0054] The execution subject of this embodiment is an electronic device. This embodiment does not limit the specific electronic device. For example, the electronic device in this embodiment can be a computer; or the electronic device in this embodiment can be a server. The log distribution task in this embodiment refers to the task of distributing logs using a selector. This embodiment does not limit the specific resource information. The resource information in this embodiment is the resource information of the current host and the event temporary buffer (Channel), wherein the relationship between the time temporary buffer and the host is 1 to 1. The resource information in this embodiment refers to various configurations, usage and status data related to the computer host hardware and software; the event temporary buffer resource information in this embodiment refers to the Channel resource information, which usually refers to the hardware and software configuration, status and usage data related to communication or data transmission. The event temporary buffer Channel in this embodiment is a passive memory responsible for temporarily storing the data sent by the Source. The data type in this embodiment is determined by the data characteristics of the data source itself and the data structure of the data source, which is fixed. In repeated tasks, unless the business party itself changes the data source to be collected, changes will occur, which is automatically sensed by the Channel selector selector itself.

[0055] It should be further explained that the host resource information includes at least one of hard disk resource information, memory resource information, central processing unit resource information, network card resource information and bandwidth resource information, and the resource information of the event temporary buffer area includes at least one of task execution time, task occupied host resources, event temporary buffer area and communication rate between hosts.

[0056] It should be further explained that, based on the above embodiment, the above-mentioned acquisition of resource information corresponding to the current log distribution task and the corresponding data type may include: using the monitoring module to acquire resource information written in each event temporary buffer area; using the monitoring module to acquire host resource information and data type. In this embodiment, the resource information is monitored in real time to acquire the resource information in a timely manner.

[0057] S102, predicting selector information according to resource information and data type; wherein the selector information includes selector type and selector configuration information.

[0058] This embodiment is not limited to predicting selector information according to resource information and data type, as long as the model can obtain output (selector information) according to input (resource type and data type). For example, the model in this embodiment can be GBDT (Gradient Boosting Decision Tree), which uses decision tree as base learner and is suitable for regression and classification tasks, or it can also be XGBoost (eXtreme Gradient Boosting): an efficient implementation of GBDT, which significantly improves computing speed and performance; or it can also be LightGBM (Light Gradient Boosting Machine): based on leaf growth strategy and histogram technology, suitable for large data sets, fast speed, and low memory occupancy; or it can also be CatBoost (Categorical Boosting, Gradient Boosting of Classification and Regression Trees), which is suitable for data sets containing category features, does not require additional hot encoding processing, and reduces time cost; or the model for prediction in this embodiment can also be a linear regression algorithm model. The selector type in this embodiment is the currently existing selector type. For example, the selector in this embodiment can be a Replicating Channel Selector. This selector copies each event to all Channels storage devices. This selector is suitable for scenarios with high availability and fault tolerance, such as redundant storage of data in multiple Channels. The problem brought by reliability is that sufficient storage space and performance are required to process duplicate data, otherwise a large amount of duplicate data in the production environment may cause system overload. Multiplexing Channel Selector This selector routes events to different Channels based on the Header attribute of the time (the Header attribute refers to the header information in the HTTP request or response, which is used to transmit additional data and control the processing method of the message). It is suitable for scenarios where data needs to be diverted according to conditions such as the source of the event type. This scenario cannot guarantee the reliability of the database because there is no redundancy, which will lead to data loss. This embodiment does not limit the specific selector configuration information, as long as the method of selecting the replication and distribution to each Channel can be determined based on the selector configuration information. For example, the selector configuration information in this embodiment may be the number of Channels and the number of hosts; or the selector configuration information in this embodiment may be the resource configuration of the Channel.

[0059] It should be further explained that the above-mentioned prediction selector information based on resource information and data type may include:

[0060] S1021, predicting a selector type using a machine learning model according to resource information and data types; wherein the machine learning model is a model whose prediction accuracy increases as the number of predictions increases; and the selector type includes a designated channel selector and a multiplexed channel selector;

[0061] S1022, using a machine learning model to determine the selector configuration information based on the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task; wherein the selector configuration information includes at least one of the number of nodes, the number of temporary cache areas, the number of hosts, the memory usage configuration of the temporary cache area, the storage configuration of the temporary cache area, and the cache configuration of the temporary cache area.

[0062] The model in this embodiment is a machine learning model. The accuracy of the prediction of the machine learning model in this embodiment will increase with the increase of the number of predictions, until the task execution time is maintained within a certain range when the predicted selector is used for log distribution. The number of nodes in this embodiment refers to the number of host nodes corresponding to the temporary cache area; the memory usage configuration of the temporary cache area in this embodiment is to increase the memory usage configuration of each Channel; the cache configuration in this embodiment refers to whether the cache is turned on. This embodiment can determine that the task execution time has become longer according to the task execution time, which means that the task execution speed has slowed down. At this time, compared with the last input parameter, whether the host node has become less, the CPU (central processing unit) resources have become less, and the memory occupancy is too high, to reduce the number of channels, increase the memory usage configuration of each Channel, storage configuration, whether to turn on the cache, etc., to modify the configuration information of the Channel selector. Each time a task is executed, the results output by the monitoring model must be compared with the results of the previous execution, and the Channel selector resource configuration (number of nodes, weight distribution, resource ratio) can be adjusted at any time. This embodiment improves the accuracy of the selector configuration by providing specific selector types and selector configuration information.

[0063] It should be further explained that, based on any of the above embodiments, predicting the selector information according to the resource information and the data type may include: predicting the task volume and task type of the next log distribution task using a linear regression model according to the resource information and the data type; determining the selector information according to the predicted task volume and task type. This embodiment needs to determine the independent variable (explanatory variable) and the dependent variable (explained variable) before establishing the linear regression model. In this scenario, the independent variable may include the task volume and the task type, while the dependent variable is the task volume and task type of the next log distribution task. This embodiment determines the selector information according to the predicted task volume and task type, which means determining the selector type and its corresponding configuration information. This embodiment considers both the task volume and the task type when making predictions. For example, when the task volume is relatively large and the current resource information is insufficient, you can choose to specify a channel selector and not cache it; when the task is more important in this embodiment, you can cache it.

[0064] It should be further explained that after the selector corresponding to the next log distribution task is determined according to the selector type and the selector configuration information, it can also include: determining the task execution time of the next log distribution task based on the selector corresponding to the next log distribution task; comparing the task execution time corresponding to the current log distribution task with the task execution time corresponding to the next log distribution task, and comparing the machine learning model parameters corresponding to the current log distribution task with the machine learning model parameters corresponding to the next log distribution task, and determining the comparison result; adjusting the parameters of the machine learning model according to the comparison result; determining to use the adjusted machine learning model for prediction, and constantly adjusting the parameters of the machine learning model until the task execution time corresponding to each log distribution task is maintained within the set range. This embodiment does not limit the specific setting range, for example, the setting range can be determined according to the shortest task execution time; or the setting range in this embodiment can also be determined according to the average task execution time. This embodiment can adjust the parameters of the machine learning model while predicting to improve the accuracy of the machine learning model in predicting.

[0065] It should be further explained that, based on any of the above embodiments, the above prediction selector information based on resource information and data type may include:

[0066] S1: Determine the host resources consumed by the current log distribution task and the resources consumed by the event temporary buffer area.

[0067] The resource information in this embodiment refers to the resource information before the task is executed. When the resource information in this embodiment is the resource information after the log distribution task is executed, the resource information is the remaining resource information and no calculation is required. This embodiment can collect the host resources (memory, hard disk, CPU, bandwidth) consumed by the task. The execution occupancy loss of each node is different, and the average resource consumption is calculated through variance.

[0068] S2: Determine remaining host resources and remaining event temporary buffer resources according to resource information of the event temporary buffer and consumed resources of the event temporary buffer, as well as host resource information and consumed host resources.

[0069] S3: Predict the selector type based on the remaining host resources, remaining event temporary buffer resources, and the task execution type.

[0070] S4: Determine whether the execution time of the current log distribution task is greater than the execution time of the previous log distribution task.

[0071] In this embodiment, the execution time of this round of tasks (each Channel will return its own execution time on the host) is calculated based on the variance to obtain the average regression time, that is, the execution time of the log distribution task.

[0072] S5: If the execution time of the current log distribution task is greater than the execution time of the previous log distribution task, compare the selector configuration information corresponding to the previous log distribution task with the selector configuration information corresponding to the current log distribution task to determine the changed selector configuration information.

[0073] S6: Configure the configuration information of the selector corresponding to the selector type according to the changed selector configuration information.

[0074] This embodiment considers the changed selector configuration information when determining the selector configuration information, and thus considers which configuration change causes the change in task execution time, so as to accurately adjust the selector configuration information.

[0075] S103: Determine a selector corresponding to the next log distribution task according to the selector type and selector configuration information.

[0076] This embodiment configures the relevant information of the determined selector type according to the selector configuration information, and can obtain the selector corresponding to the next log distribution task, so that the next log distribution task can distribute logs according to the determined selector and the Channel corresponding to the selector.

[0077] It should be further explained that, based on any of the above embodiments, in order to improve the efficiency of log processing, after determining the selector corresponding to the next log distribution task according to the selector type and selector configuration information, it can also include: determining the parsing rules based on the data source and the selector, and sending them to the destination, so that different destination sinks can be parsed accordingly according to different parsing rules for log information with different characteristics (such as IP information, time information, etc.). That is, log information with different characteristics can be parsed differentially, which improves the scalability of the collection system, and by selectively classifying and parsing the log information and storing it, rather than using different parsing rules to parse redundant data, the amount of log information data processed is greatly reduced, saving the computing resources used by the system.

[0078] The selector determination method provided by the embodiment of the present invention may include: S101, obtaining resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer area and host resource information; S102, predicting selector information based on the resource information and data type; wherein the selector information includes selector type and selector configuration information; S103, determining the selector corresponding to the next log distribution task based on the selector type and selector configuration information. Compared with the current manual configuration of the selector, the present invention predicts the selector information based on the resource information and data type corresponding to the current log distribution task, and the selector information includes the selector type and the corresponding selector parameters. Since each selector type can correspond to multiple selector parameters, it is equivalent to having many selectors, which improves the accuracy of the selector determination.

[0079] The design of the present invention provides a method for automatically switching and optimizing the configuration of the Channel selector according to the scene requirements for the fixed Channel selector. By introducing the monitoring module, the Channel selector is adjusted in real time, and a gradient boosting algorithm model is established to achieve the growth and learning of Flume collected data and improve the efficiency of subsequent data cleaning.

[0080] In order to make the present invention easier to understand, please refer to Figure 2 , Figure 2 An example flow chart of a selector determination method provided in an embodiment of the present invention may specifically include:

[0081] S201, obtaining target data sent by a data source.

[0082] This embodiment obtains the Event sent by the Source.

[0083] S202, writing target data into a temporary event buffer based on the current selector.

[0084] This example attempts to write an Event to a Channel.

[0085] S203: Utilize the monitoring module to obtain resource information of the temporary event buffer area and host resource information corresponding to the temporary event buffer area.

[0086] In this embodiment, the Channel writes the resource information of the temporary event buffer to the monitoring module, and the monitoring module Monitor will query the Host, that is, the host, for resource conditions such as computing and storage. In this embodiment, the host resource information may include resource information such as hard disk, memory, CPU, network card and bandwidth; the Channel resource information may include task execution time, task occupation of host resources, and communication rate between Channel and host.

[0087] S204, predicting the selector type and selector configuration information based on the resource information of the event temporary buffer and the resource information of the host using a gradient boosting algorithm.

[0088] The resource information of the temporary event buffer and the resource information of the host in this embodiment are the currently remaining resource information. For example, this embodiment can use the linear regression algorithm model according to the time parameters of the Channel operation to predict the next task volume, dynamically configure the decision mode of the Channel selector, whether to use a multiplexed channel selector, or give a recommended value to increase or decrease the number of event replication nodes (the number of hosts). In this embodiment, each node uses the HA (High Availability) technical solution. This embodiment can import the task execution type based on the gradient boosting algorithm, the remaining computing and storage resources at this time, determine whether the channel selector should be switched, and modify the parameters of the channel selector.

[0089] S205: Obtain a target selector based on the selector type and the selector configuration information, and distribute the data sent by the data source based on the target selector.

[0090] For easier understanding, please refer to Figure 3 , Figure 3 A framework diagram of a selector determination method provided by an embodiment of the present invention, from Figure 3It can be seen that based on the resource information of the Channel monitored by the monitoring module and the resource information of the host, the selector information will be predicted, and the log distribution will be performed based on the selector determined by the selector information. In this embodiment, the Source is the component that obtains data in Flume and writes the data to the Channel. It is the starting point of the data flow, responsible for collecting data from various data sources and passing it to the next component; Channel is a passive storage device responsible for temporarily storing data sent by the Source; Sink: Sink is responsible for consuming data in the Channel and sending it to the corresponding destination. Channel Selector: Responsible for distributing the data from the Source to the Channel.

[0091] Currently, there are only two fixed Channel selectors to choose from. In the production environment, resources may not be fully utilized or resource overflow may cause node downtime. Since the existing Channel selector configuration mode is manual configuration, it cannot be used for complex business scenarios in the production environment, such as when event replication scenarios are needed to ensure data security, and when event-driven Channel selectors are needed for complex data source types.

[0092] In the embodiment of the present invention, based on different scenarios, the monitoring model can obtain how to configure and select the Channel selector for the current business scenario through algorithm optimization according to the collection of indicator items, and automatically switch. In addition, the advantage of the machine learning model lies in its growth potential. As the number of collected task samples increases, the prediction of the current business scenario will be more accurate, making the configuration of the Channel selector more accurate. As the number of model parameters increases, the Channel selector unique to each business scenario can be customized, and the configuration of the Channel selector can be optimized to the theoretical optimum.

[0093] The following is an introduction to a selector determination device provided in an embodiment of the present invention. The selector determination device described below and the selector determination method described above can be referenced to each other.

[0094] Figure 4 A schematic diagram of a structure of a selector determination device provided by an embodiment of the present invention may include:

[0095] The input data acquisition module 100 is used to acquire resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer area and host resource information;

[0096] A prediction module 200, configured to predict selector information according to the resource information and the data type; wherein the selector information includes a selector type and selector configuration information;

[0097] The selector determination module 300 is used to determine the selector corresponding to the next log distribution task according to the selector type and the selector configuration information.

[0098] Further, based on the above embodiment, the prediction module 200 may include:

[0099] A selector type determination unit, configured to predict the selector type using a machine learning model according to the resource information and the data type; wherein the machine learning model is a model whose prediction accuracy increases with an increase in the number of predictions; and the selector type includes a designated channel selector and a multiplexed channel selector;

[0100] A selector configuration information determination unit is used to determine the selector configuration information based on the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task by using the machine learning model; wherein the selector configuration information includes at least one of the number of nodes, the number of temporary cache areas, the number of hosts, the memory usage configuration of the temporary cache area, the storage configuration of the temporary cache area, and the cache configuration of the temporary cache area.

[0101] Further, based on the above embodiment, the above selector determination device may further include:

[0102] A task execution time determination module, used to determine the task execution time of the next log distribution task based on the selector corresponding to the next log distribution task;

[0103] A comparison result determination module is used to compare the task execution time corresponding to the current log distribution task with the task execution time corresponding to the next log distribution task, and to compare the machine learning model parameters corresponding to the current log distribution task with the machine learning model parameters corresponding to the next log distribution task to determine the comparison result;

[0104] A parameter adjustment module, used to adjust the parameters of the machine learning model according to the comparison result;

[0105] The machine learning model parameter adjustment module is used to determine the use of the adjusted machine learning model for prediction and continuously adjust the parameters of the machine learning model until the task execution time corresponding to each log distribution task is maintained within a set range.

[0106] Further, based on any of the above embodiments, the input data acquisition module 100 may include:

[0107] The resource information acquisition unit of the event temporary buffer area is used to acquire the resource information written in each event temporary buffer area by using the monitoring module;

[0108] The host information acquisition unit is used to acquire the host resource information and the data type by using the monitoring module.

[0109] Further, based on the above embodiment, the prediction module 200 may include:

[0110] A consumed host resource determination unit, used to determine the host resources consumed by the current log distribution task and the resources consumed in the event temporary buffer area;

[0111] a remaining resource determination unit, configured to determine remaining host resources and remaining event temporary buffer resources according to the resource information of the event temporary buffer and the consumed event temporary buffer resources, as well as the host resource information and the consumed host resources;

[0112] A selector type prediction unit, configured to predict the selector type according to the remaining host resources, the remaining event temporary buffer resources, and the task execution type;

[0113] An execution time judgment unit, used to determine whether the execution time of the current log distribution task is greater than the execution time of the previous log distribution task;

[0114] A selector configuration information determining unit, configured to compare the selector configuration information corresponding to the previous log distribution task with the selector configuration information corresponding to the current log distribution task if the execution time of the current log distribution task is greater than the execution time of the previous log distribution task, and determine the changed selector configuration information;

[0115] The selector configuration unit is used to configure the configuration information of the selector corresponding to the selector type according to the changed selector configuration information.

[0116] Further, based on any of the above embodiments, the host resource information includes at least one of hard disk resource information, memory resource information, central processing unit resource information, network card resource information and bandwidth resource information, and the resource information of the event temporary buffer area includes at least one of task execution time, task occupied host resources, event temporary buffer area and communication rate between hosts.

[0117] Further, based on any of the above embodiments, the prediction module 200 may include:

[0118] A task amount and task type determination unit, used to predict the task amount and task type of the next log distribution task by using a linear regression model according to the resource information and the data type;

[0119] The selector information determining unit is used to determine the selector information according to the predicted task volume and task type.

[0120] It should be noted that the order of the modules and units in the above selector determination device can be changed without affecting the logic.

[0121] Figure 4 The description of the features in the corresponding embodiments can be found in Figure 4 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0122] The selector determination device provided by the embodiment of the present invention may include: an input data acquisition module 100, used to obtain resource information corresponding to the current log distribution task, and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer area and host resource information; a prediction module 200, used to predict selector information based on the resource information and the data type; wherein the selector information includes selector type and selector configuration information; a selector determination module 300, used to determine the selector corresponding to the next log distribution task based on the selector type and the selector configuration information. Compared with the current manually configured selector, the present invention predicts selector information based on the resource information and data type corresponding to the current log distribution task, and the selector information includes selector type and corresponding selector parameters. Since each selector type can correspond to multiple selector parameters, it is equivalent to having many selectors, which improves the accuracy of selector determination.

[0123] A selector determination device provided by an embodiment of the present invention is introduced below. The selector determination device described below and the selector determination method described above can be referred to each other.

[0124] Figure 5 A schematic diagram of a selector determination device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the selector determination device includes: a memory 60 for storing a computer program;

[0125] The processor 61 is used to implement the steps of the selector determination method in the above embodiment when executing a computer program.

[0126] The selector determination device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0127] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0128] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the selector determination method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include but is not limited to data required in the selector determination process, etc.

[0129] In some embodiments, the selector determination device may further include a display screen 62 , an input and output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0130] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the selector determination device, and may include more or less components than shown in the figure.

[0131] It is understandable that if the selector determination method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the current technology or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk and other media that can store program codes.

[0132] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the selector determination method described above are implemented.

[0133] The above is a detailed introduction to a selector determination method, device, equipment and computer-readable storage medium provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0134] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0135] The above is a detailed introduction to a selector determination method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A selector determination method, characterized in that: include: Obtain resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes resource information of the event temporary buffer area and host resource information; Predicting selector information according to the resource information and the data type; wherein the selector information includes a selector type and selector configuration information; A selector corresponding to the next log distribution task is determined according to the selector type and the selector configuration information.

2. The selector determination method according to claim 1, characterized in that: The predicting selector information according to the resource information and the data type includes: According to the resource information and the data type, the selector type is predicted using a machine learning model; wherein the machine learning model is a model whose prediction accuracy increases with an increase in the number of predictions; the selector type includes a designated channel selector and a multiplexed channel selector; The selector configuration information is determined by using the machine learning model according to the configuration information corresponding to the previous log distribution task and the configuration information corresponding to the current log distribution task; wherein the selector configuration information includes at least one of the number of nodes, the number of temporary cache areas, the number of hosts, the memory usage configuration of the temporary cache area, the storage configuration of the temporary cache area, and the cache configuration of the temporary cache area.

3. The selector determination method according to claim 2, characterized in that: After determining the selector corresponding to the next log distribution task according to the selector type and the selector configuration information, the method further includes: Determine a task execution time for executing the next log distribution task based on the selector corresponding to the next log distribution task; Compare the task execution time corresponding to the current log distribution task with the task execution time corresponding to the next log distribution task, and compare the machine learning model parameters corresponding to the current log distribution task with the machine learning model parameters corresponding to the next log distribution task to determine the comparison result; Adjusting the parameters of the machine learning model according to the comparison result; Determine to use the adjusted machine learning model for prediction, and continuously adjust the parameters of the machine learning model until the task execution time corresponding to each log distribution task is maintained within a set range.

4. The selector determination method according to claim 1, characterized in that: The resource information corresponding to the current log distribution task and the corresponding data type are obtained, including: Use the monitoring module to obtain the resource information written into the temporary buffer area of ​​each event; The monitoring module is used to obtain the host resource information and the data type.

5. The selector determination method according to any one of claims 1 to 4, characterized in that: Predict selector information according to the resource information and the data type, including: Determine the host resources consumed by the current log distribution task and the resources consumed by the event temporary buffer area; Determine remaining host resources and remaining event temporary buffer resources according to the resource information of the event temporary buffer and the consumed resources of the event temporary buffer, as well as the host resource information and the consumed host resources; Predicting the selector type according to the remaining host resources, the remaining event temporary buffer resources, and the task execution type; Determine whether the execution time of the current log distribution task is greater than the execution time of the previous log distribution task; If the execution time of the current log distribution task is greater than the execution time of the previous log distribution task, compare the selector configuration information corresponding to the previous log distribution task with the selector configuration information corresponding to the current log distribution task to determine the changed selector configuration information; The configuration information of the selector corresponding to the selector type is configured according to the changed selector configuration information.

6. The selector determination method according to claim 1, characterized in that: The host resource information includes at least one of hard disk resource information, memory resource information, central processing unit resource information, network card resource information and bandwidth resource information, and the resource information of the event temporary buffer area includes at least one of task execution time, task occupied host resources, event temporary buffer area and communication rate between hosts.

7. The selector determination method according to claim 1, characterized in that: Predict selector information according to the resource information and the data type, including: According to the resource information and the data type, a linear regression model is used to predict the task volume and task type of the next log distribution task; The selector information is determined according to the predicted task volume and task type.

8. A selector determination device, characterized in that: include: The input data acquisition module is used to obtain the resource information corresponding to the current log distribution task and the corresponding data type; wherein the resource information includes the resource information of the event temporary buffer area and the host resource information; A prediction module, used to predict selector information according to the resource information and the data type; wherein the selector information includes a selector type and selector configuration information; A selector determination module is used to determine the selector corresponding to the next log distribution task according to the selector type and the selector configuration information.

9. A selector determination device, characterized in that include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the selector determination method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the selector determination method according to any one of claims 1 to 7 are implemented.

Images