Parameter dynamic configuration management method and device, equipment and medium

Abstract

The application provides a parameter dynamic configuration management method and device, equipment and medium. The method comprises the following steps: obtaining a running index corresponding to middleware in a preset unit time; analyzing the running index to determine whether the initial configuration of the consumption parameter of the middleware is dynamically adjusted; if the message accumulation quantity corresponding to the preset running index is greater than the maximum message accumulation quantity, the maximum consumption data quantity is increased for dynamic adjustment until the current running index is the optimal configuration; or / and, if the consumption data execution time in the preset running index is close to the maximum interval time of twice consumption, the maximum consumption data quantity is reduced for dynamic adjustment until the current running index is the optimal configuration. Through real-time analysis of the running index, the software can run in the optimal configuration, data accumulation is avoided, the consumption capacity of the middleware is improved, and the software has the advantages of high availability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software development, and particularly relates to a parameter dynamic configuration management method and device, equipment and a medium. BACKGROUND

[0002] With the vigorous development of Internet technology, the business logic of a program is very complex, so a plurality of software middleware technologies need to be referenced to improve software performance, such as kafka, redis, Elasticesearch, etc. These middleware all have a large number of parameter configurations, and how to reasonably configure the related parameters is particularly important for maximizing the advantages of the middleware and improving the software performance to the optimal. At the present stage of development, these parameters are mostly set by the experience of the developers, and cannot completely adapt to the complex and variable requirements of the program. For example, in the initial stage of project running, the software uses few users and the data volume is small, and the current settings can meet the requirements of the software, but when running for a period of time, the data volume becomes larger and larger, and the current configuration cannot meet the requirements.

[0003] However, in the traditional mode, the requirements can be met only by manually modifying the configuration and then re-releasing online. On the one hand, this increases the workload of the users and also makes the software operation configuration more cumbersome and complex. On the other hand, due to the differences in the technical level of the users, the modified configuration may not meet the requirements, and cannot meet the current configuration requirements of the software. On the other hand, the manual modification of the configuration by the users leads to poor flexibility and poor intelligence of the software.

[0004] CONTENT

[0005] In view of the above-mentioned shortcomings of the prior art, the present application provides a parameter dynamic configuration management method, device, equipment and medium to solve the technical problem that the above-mentioned parameter dynamic configuration management method cannot dynamically adapt to the requirements of the software and adaptively perform parameter dynamic configuration management.

[0006] In a first aspect, the present application provides a parameter dynamic configuration management method, which comprises:

[0007] acquiring a running index corresponding to the middleware in a preset unit of time;

[0008] analyzing the running index to determine whether to dynamically adjust the initial configuration of the consumption parameter of the middleware; the consumption parameter at least includes the maximum consumption data each time, the maximum message accumulation quantity and the maximum interval time between two consumptions;

[0009] if the message accumulation quantity corresponding to the preset running index is greater than the maximum message accumulation quantity, the maximum consumption data quantity each time is increased for dynamic adjustment until the current running index is optimally configured; or / and,

[0010] If the preset running index consumption data execution time is close to the two maximum consumption interval times, the maximum consumption data amount each time is reduced for dynamic adjustment until the current running index is the optimal configuration.

[0011] In a possible implementation, before the running index corresponding to the preset unit time, the acquisition middleware further includes: collecting running indexes, configuration parameters, and server data generated by at least one application program calling the middleware, wherein the running indexes include current consumption data of the middleware, a message accumulation quantity, and a consumption processing time length; the configuration parameters include a maximum consumption data amount each time, a maximum interval time between two consumptions, and a topic partition number; and the server data includes hardware indexes and cluster specifications of the middleware server.

[0012] In a possible implementation, the increasing the maximum consumption data amount each time for dynamic adjustment further includes:

[0013] According to the consumption data amount corresponding to each running record in the running index and the consumption time corresponding to each running record, an average consumption processing time length of each running record is determined.

[0014] According to the current maximum consumption data amount each time, the average consumption processing time length, and a time threshold value, a first adjustment amount each time of the maximum consumption data amount each time is determined, and the time threshold value is determined by a difference between the maximum interval time between two consumptions and a preset proportion of the maximum interval time between two consumptions.

[0015] According to the maximum message accumulation quantity, a current message accumulation quantity, and a running record number in a preset unit time, a second adjustment amount each time of the maximum consumption data amount each time is determined.

[0016] The value range corresponding to each of the first adjustment amount each time and the second adjustment amount each time is fused to determine an adjustment range of an adjustment amount of the maximum consumption data amount each time.

[0017] A maximum value and a minimum value of the adjustment range of the adjustment amount are mean calculated to determine a first adjustment amount approximate value of the maximum consumption data amount each time for dynamic adjustment.

[0018] In a possible implementation, the reducing the maximum consumption data amount each time for dynamic adjustment further includes:

[0019] According to the consumption data amount corresponding to each running record in the running index and the consumption time corresponding to each running record, an average consumption processing time length of each running record is determined.

[0020] determining a first adjustment quantity of the maximum consumption data quantity per time according to the current maximum consumption data quantity per time, the average consumption processing time and a time threshold value, the time threshold value being determined by a difference between the maximum interval time of two consumptions and a preset proportion of the maximum interval time of two consumptions;

[0021] determining the number of consumptions per unit time based on a correlation between a difference between the current maximum consumption data quantity per time and a third adjustment quantity and the average consumption processing time;

[0022] determining a third adjustment quantity of the maximum consumption data quantity per time according to a correlation between the number of consumptions per unit time, the running record number, the consumption data quantity and the current maximum consumption data quantity per time;

[0023] fusing the value range corresponding to the first adjustment quantity and the third adjustment quantity respectively to determine an adjustment range of the adjustment quantity of the maximum consumption data quantity per time;

[0024] performing mean value calculation on the maximum value and the minimum value of the adjustment range of the adjustment quantity to determine a second adjustment quantity approximation of the dynamic adjustment of the maximum consumption data quantity per time.

[0025] In a possible implementation, if the following conditions are met simultaneously: the message accumulation quantity corresponding to the preset running index is greater than the maximum message accumulation quantity, and the execution time of the consumption data in the preset running index is close to the maximum interval time of two consumptions, the method further includes:

[0026] expanding the number of topic partitions corresponding to the middleware configuration parameter, increasing the number of topic partitions and the number of consumption threads to increase the number of consumptions per unit time and thus improve the consumption quantity, until the expanded running index meets the preset index, and then stopping expanding the number of topic partitions, wherein the preset index includes real-time analysis of the expanded running index, and the expanded running index has no change relative to the running index before expansion.

[0027] In a possible implementation, if the running index is still in a high-risk state after being optimized by dynamic adjustment, an alarm signal is generated to remind a user to perform manual processing.

[0028] In a possible implementation, the current running index is optimal configuration, including: classifying the middleware server data based on the cluster specification, determining the same kind of middleware server data; analyzing the running index involved in the same kind of middleware server data to determine the execution standard of the amount of consumed data in unit time, the number of message accumulation without accumulation, and the consumption processing time being lower than the risk threshold, to obtain optimal configuration data; and dynamically analyzing the optimal configuration data and the initial configuration consumption parameter to determine that the current running index is optimal configuration.

[0029] In a second aspect, the present application further provides a parameter dynamic configuration management device, the device comprising:

[0030] An acquisition module is configured to acquire a running index corresponding to middleware in a preset unit of time.

[0031] An index analysis module is configured to analyze the running index and determine whether to dynamically adjust the initial configuration consumption parameter of the middleware; the consumption parameter at least includes maximum consumption data, maximum message accumulation quantity, and maximum interval time between two consumptions.

[0032] A first configuration management module is configured to increase the maximum consumption data for dynamic adjustment if the message accumulation quantity in the preset running index is greater than the maximum message accumulation quantity, until the current running index is optimal configuration; or / and,

[0033] A second configuration management module is configured to reduce the maximum consumption data for dynamic adjustment if the consumption data execution time in the preset running index is close to the maximum interval time between two consumptions, until the current running index is optimal configuration.

[0034] In a third aspect, the present application further provides an electronic device comprising a processor, a memory, and a communication bus.

[0035] The communication bus is configured to connect the processor and the memory.

[0036] The processor is configured to execute a computer program stored in the memory to implement the parameter dynamic configuration management method according to any one of the above embodiments.

[0037] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, the computer program being configured to cause a computer to execute the parameter dynamic configuration management method according to any one of the above embodiments.

[0038] The beneficial effects of the present application: the parameter dynamic configuration management method, device, equipment and medium provided by the present application, the method obtains the running index corresponding to the middleware in the preset unit time; the running index is analyzed to determine whether the initial configuration consumption parameter of the middleware is dynamically adjusted; since the parameter configuration of the middleware can be dynamically adjusted in software development, the software can meet the needs of users and data flow in various aspects in real time, through real-time analysis of the running index, not only ensures that the software runs with the optimal configuration, avoids data accumulation, but also improves the consumption capacity of the middleware, and makes the software have the advantage of high availability. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a parameter dynamic configuration management method flowchart provided in an embodiment of the present application;

[0040] Figure 2 is another flowchart of the parameter dynamic configuration management method provided in an embodiment of the present application;

[0041] Figure 3 is another flowchart of the parameter dynamic configuration management method provided in an embodiment of the present application;

[0042] Figure 4 is a flowchart of the access control system using kafka to consume access control events provided in an embodiment of the present application;

[0043] Figure 5 is a kafka middleware parameter dynamic configuration flowchart provided in an embodiment of the present application;

[0044] Figure 6 is a parameter dynamic configuration management system framework diagram provided in an embodiment of the present application;

[0045] Figure 7 is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The embodiments of the present application will be described below through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure of the present specification. The present application can also be implemented or applied through other different specific embodiments, and the details in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0047] It is to be noted that the diagrams provided in the following embodiments merely schematically illustrate the basic concepts of the present application, and only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape and size of the components in actual implementation, and the shapes, number and ratio of the components in actual implementation can be arbitrarily changed, and the component layout pattern can be more complex.

[0048] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application, however, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details, and in other embodiments, the known structures and devices are shown in the form of block diagrams rather than in the form of details, to avoid making the embodiments of the present application difficult to understand.

[0049] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0050] In the related art, with the vigorous development of Internet technology, the business logic of the program is very complex, so it is necessary to refer to various software middleware technologies to improve the software performance, such as kafka, redis, Elasticesearch, etc. These middleware all have a large number of parameter configurations, and it is particularly important to maximize the advantages of the middleware and optimize the software performance. However, in the current development, these parameters are mostly set by the experience of the developers, and cannot completely adapt to the complex and variable requirements of the program. For example, at the initial stage of project operation, the software uses few users and the data volume is small, and the current settings can meet the requirements of the software. However, after a period of operation, the data volume becomes larger and larger, and the current configuration cannot meet the requirements. However, in the traditional mode, the configuration is modified manually, and the software can only be released online to meet the requirements.

[0051] In the above manner, manual parameter configuration not only increases the workload of the user, but also makes the software operation and configuration more cumbersome and complex. Due to the difference in technical level of the user, the modified configuration may not meet the requirements, and cannot meet the current software configuration requirements. On the other hand, manual configuration by the user leads to poor flexibility and poor intelligence of the software.

[0052] Therefore, there is an urgent need for a dynamic adaptive software requirement and a self-adaptive parameter dynamic configuration management scheme, which will be described in detail below.

[0053] Please refer to Figure 1 The parameter dynamic configuration management method flowchart provided in an embodiment of the present application is described in detail as follows:

[0054] In step S110, the running index corresponding to the middleware in a preset unit of time is obtained.

[0055] Wherein, the middleware is a kind of software between application system and system software, it uses the basic service (function) provided by system software, links the various parts or different applications of application system on network, can reach the purpose of resource sharing, function sharing.In addition, preset unit time can be set according to user demand, for example, 1 minute, 2 minutes etc.

[0056] It should be noted that, before the running index corresponding to the middleware in the preset unit time is obtained, at least one application program calling the middleware is further collected to generate running index, configuration parameter and server data.

[0057] The running index includes current consumption data, message accumulation quantity and consumption processing time length of the middleware;The configuration parameter includes maximum consumption data amount, maximum interval time between two consumptions and theme partition number;The server data includes hardware index and cluster specification of middleware server.

[0058] In step S120, the running index is analyzed to determine whether the initial configuration of the consumption parameter of the middleware is dynamically adjusted;The consumption parameter at least includes maximum consumption data, maximum message accumulation quantity and maximum interval time between two consumptions;

[0059] Specifically, after analyzing the running index, it is determined whether the initial configuration of the consumption parameter of the middleware is dynamically adjusted, wherein the two situations triggered by dynamic adjustment include the following steps S130 and S140.

[0060] In step S130, if the message accumulation quantity corresponding to the preset running index is greater than the maximum message accumulation quantity, the maximum consumption data amount is increased for dynamic adjustment until the current running index is optimally configured.

[0061] Or / and,

[0062] Specifically, if the message accumulation quantity corresponding to the preset running index is less than or equal to the maximum message accumulation quantity, it is not processed;In addition, the rule of dynamic adjustment corresponding to step S130 is different from the rule of dynamic adjustment corresponding to step S140, resulting in different modulation quantities calculated therein.

[0063] In step S140, if the consumption data execution time of the preset running index is close to the maximum interval time between two consumptions, the maximum consumption data amount is reduced for dynamic adjustment until the current running index is optimally configured.

[0064] Specifically, if the preset running index execution consumption data execution time is not close to the two maximum consumption interval times, that is, the time difference between the two is large, it is not processed.

[0065] In the above two cases, dynamic adjustment of the maximum consumption data amount each time is required to meet any one of the cases, and then the current running index is adjusted to the optimal configuration. In this embodiment, the running index corresponding to the middleware in a preset unit time is obtained. The running index is analyzed to determine whether the initial configuration of the consumption parameter of the middleware is dynamically adjusted. Since the parameter configuration of the middleware can be dynamically adjusted in software development, the real-time needs of the software in terms of users and data traffic can be met. Through real-time analysis of the running index, not only is the software ensured to run in the optimal configuration, avoiding data accumulation, but also the consumption capacity of the middleware is improved, and the software has the advantage of high availability.

[0066] Referring to Figure 2 Another flowchart of the parameter dynamic configuration management method provided in an embodiment of the present application is provided. The increase in the dynamic adjustment of the maximum consumption data amount each time further includes:

[0067] In step S210, the average consumption processing time length of each running record is determined according to the consumption data amount corresponding to each running record in the running index and the consumption time corresponding to each running record.

[0068] For example, the sum of the consumption data amounts corresponding to each running record in the running index is counted, the sum of the consumption times corresponding to each running record is counted, the ratio of the sum of the consumption times to the sum of the consumption data amounts is calculated, and the average consumption processing time length of each running record is determined.

[0069] In step S220, the first adjustment amount each time of the maximum consumption data amount each time is determined according to the current maximum consumption data amount each time, the average consumption processing time length, and a time threshold value, and the time threshold value is determined by the difference between the two maximum consumption interval times and the two maximum consumption interval times of the preset proportion.

[0070] For example, the time obtained by multiplying the sum of the current maximum consumption data amount each time and the first adjustment amount each time by the average consumption processing time length is less than the time threshold value, and the first inequality relationship is used to determine the range of the first adjustment amount each time.

[0071] In step S230, the second adjustment amount each time of the maximum consumption data amount each time is determined according to the maximum message accumulation amount, the current message accumulation amount, and the number of running records in a preset unit time.

[0072] For example, according to the product of the second adjustment quantity per time and the running record number in the preset unit time, greater than or equal to the difference between the current message accumulation quantity and the maximum message accumulation quantity, a second inequality relationship is formed to determine the second adjustment quantity per time of the maximum consumption data quantity per time.

[0073] In step S240, the value ranges corresponding to the first adjustment quantity per time and the second adjustment quantity per time are fused to determine the adjustment range of the adjustment quantity of the maximum consumption data quantity per time.

[0074] For example, the two inequalities are combined to calculate, and since the value ranges of the first adjustment quantity per time and the second adjustment quantity per time are different, the intersection is calculated by cross-fusion to determine the adjustment range of the adjustment quantity of the maximum consumption data quantity per time.

[0075] In step S250, the maximum value and the minimum value of the adjustment range of the adjustment quantity are calculated to determine the first adjustment quantity approximation of the maximum consumption data quantity per time.

[0076] For example, the first adjustment quantity approximation of the maximum consumption data quantity per time is determined by the mean value calculation of the adjustment range of the adjustment quantity of the maximum consumption data quantity per time.

[0077] In the above manner, the current maximum consumption quantity per time + the first adjustment quantity approximation can be quickly obtained. After adjustment according to the above manner, the running index under the current configuration is continuously analyzed and dynamically adjusted to ensure that the current running index is the optimal configuration, and thus the optimal configuration of the running index is obtained.

[0078] Please refer to Figure 3 Another flowchart of the parameter dynamic configuration management method provided in an embodiment of the present application is provided, and the reduction of the maximum consumption data quantity per time dynamically adjusted further includes:

[0079] In step S310, according to the consumption data quantity corresponding to each running record in the running index and the consumption time corresponding to each running record, the average consumption processing time length of each running record is determined.

[0080] For example, the sum of the consumption data quantities corresponding to each running record in the running index is counted, the sum of the consumption time corresponding to each running record is counted, the ratio of the sum of the consumption time to the sum of the consumption data quantity is calculated, and the average consumption processing time length of each running record is determined.

[0081] Step S320, determining a first adjustment quantity of the maximum consumption data quantity according to the current maximum consumption data quantity, the average consumption processing time and a time threshold, the time threshold being determined by a difference between the maximum interval time of two consumptions and a preset proportion of the maximum interval time of two consumptions;

[0082] For example, the range of the first adjustment quantity is determined by the first inequality relationship that a time obtained by multiplying the sum of the current maximum consumption data quantity and the first adjustment quantity by the average consumption processing time is less than the time threshold.

[0083] The preset proportion can be 10%, so that the time threshold is determined by a difference between the maximum interval time of two consumptions and 10% of the maximum interval time of two consumptions.

[0084] Step S330, determining the number of consumptions in a unit time based on a correlation between a difference between the current maximum consumption data quantity and a third adjustment quantity and the average consumption processing time;

[0085] For example, the number of consumptions in a unit time is equal to a quotient obtained by dividing a unit time by a product between the difference between the current maximum consumption data quantity and the third adjustment quantity and the average consumption processing time, so that the number of consumptions in a unit time expressed by the third adjustment quantity is determined by the equation.

[0086] Step S340, determining the third adjustment quantity of the maximum consumption data quantity according to a correlation between the number of consumptions in a unit time, the number of running records, the consumption data quantity and the current maximum consumption data quantity;

[0087] For example, the number of consumptions in a unit time expressed by the third adjustment quantity is substituted into the following third inequality:

[0088] A product between the difference between the current maximum consumption data quantity and the third adjustment quantity and the number of consumptions in a unit time is greater than or equal to a sum of the consumption data quantities corresponding to each running record in the running index.

[0089] The range of the third adjustment quantity of the maximum consumption data quantity is determined by the inequality.

[0090] Step S350, fusing the value range corresponding to the first adjustment quantity and the third adjustment quantity respectively to determine an adjustment range of the adjustment quantity of the maximum consumption data quantity;

[0091] For example, the first inequality and the third inequality are combined to calculate, and because the first adjustment amount and the third adjustment amount have different ranges, the intersection is calculated by cross-fusion to determine the adjustment range of the adjustment amount of the maximum consumption data amount each time.

[0092] In step S360, the maximum value and the minimum value of the adjustment range of the adjustment amount each time are calculated to determine the second adjustment amount approximation of the dynamic adjustment of the maximum consumption data amount each time.

[0093] For example, the second adjustment amount approximation of the dynamic adjustment of the maximum consumption data amount each time is determined by the mean calculation of the adjustment range of the adjustment amount of the maximum consumption data amount each time.

[0094] In the above manner, the second adjustment amount approximation of the maximum consumption amount each time can be quickly obtained. After adjustment according to the above manner, the running index under the current configuration is continuously analyzed and dynamically adjusted to ensure that the current running index is the optimal configuration, and thus the running index under the current configuration is the optimal configuration.

[0095] In some other embodiments, if the following conditions are met simultaneously: the message accumulation amount corresponding to the preset running index is greater than the maximum message accumulation amount, and the execution time of the consumption data in the preset running index is close to twice the maximum consumption interval time, the method further includes:

[0096] The number of topic partitions corresponding to the middleware configuration parameter is expanded, the number of topic partitions and the number of consumption threads are increased to increase the number of consumptions per unit time and thus increase the consumption amount, until the running index after expansion meets the preset index, and then the expansion of the number of topic partitions is stopped, wherein the preset index includes real-time analysis of the running index after expansion, and the running index after expansion has no change relative to the running index before expansion (i.e., the index after expansion has no obvious improvement).

[0097] In the above manner, the number of topic partitions corresponding to the middleware configuration parameter is expanded when the above two trigger conditions are met simultaneously, the number of topic partitions and the number of consumption threads are increased to increase the number of consumptions per unit time and thus increase the consumption amount, until the running index after expansion meets the preset index, and then the expansion of the number of topic partitions is stopped.

[0098] In some other embodiments, if the running index is still in a high-risk state after being optimized by dynamic adjustment, an alarm signal is generated to remind the user to perform manual processing.

[0099] In the above manner, if the operation index is in a high-risk state, it indicates that the dynamic adjustment of parameters cannot handle the software requirements, and the problem needs to be solved from the hardware level. Therefore, the background alarm system is triggered to send an alarm information, and manual intervention is performed to minimize the impact on the business.

[0100] The following describes the middleware using kafka, as follows:

[0101] I. Dynamic setting of kafka parameters

[0102] Initial setting: Set the initial value of the kafka consumption parameter, and the main parameters include the maximum consumption data amount (max-poll-records) each time, the maximum interval time (max.poll.interval.ms) between two consumptions, the number of topic partitions, and the maximum allowed kafka accumulation amount.

[0103] Dynamic adjustment: During software operation, record the operation index of kafka consumption data each time, analyze the operation index of the software, and determine whether the kafka consumption parameter needs to be adjusted.

[0104] 1. Operation index: kafka current consumption amount, kafka message accumulation amount, and kafka one-time consumption processing time.

[0105] 2. Dynamic adjustment trigger condition: Analyze all operation indexes within a unit time (default 1 minute), and if the following two situations exist, the parameter needs to be adjusted:

[0106] (1) There are 2 / 3 of the operation records with a message accumulation amount greater than the maximum allowed accumulation amount.

[0107] (2) There are 2 / 3 of the operation indexes with a consumption data execution time close to the maximum interval time (and the maximum time difference is less than 10%, indicating that the consumer is at risk of being kicked out).

[0108] The trigger condition can use a rule engine to configure rules to ensure the optimal triggering rules.

[0109] Among them, the operation index data, configuration data, and kafka server data when each software uses kafka are collected:

[0110] Operation index: kafka current consumption amount, kafka message accumulation amount, and kafka one-time consumption processing time.

[0111] Kafka configuration parameters: maximum consumption data amount (max-poll-records) each time, maximum interval time (max.poll.interval.ms) between two consumptions, and number of topic partitions.

[0112] kafka server data: the hardware specification, cluster specification of the service.

[0113] According to the specification, different kafka server data are classified, and in the case of ensuring that the kafka server data is of the same class, the optimal configuration data is obtained by analyzing the running index data, with the standards of high data consumption per unit time, no data accumulation, and processing time lower than the risk value. The result is stored and updated according to the analysis result to ensure that the recommended value is the optimal configuration.

[0114] When software developers set the initial value of the middleware parameter, they can obtain the recommended initial configuration according to the software user quantity, data quantity, kafka server specification, and storage result of big data analysis comparison, to ensure that the software is in the optimal state from the beginning of running.

[0115] 3. Adjustment rule:

[0116] (1) If condition 1 for triggering dynamic adjustment is met, only the maximum consumption data quantity per time needs to be adjusted, and the approximate adjustment quantity can be obtained according to the following formula

[0117] ET = (running record 1 consumption data quantity + running record 2 consumption data quantity + … + running record n consumption data quantity) / (running record 1 consumption time + running record 2 consumption time + … + running record n consumption time), to obtain the processing time of each data.

[0118] (Current maximum consumption quantity per time + adjustment quantity per time) * ET < maximum interval time - maximum interval time * 10%, to ensure that the processing time of one consumption is less than the time critical value.

[0119] Adjustment quantity per time * running record number per unit time >= current accumulation quantity - maximum allowed accumulation quantity, to ensure that the adjusted accumulation quantity is not greater than the maximum allowed accumulation quantity

[0120] The above three steps can obtain the adjustment quantity range, and the average of the maximum and minimum values of the adjustment quantity range is obtained to obtain the approximate adjustment quantity.

[0121] The maximum consumption quantity is dynamically set to the current maximum consumption quantity per time + adjustment quantity. After adjustment, the running index under the current configuration is continuously analyzed and dynamically adjusted to ensure that the current running index is the optimal configuration.

[0122] (2) If condition 2 for triggering dynamic adjustment is met, there is a risk of rebalancing of kafka, and the maximum consumption data quantity per time needs to be reduced, and the approximate adjustment quantity can be obtained according to the following formula

[0123] (current maximum number of consumption per time - adjustment number per time) * ET < maximum interval time - maximum interval time * 10%, to ensure that the processing time of a consumption is less than the time threshold.

[0124] EC = unit time / ((current maximum number of consumption per time - adjustment number per time) * ET), the number of consumptions per unit time after adjustment.

[0125] (current maximum number of consumption per time - adjustment number per time) * EC >= running record number 1 consumption data volume + running record number 2 consumption data volume + … + running record number n consumption data volume, to ensure that no data accumulation occurs.

[0126] The above three steps can obtain the adjustment number range, and the average of the maximum and minimum values of the adjustment number range is obtained to obtain the approximate adjustment number.

[0127] Dynamically set the maximum consumption number to the current maximum consumption number per time - adjustment number. After adjustment, the running indicators under the current configuration are continuously analyzed and dynamically adjusted to ensure that the current running indicators are optimally configured.

[0128] (3) At the same time, conditions 1 and 2 are met, at this time, the number of topic partitions can be dynamically modified, the number of consumption threads is increased, the number of consumptions per unit time is increased, and the consumption number is increased. The partition expansion step is 1, and the maximum number of kafka service partitions is reached. In the case of allowing concurrent consumption, the number of concurrent threads is increased, and the step is also 1. After expansion, the running indicators are continuously analyzed, and if the indicators have no obvious improvement after expansion, the partition expansion is stopped.

[0129] Alarm: After the above dynamic adjustment, if the running indicators are in a high-risk state, it means that the software requirements cannot be handled through dynamic adjustment of parameters, and the problem needs to be solved from the hardware level. Then trigger the background alarm system to issue an alarm information, and manually intervene to minimize the impact on business.

[0130] In the present embodiment, in the current software development, at the initial stage of project operation, the software traffic is small, the data volume collected by kafka is not large, and the real-time requirement of data is not high. The initial configuration can fully meet the requirements. Even if the consumption is slow, it will not affect the normal business. As long as the final consumption is completed, it is OK. With the increase of users and data traffic, the consumption requirement of kafka is higher. The initial configuration cannot meet the current software requirements. If the related parameters of kafka cannot be dynamically adjusted, not only will a large amount of data be accumulated, but also the consumption capacity will be insufficient and be rejected by kafka. This will affect the normal business and cannot meet the high availability of software. In the traditional mode, only when the problem occurs, it is found that the software is unavailable. It cannot achieve instant alarm. In addition, the processing method of the problem is to manually intervene to modify the related parameters and then release again. The operation cycle is long and cannot respond instantly to ensure the high performance and high availability of software.

[0131] After introducing dynamic configuration, the technical effects are as follows:

[0132] (1) Through self-learning and decision recommendation of the system, the parameter value can be accurately obtained according to the software requirements, the initial parameters of kafka are accurately set, overestimation or underestimation of the parameter value is avoided, and the software is ensured to run with optimal performance after online.

[0133] (2) In the process of software running, the running index of software is monitored in real time, the running index is analyzed instantly, the related parameters are dynamically changed according to the analysis result, the consumption capacity of kafka is improved, and the software is ensured to run with optimal performance and high availability.

[0134] (3) When dynamically configuring, no human intervention is needed. The system adjusts the parameters through self-analysis, thereby improving the flexibility and adaptability of software.

[0135] (4) Instant alarm can be achieved. After dynamically optimizing the parameters, if the running index is still in an unhealthy state, the alarm is triggered at this time, human intervention is notified for processing, and the high availability of service is ensured.

[0136] Please refer to Figure 4 , the access control system provided in an embodiment of the present application uses a kafka consumption access control event flow chart; the details are as follows:

[0137] By comparing the predicted generated access control event QPS, kafka server parameters, single data processing time consumption and the running data storage result of big data analysis, the recommended configuration of kafka parameters of the system is obtained. The maximum number of pulling each time (max.poll.records) of the consumer kafka, the number of concurrent, the pulling interval (max.poll.interval.ms) and the like are set. The operation can ensure that the parameter setting meets the current requirements and runs in a good state.

[0138] Under the current settings, record the running index of each time the access control event processing data is consumed, and analyze the current number of access control events, the number of access control event message accumulation, the processing time of one access control event consumption, etc. in the background. Through dynamic adjustment, the problems caused by the change in the number of access control events can be solved. There are several situations that need to be dynamically adjusted.

[0139] (1) Adding access control devices, increasing the frequency of access control points, etc. will increase the number of access control events, so the ability to process access control events needs to be improved. At this time, data accumulation will occur, and the system will automatically trigger parameter dynamic adjustment. After dynamic adjustment according to the dynamic adjustment rules, most of these requirements can be met to ensure normal processing of access control events.

[0140] (2) If two of the five servers in the server cluster that processes access control events are down, the data processing capacity will decrease. At this time, data accumulation may occur, and the system will automatically trigger parameter dynamic adjustment. After dynamic adjustment according to the dynamic adjustment rules, most of these requirements can be met to ensure normal processing of access control events.

[0141] (3) Due to network or database reasons, the access control event storage becomes slow, so the processing time of access control events becomes longer, which may cause the consumer to be kicked out and kafka to send rebalance. The system will automatically trigger parameter dynamic adjustment. After dynamic adjustment according to the dynamic adjustment rules, most of these requirements can be met to ensure normal processing of access control events.

[0142] If the dynamic adjustment of the kafka related parameters still causes the accumulation of access control events and reaches the threshold or causes the kafka consumer to rebalance, an alarm message will be sent, and manual intervention will be performed.

[0143] For example, the access control device generates an access control event, kafka receives the access control event data, consumes the access control event data, records and analyzes the kafka execution, judges whether it is abnormal risk, if so, adjusts the parameters; if not, determines whether it meets the specifications; if so, re-record and analyze the kafka execution; if not, adjust the parameters; after parameter adjustment, determine whether the program is in the optimal state after multiple adjustments, if so, re-record and analyze the kafka execution; if not, generate an alarm (manual intervention), and re-record and analyze the kafka execution.

[0144] Please refer to Figure 5 The kafka middleware parameter dynamic configuration process provided in an embodiment of the present application also includes:

[0145] The application program inputs system data volume, learns and decides the kafka server parameters, and the kafka intermediate running record collection collects running data and configuration data to send learning decisions to analyze system running indexes, obtain optimal data configuration, and store the same, and the learning decision output is used to obtain the kafka middleware parameter recommended configuration.

[0146] During software running, the kafka running indexes are obtained by the application program and continuously analyzed, the mobilization rules are triggered and the parameters are dynamically adjusted, and if the kafka messages are always accumulated and the kafka is abnormal, the alarm information is sent back to the application program.

[0147] A large number of kafka middleware running indexes, kafka configuration parameters and kafka server data are collected, these data are analyzed and decided, and the optimal kafka parameter configuration under different dimensions is obtained and stored.

[0148] The application program kafka parameter initial value is set, and the kafka parameter recommended value is obtained through the application program data volume, kafka server specification and single business data processing time.

[0149] During software running, the running indexes of each kafka consumption data are recorded, the data are analyzed, the system automatically determines whether to trigger the automatic adjustment rules according to the trigger conditions, the required parameter dynamic setting is completed, and the system normal running is ensured. The operation is a continuous process, the system continuously analyzes the running indexes, and adjusts the parameters in time to meet the software needs.

[0150] During the analysis of the running indexes, the kafka still has message accumulation or program abnormality through dynamic parameter adjustment, and instant alarm is given to ensure manual intervention to solve the problem in the shortest time.

[0151] Please refer to Figure 6 The embodiment provides a parameter dynamic configuration management device, which comprises:

[0152] An acquisition module 601 is configured to acquire running indexes corresponding to a middleware in a preset unit time.

[0153] An index analysis module 602 is configured to analyze the running indexes and determine whether to dynamically adjust the initial configuration of the consumption parameters of the middleware; the consumption parameters at least include maximum consumption data, maximum message accumulation quantity and maximum interval time between two consumptions.

[0154] A first configuration management module 603 is configured to increase the maximum consumption data for dynamic adjustment until the current running indexes are optimal configuration if the message accumulation quantity corresponding to the preset running indexes is greater than the maximum message accumulation quantity; or / and,

[0155] The second configuration management module 604 is configured to reduce the maximum consumption data amount each time for dynamic adjustment until the running index is the optimal configuration if the preset running index execution consumption data execution time approaches the twice consumption maximum interval time.

[0156] In the embodiment, the device is essentially provided with a plurality of modules to execute the method in the above embodiment. The specific functions and technical effects can refer to the above method embodiment, and will not be described here.

[0157] Referring to Figure 7 The embodiment of the present application further provides an electronic device 700, which comprises a processor 701, a memory 702 and a communication bus 703.

[0158] The communication bus 703 is used to connect the processor 701 and the memory 702.

[0159] The processor 701 is used to execute the computer program stored in the memory 702 to realize the method of one or more of the above embodiments.

[0160] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is used to make a computer execute the method of any one of the above embodiments.

[0161] The embodiment of the present application further provides a non-volatile readable storage medium, which stores one or more modules. When the one or more modules are applied to a device, the device can execute the instructions of the steps contained in the embodiment one of the embodiment of the present application.

[0162] Note that the computer readable medium described above can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer readable signal medium can include a computer readable program code propagated on or through a computer readable medium, in baseband or as part of a carrier wave. The computer readable signal medium can take a variety of forms, including but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0163] The computer readable medium described above can be included in the electronic device described above; alternatively, the computer readable medium can exist as a separate entity in which the electronic device is incorporated.

[0164] Computer program code for carrying out operations of the disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0166] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for managing dynamic configuration of parameters, characterized in that, The method comprises: acquiring a running index corresponding to the middleware in a preset unit time; analyzing the running index to determine whether to dynamically adjust a consumption parameter of the initial configuration of the middleware; the consumption parameter at least includes maximum consumption data per time, maximum message accumulation quantity, and maximum interval time between two consumptions; if the message accumulation quantity corresponding to the preset running index is greater than the maximum message accumulation quantity, increasing the maximum consumption data per time for dynamic adjustment until the current running index is an optimal configuration; wherein, according to the consumption data quantity corresponding to each running record in the running index and the consumption time corresponding to each running record, determining the average consumption processing time length of each running record; according to the current maximum consumption data per time, the average consumption processing time length, and a time threshold value, determining a first adjustment quantity per time of the maximum consumption data per time, the time threshold value being determined by the difference between the maximum interval time between two consumptions and the maximum interval time between two consumptions multiplied by a preset ratio; according to the maximum message accumulation quantity, the current message accumulation quantity, and the number of running records in a preset unit time, determining a second adjustment quantity per time of the maximum consumption data per time; fusing the value range corresponding to the first adjustment quantity per time and the second adjustment quantity per time respectively to determine the adjustment range of the adjustment quantity of the maximum consumption data per time; performing mean value calculation on the maximum value and the minimum value of the adjustment range of the adjustment quantity to determine the first adjustment quantity approximation of the dynamic adjustment of the maximum consumption data per time; or / and, if the consumption data execution time of the preset running index is close to the maximum interval time between two consumptions, reducing the maximum consumption data per time for dynamic adjustment until the current running index is an optimal configuration; wherein, according to the consumption data quantity corresponding to each running record in the running index and the consumption time corresponding to each running record, determining the average consumption processing time length of each running record; according to the current maximum consumption data per time, the average consumption processing time length, and a time threshold value, determining a first adjustment quantity per time of the maximum consumption data per time, the time threshold value being determined by the difference between the maximum interval time between two consumptions and the maximum interval time between two consumptions multiplied by a preset ratio; based on the correlation between the difference between the current maximum consumption data per time and a third adjustment quantity per time and the average consumption processing time length, determining the number of consumptions per unit time; according to the correlation between the number of consumptions per unit time, the number of running records, the consumption data quantity, and the current maximum consumption data per time, determining a third adjustment quantity of the maximum consumption data per time; fusing the value range corresponding to the first adjustment quantity per time and the third adjustment quantity per time respectively to determine the adjustment range of the adjustment quantity of the maximum consumption data per time; performing mean value calculation on the maximum value and the minimum value of the adjustment range of the adjustment quantity to determine the second adjustment quantity approximation of the dynamic adjustment of the maximum consumption data per time.

2. The method of claim 1, wherein, The acquisition middleware further comprises collecting running indexes, configuration parameters and server data generated by the middleware called by at least one application program in a preset unit time, wherein the running indexes comprise current consumption data, message accumulation quantity and consumption processing time length of the middleware; the configuration parameters comprise maximum consumption data quantity each time, maximum interval time between two consumptions and topic partition quantity; and the server data comprises hardware indexes and cluster specifications of the middleware server.

3. The method of any one of claims 1 to 2, wherein, If the preset running indexes simultaneously satisfy the conditions that the message accumulation quantity corresponding to the preset running indexes is greater than the maximum message accumulation quantity and the execution time of the preset running indexes is close to the maximum interval time between two consumptions, the method further comprises: expanding the topic partition quantity corresponding to the configuration parameters of the middleware, increasing the topic partition quantity and the consumption thread quantity, so as to increase the consumption times in a unit time and improve the consumption quantity, until the running indexes after expansion satisfy the preset indexes, and then stopping expanding the topic partition quantity, wherein the preset indexes comprise real-time analysis of the running indexes after expansion, and the running indexes after expansion have no change relative to the running indexes before expansion.

4. The method of any one of claims 1 to 2, wherein, If the running indexes are still in a high-risk state after being optimized by dynamic adjustment, an alarm signal is generated to remind a user to perform manual processing.

5. The method of claim 2, wherein, If the running indexes are in an optimal configuration, the method comprises: classifying the server data of the middleware based on the cluster specifications, determining the same type of the server data of the middleware; analyzing the running indexes related to the same type of the server data of the middleware, and determining the optimal configuration data according to the execution standards that the consumption data quantity in a unit time is large, the message accumulation quantity has no accumulation and the consumption processing time length is lower than a risk threshold; and dynamically analyzing the optimal configuration data and the initial configuration consumption parameters to determine that the running indexes are in the optimal configuration.

6. A parameter dynamic configuration management apparatus, characterized by, The apparatus comprises: an acquisition module configured to acquire running indexes corresponding to middleware in a preset unit time; an index analysis module configured to analyze the running indexes to determine whether to dynamically adjust initial configuration consumption parameters of the middleware; and the consumption parameters at least comprise maximum consumption data each time, maximum message accumulation quantity and maximum interval time between two consumptions. The first configuration management module is configured to increase the maximum data consumption amount for dynamic adjustment if the corresponding message accumulation quantity in the preset operation index is greater than the maximum message accumulation quantity, until the current operation index is the optimal configuration; wherein, the average consumption processing time length of each operation record is determined according to the consumption data amount corresponding to each operation record in the operation index and the consumption time corresponding to each operation record; the first adjustment amount of the maximum data consumption amount is determined according to the current maximum data consumption amount, the average consumption processing time length and the time threshold value, and the time threshold value is determined by the difference between the two maximum consumption intervals and the preset proportion of the two maximum consumption intervals; the second adjustment amount of the maximum data consumption amount is determined according to the maximum message accumulation quantity, the current message accumulation quantity and the number of operation records in a preset unit time; the adjustment range of the adjustment amount of the maximum data consumption amount is determined by fusing the value range corresponding to the first adjustment amount and the second adjustment amount respectively; the first adjustment amount approximation value of the maximum data consumption amount for dynamic adjustment is determined by mean calculation of the maximum value and the minimum value of the adjustment range of the adjustment amount; or / and, The second configuration management module is configured to reduce the maximum data consumption amount for dynamic adjustment if the execution time of the consumption data in the preset operation index is close to the two maximum consumption intervals, until the current operation index is the optimal configuration, wherein, the average consumption processing time length of each operation record is determined according to the consumption data amount corresponding to each operation record in the operation index and the consumption time corresponding to each operation record; the first adjustment amount of the maximum data consumption amount is determined according to the current maximum data consumption amount, the average consumption processing time length and the time threshold value, and the time threshold value is determined by the difference between the two maximum consumption intervals and the preset proportion of the two maximum consumption intervals; the number of consumptions in a unit time is determined based on the correlation between the difference between the current maximum data consumption amount and the third adjustment amount and the average consumption processing time length; the third adjustment amount of the maximum data consumption amount is determined according to the correlation between the number of consumptions in a unit time, the number of operation records, the consumption data amount and the current maximum data consumption amount; the adjustment range of the adjustment amount of the maximum data consumption amount is determined by fusing the value range corresponding to the first adjustment amount and the third adjustment amount respectively; the second adjustment amount approximation value of the maximum data consumption amount for dynamic adjustment is determined by mean calculation of the maximum value and the minimum value of the adjustment range of the adjustment amount.

7. An electronic device, comprising: The processor, the memory and the communication bus are included; The communication bus is used to connect the processor and the memory; The processor is used to execute the computer program stored in the memory to realize the method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, a computer program for causing a computer to perform the method of any one of claims 1-5, stored on a computer-readable medium.

Images