Web cache setting optimization method and electronic device

Abstract

The present invention relates to the technical field of Web services, and in particular to a Web cache setting optimization method and an electronic device. The method comprises obtaining the access time interval of each API, and writing the access time interval into a first queue; using the access time interval in the first queue, respectively calculating the long-term trend value, the short-term trend value and the average value to determine the long-term trend array and the short-term trend array; performing variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine the trend difference array; calculating the current trend value based on the last value in the trend difference array and the previous trend value; obtaining the current performance data of the target server to determine the hotspot threshold; determining the target API according to the size relationship between the current trend value corresponding to each API and the hotspot threshold, determining the target API as hotspot data and caching the target API. Adaptive decision-making is used to determine whether the visited API belongs to hotspot data, thereby improving the hit rate of hotspot data.

Description

Technical Field

[0001] The present invention relates to the technical field of Web services, and in particular to a Web cache setting optimization method and an electronic device. Background Art

[0002] In order to support massive large-scale http access, there are many Web service optimization technologies that can be used, among which Web caching technology is one of the most important basic technologies. An excellent Web caching system can greatly reduce the load pressure of the back-end database. By caching hot data, there is no need to directly access the back-end database. While reducing the database load, the response speed of reading data from the cache system will be greatly accelerated, thereby shortening the network response time for users to access Web services and greatly improving the customer experience.

[0003] Currently, the commonly used web cache technology includes dynamic data cache. In order to solve the problem of insufficient database performance in high concurrency and big data scenarios, Nosql technology is generally used to implement cache functions. Nosql database is a database that stores data in memory. Its data persistence function is weaker than traditional databases and is not suitable for persistence, but its access speed is extremely fast, so it is naturally suitable for use as a cache system.

[0004] The existing Web cache technology solution is: relying on the experience of technical personnel, configure the URL path set that needs the cache function in the code. When the http access reaches the system: according to the API configuration query, if it is an API without a cache function, it will directly access the database and return the data; if it is an API with a cache function, it will access the corresponding cache key in the redis system and return the cached data; if the key does not exist, it will access the database and store the data in redis.

[0005] However, this solution will lead to inaccurate hotspot caching judgments: First, when deciding which APIs need to use the cache function, existing technologies often rely on manual experience to predict hotspot caching. In actual scenarios, HTTP traffic often changes frequently over time. A URL may be a frequently visited hotspot API during certain periods of the day, but not a hotspot during other periods. Therefore, whether to enable the cache function for a URL also requires real-time decision changes. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method and an electronic device for optimizing Web cache settings to solve the problem of inaccurate hotspot data judgment in existing Web cache systems.

[0007] According to a first aspect, an embodiment of the present invention provides a method for optimizing Web cache settings, including:

[0008] Obtain the access time interval of each API, and write the access time interval into the first queue;

[0009] Using the access time intervals in the first queue, respectively calculating the long-term trend value, the short-term trend value and the average value to determine a long-term trend array and a short-term trend array;

[0010] Perform variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine a trend difference array;

[0011] Calculating the current trend value based on the last value in the trend difference array and the previous trend value;

[0012] Obtain current performance data of the target server to determine the hotspot threshold;

[0013] The target API is determined according to the magnitude relationship between the current trend values ​​corresponding to the various APIs and the hotspot threshold, and the target API is determined as hotspot data and cached.

[0014] The Web cache setting optimization method provided by the embodiment of the present invention improves the hit rate of hot data by adaptively determining whether the accessed API belongs to hot data. The prior art can only select hot URLs manually, while the present invention can decide whether the current API should enter the hot range based on real-time access data. This function can greatly optimize the problem of Nosql system occupying a large amount of memory; traditional data uses disk to store data, and Nosql cannot copy all hot data into memory due to limited memory capacity.

[0015] Optionally, the current trend value is calculated using the following formula:

[0016] RLT N-1 =((M-1)*RLT N-2 +2*DIFF N-1 )÷(M+1)

[0017] Among them, RLT N-1 is the current trend value, RLT N-2 is the previous trend value, DIFF N-1 is the last value in the trend difference array, and M is the period constant.

[0018] Optionally, obtaining current performance data of the target server to determine the hotspot threshold includes:

[0019] Obtain the CPU usage percentage, memory usage percentage, and disk usage percentage of the target server;

[0020] A weighted calculation is performed based on the CPU usage percentage, the memory usage percentage, and the disk usage percentage to determine the hotspot threshold.

[0021] The Web cache setting optimization method provided by the embodiment of the present invention fully considers the impact of the real-time performance data of the target server on the cache business. Under different server load conditions, the hotspot threshold will change dynamically, thereby ensuring that the cache server runs with the highest possible performance while also ensuring system security.

[0022] Optionally, determining a target API according to a magnitude relationship between current trend values ​​corresponding to each API and the hotspot threshold, determining the target API as hotspot data and caching the target API includes:

[0023] Corresponding to each API, determining whether the current trend value corresponding to the API is less than the hotspot threshold;

[0024] When the current trend value corresponding to the api is less than the hotspot threshold, the api is determined to be the target api and the hotspot data and the target api is cached.

[0025] Optionally, the step of determining a target API according to the magnitude relationship between the current trend values ​​corresponding to the respective APIs and the hotspot threshold, determining the target API as hotspot data and caching the target API further includes:

[0026] When the current trend value corresponding to the API is greater than or equal to the hotspot threshold, the database corresponding to the API is accessed.

[0027] Optionally, before the step of obtaining the access time interval of each API, the step further includes:

[0028] Get access request;

[0029] Determining whether a corresponding interface keyword exists in the cache based on the access request;

[0030] When the interface keyword does not exist, the step of obtaining the access time interval of each API is executed.

[0031] Optionally, the method further comprises:

[0032] Obtaining the update time interval of each database, and writing the update time interval into the second queue;

[0033] Using the update time interval in the second queue, respectively calculate the updated long-term trend value, the updated short-term trend value and the updated average value to determine the updated long-term trend array and the updated short-term trend array;

[0034] Perform variance and calculation based on the updated long-term trend array, the updated short-term trend array and the updated average value to determine an updated trend difference array;

[0035] The current update trend value is calculated based on the last value in the update trend difference array and the previous update trend value;

[0036] The cache expiration time of the database is determined according to the current update trend value and the update average value.

[0037] The Web cache setting optimization method provided by the embodiment of the present invention uses an adaptive algorithm to calculate a scientific and reasonable cache expiration time according to the database update interval data. When the database update trend value continues to increase, it means that the database data changes slowly, and the adaptive algorithm will increase the cache expiration time. On the contrary, it means that the database changes faster and the cache expiration time will be reduced. Thereby optimizing the performance of the cache system and reasonably maximizing the cache effect.

[0038] Optionally, the cache expiration time is calculated using the following formula:

[0039]

[0040] Among them, RLT′ n-1 is the current update trend value, RATE is the update weight, n is the number of update time intervals in the second queue, db i is the i-th update time interval in the second queue, and RANDOM is a random value.

[0041] The Web cache setting optimization method provided by the embodiment of the present invention takes into account the problems of avoiding cache breakdown and cache avalanche, introduces random values ​​in the calculation process, improves system security, and is more suitable as a solution for a cache system in a high-concurrency scenario.

[0042] According to a second aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the Web cache setting optimization method described in the first aspect or any one of the embodiments of the first aspect by executing the computer instructions.

[0043] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable the computer to execute the Web cache setting optimization method described in the first aspect or any one of the implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 The system architecture diagram of the cache system in the present invention is shown;

[0046] Figure 2 The internal functional module diagram of the monitoring system used in the present invention is shown;

[0047] Figure 3 The internal functional module diagram of the log system used in the present invention is shown;

[0048] Figure 4 is a flow chart of a method for optimizing Web cache settings according to an embodiment of the present invention;

[0049] Figure 5 is a flow chart of a method for optimizing Web cache settings according to an embodiment of the present invention;

[0050] Figure 6 is a flow chart of a method for optimizing Web cache settings according to an embodiment of the present invention;

[0051] Figure 7 is a flow chart of a method for optimizing Web cache settings according to an embodiment of the present invention;

[0052] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0054] Figure 1 It is a system architecture embodiment of the cache system described in the present invention. The entire system framework includes a web backend service, a cache system, a database, a monitoring system and a log service system.

[0055] The cache system uses redis as an embodiment in the present invention. Redis data uses memory to store data, and the reading and writing speed is extremely fast. The database uses mysql as an embodiment in the present invention. MySQL database is one of the most popular mainstream databases, and data is persistently stored in the local disk.

[0056] The present invention uses Zabbix as a monitoring system embodiment, and the monitoring function module diagram is shown in Figure 2 , Zabbix is ​​divided into two parts: Zabbix server service and Zabbix agent service. Zabbix agent can collect corresponding data on the server side according to the collection items configured by Zabbix server. If the native Zabbix does not support the required collection indicators, it can also support scripts developed by developers themselves. After Zabbix agent collects data, it uploads it to Zabbix server; Zabbix server can provide external API access to provide collection data query. The monitoring system also needs a background service to regularly access the Zabbix server API to obtain the collection data, and then write it to redis.

[0057] The present invention uses the log service system ELK as an embodiment, and the log service function module diagram is shown in Figure 3 ELK is a combination of three subsystems: ElasticSearch, LogStash, and Kibana. LogStash is responsible for collecting and filtering logs, and finally storing them for use; ElasticSearch provides a full-text search engine with distributed multi-user capabilities and a Restful web interface; Kibana can display and analyze the logs provided by LogStash and LogStash in a friendly web interface. After the log system collects the database log files, it stores the database update information that matches the rules. The log system requires a background service to regularly access the ElasticSearch API to write the required data into redis.

[0058] Common Web backend services include spring boot service, PHP, Python's Django, flask and many other Web technology frameworks. The present invention uses flask as an embodiment.

[0059] According to an embodiment of the present invention, an embodiment of a method for optimizing Web cache settings is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] In this embodiment, a method for optimizing Web cache settings is provided, which can be used in electronic devices such as terminals, servers, etc. Figure 4 is a flow chart of a method for optimizing Web cache settings according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0061] S11, obtaining the access time interval of each API, and writing the access time interval into the first queue.

[0062] When network traffic accesses the Web backend system, the program records the timestamp of the URL access, converts the timestamp data into time interval data, and continuously writes the time interval data of each incoming ApiKey into the first queue corresponding to the respective ApiKey, that is, the redis queue. The redis queue here is set with a fixed length N according to the actual business situation. The data marker variable in this queue is the array data, with subscripts from 0 to N-1, N-1 is the latest data, and the array data will be used as the input parameter of the subsequent adaptive algorithm to determine the current trend value.

[0063] S12, using the access time interval in the first queue, respectively calculating the long-term trend value, the short-term trend value and the average value to determine the long-term trend array and the short-term trend array.

[0064] In this embodiment, the core idea of ​​the adaptive algorithm is to consider the impact of the time dimension and historical data on the current data. Based on this, two trend data lines are designed, one uses the long-term constant L to calculate the data trend change; the other uses the short-term constant S to calculate the data trend change; the difference between the short-term trend data and the long-term trend data can more accurately reflect the overall trend change. At the same time, the variance and of the overall data are considered in order to consider the consistency of data fluctuations. In order to calculate the impact of the time dimension on the trend, the calculation of the current values ​​of all key steps is based on the data calculation of the previous time unit, so that the time dimension can be the final trend data result.

[0065] Among them, the electronic device uses the access time interval in the first queue, the long-period constant L and the last long-term trend value to calculate the current long-term trend value; uses the access time interval in the first queue, the short-period constant S and the last short-term trend value to calculate the current short-term trend value; and then uses the access time interval in the first queue to calculate the average value.

[0066] S13, performing variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine the trend difference array.

[0067] The electronic device calculates the sum of variances using the long-term trend array, the short-term trend array and the average value to obtain a trend difference array. Each element in the trend difference array is calculated by the corresponding element in the long-term trend array, the corresponding element in the short-term trend array and the average value.

[0068] S14, calculating the current trend value based on the last value in the trend difference array and the previous trend value.

[0069] The current trend value can also be considered as the most recent trend value, which is calculated using the last value in the trend difference array and the previous trend value. The last value of the calculated trend data RLT N-1 Represents the current trend value. If RLT N-1 If it is greater than 0, it indicates that the current API access interval value is in a trend of extension. N-1 The larger the value, the longer the predicted future interval is and the less frequent the access is. N-1 Less than 0, indicating that the API access interval is decreasing. N-1 The smaller the value, the shorter the predicted future interval and the more frequent the access.

[0070] S15, obtaining current performance data of the target server to determine a hotspot threshold.

[0071] The backend server performance indicator data written by the monitoring system is read from redis. The idler the server is, the more hotspot data will enter the cache system, and the hotspot threshold will be increased; when the server load is higher, less hotspot data will enter the cache system, and the hotspot threshold will be reduced. Therefore, the electronic device determines the current hotspot threshold by obtaining the current performance data of the target server in real time.

[0072] S16, determining the target API according to the size relationship between the current trend value corresponding to each API and the hotspot threshold, determining the target API as hotspot data and caching the target API.

[0073] The electronic device compares the current trend value with the calculated hotspot threshold. If it is smaller, it indicates that the API is the target API and is hotspot data. Otherwise, it is not hotspot data and the data can be accessed normally.

[0074] The Web cache setting optimization method provided in this embodiment improves the hit rate of hot data by adaptively determining whether the accessed API belongs to hot data. The prior art can only select hot URLs manually, while the present invention can decide whether the current API should enter the hot range based on real-time access data. This function can greatly optimize the problem of Nosql system occupying a large amount of memory; traditional data uses disk to store data, and Nosql cannot copy all hot data into memory due to limited memory capacity.

[0075] In this embodiment, a method for optimizing Web cache settings is provided, which can be used in electronic devices such as terminals, servers, etc. Figure 5 is a flow chart of a method for optimizing Web cache settings according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0076] S21, obtaining the access time interval of each API, and writing the access time interval into the first queue.

[0077] For details, please see Figure 4 S11 of the illustrated embodiment will not be described in detail here.

[0078] S22, using the access time interval in the first queue, respectively calculating the long-term trend value, the short-term trend value and the average value to determine the long-term trend array and the short-term trend array.

[0079] Specifically, it includes:

[0080] (1) Set the long-term trend data cycle constant to L, the short-term trend data cycle constant to S, and the cycle constant to M. Set different values ​​of L, S, and M according to actual needs. For example, if you want high sensitivity, you can select parameters S = 11, L = 27, M = 9; if you want high accuracy, you can select parameters S = 131, L = 287, M = 110.

[0081] (2) Initialize all calculation parameters required by the adaptive algorithm:

[0082] SEMA0=SEMA1=0

[0083] LEMA0=LEMA1=0

[0084] LEMA0=LEMA1=0

[0085] RLT0=0

[0086] (3) Calculate the short-term trend array SEMA, and use S as the period constant

[0087]

[0088] (4) Calculate the long-term trend array LEMA, and use L as the period constant

[0089]

[0090] (5) Calculate the average value

[0091]

[0092] S23, performing variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine the trend difference array.

[0093] Specifically, the trend difference array DIFF is calculated using the following formula:

[0094]

[0095] S24, calculating the current trend value based on the last value in the trend difference array and the previous trend value.

[0096] Among them, the trend value array RLT is calculated using the following formula:

[0097] RLT i =((M-1)*RLT i-1 +2*DIFF i )÷(M+1)

[0098] Accordingly, the current trend value is calculated using the following formula:

[0099] RLT N-1 =((M-1)*RLT N-2 +2*DIFF N-1 )÷(M+1)

[0100] Among them, RLT N-1 is the current trend value, RLT N-2 is the previous trend value, DIFF N-1 is the last value in the trend difference array, and M is the period constant.

[0101] S25, obtaining current performance data of the target server to determine a hotspot threshold.

[0102] Specifically, the above S25 includes:

[0103] S251, obtaining the CPU usage percentage, memory usage percentage, and disk usage percentage of the target server.

[0104] The monitoring system can sample the key performance parameters of the target server in real time, such as CPU, memory, disk, etc., and write the performance ratio into redis periodically.

[0105] S252: Perform weighted calculation based on the CPU usage percentage, the memory usage percentage, and the disk usage percentage to determine a hotspot threshold.

[0106] Read the backend server performance indicator data written by the monitoring system from redis, and calculate the impact of the current server performance on the cache strategy according to the following formula. The calculation formula for the hotspot threshold is as follows: Threshold = Max*((1-CPU used )*Rate cpu +(1-MEM used )*Rate mem +(1-DISK used )*Rate disk )

[0107] Max: [-10, 0] The maximum value of the threshold, adjusted according to the business usage scenario

[0108] CPU used :[0, 1.0] CPU usage ratio

[0109] Rate cpu : [0, 1.0] CPU weight, adjusted according to business usage scenarios

[0110] MEM used : [0, 1.0] memory usage percentage

[0111] Rate mem : [0, 1.0] Memory usage weight, adjusted according to business usage scenarios

[0112] DISK used :[0, 1.0]disk usage percentage

[0113] Rate disk : [0, 1.0] disk weight, adjusted according to business usage scenarios

[0114] The rate is adjusted according to the business usage scenario.

[0115] S26, determining the target API according to the size relationship between the current trend value corresponding to each API and the hotspot threshold, determining the target API as hotspot data and caching the target API.

[0116] Specifically, the above S26 includes:

[0117] S261, corresponding to each API, determining whether the current trend value corresponding to the API is less than the hotspot threshold;

[0118] When the current trend value corresponding to the api is less than the hotspot threshold, execute S262; otherwise, execute S263.

[0119] S262, determine the api as the target api and hotspot data and cache the target api.

[0120] S263, access the database corresponding to the API.

[0121] If RLT N-1 If it is less than the set hotspot threshold, it can be listed as hotspot data and the data can be written into the redis cache. Otherwise, the database is accessed to obtain the data.

[0122] The Web cache setting optimization method provided in this embodiment fully considers the impact of the real-time performance data of the target server on the cache business. Under different server load conditions, the hotspot threshold will change dynamically, thereby ensuring that the cache server runs at the highest possible performance while also ensuring system security.

[0123] In some optional implementations of this embodiment, the above S21 may further include:

[0124] (1) Get access request.

[0125] (2) Determine whether a corresponding interface keyword exists in the cache based on the access request.

[0126] (3) When the interface keyword does not exist, the step of obtaining the access time interval of each API is executed.

[0127] The electronic device obtains the cache interface keyword, namely ApiKey, according to the access request URL plus multiple access parameters, which are converted into a string after sorting; if the key exists, the cache data is directly read and returned; if the ApiKey does not exist, it enters S21.

[0128] When the Flask service receives an API access, it can obtain the URL path and access parameter list for each API access, concatenate the URL with the string converted after the parameters are sorted, and take the API parameters into consideration because different URL parameters will result in different database retrieval data; thus, the unique cache key of the API access can be obtained. The cache key is taken to query in redis. If the cache key of the API access exists, the cache content is directly read and returned, otherwise the adaptive algorithm calculation begins to make a judgment.

[0129] In this embodiment, a method for optimizing Web cache settings is provided, which can be used in electronic devices such as terminals, servers, etc. Figure 6 is a flow chart of a method for optimizing Web cache settings according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps:

[0130] S301, obtaining the access time interval of each API, and writing the access time interval into the first queue.

[0131] For details, please see Figure 5 S21 of the illustrated embodiment will not be described in detail here.

[0132] S302, using the access time interval in the first queue, respectively calculating the long-term trend value, the short-term trend value and the average value to determine the long-term trend array and the short-term trend array.

[0133] For details, please see Figure 5 S22 of the illustrated embodiment will not be described in detail here.

[0134] S303, performing variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine the trend difference array.

[0135] For details, please see Figure 5 S23 of the illustrated embodiment will not be described in detail here.

[0136] S304, calculating the current trend value based on the last value in the trend difference array and the previous trend value.

[0137] For details, please see Figure 5 S24 of the illustrated embodiment will not be described in detail here.

[0138] S305: Acquire current performance data of the target server to determine a hotspot threshold.

[0139] For details, please see Figure 5 S25 of the illustrated embodiment will not be described in detail here.

[0140] S306, determining the target API according to the magnitude relationship between the current trend value corresponding to each API and the hotspot threshold, determining the target API as hotspot data and caching the target API.

[0141] For details, please see Figure 5 S26 of the illustrated embodiment will not be described in detail here.

[0142] S307, obtaining the update time interval of each database, and writing the update time interval into the second queue.

[0143] To obtain the update time intervals of each database, a log collection system is used. The log collection agent program obtains the database table update timestamp data by real-time parsing of the database log file, and then reports it to the log server. The log server program converts the timestamp data into update time interval data. Using the implicit relationship between the URL and the database table, the database update interval data sampling array corresponding to the URL can be obtained and recorded as a variable db, with subscripts from 0 to n-1, where n-1 is the latest data. The log system will write the update time interval in real time to the second queue, that is, the redis queue, for the adaptive algorithm to call when calculating.

[0144] S308, using the update time interval in the second queue, respectively calculate the updated long-term trend value, the updated short-term trend value and the updated average value to determine the updated long-term trend array and the updated short-term trend array.

[0145] S309, performing variance and calculation based on the updated long-term trend array, the updated short-term trend array, and the updated average value to determine an updated trend difference array.

[0146] S310, calculating a current update trend value based on the last value in the update trend difference array and the previous update trend value.

[0147] The specific calculation method of the current update trend value is similar to the method of calculating the current trend value in the above S22-S24. The difference is that the calculation data of the current update trend value is the database update time interval, while the calculation data of the current trend value in S22-S2 is the access time interval.

[0148] S311, determining a cache expiration time of the database according to the current update trend value and the update average value.

[0149] Specifically, the cache expiration time is calculated using the following formula:

[0150]

[0151] Among them, RLT n ' -1 is the current update trend value, RATE is the update weight, n is the number of update time intervals in the second queue, db i is the i-th update time interval in the second queue, and RANDOM is a random value.

[0152] After calculating the cache expiration time, the electronic device can clear the cache based on the cache expiration time.

[0153] The Web cache setting optimization method provided in this embodiment uses an adaptive algorithm to calculate a scientific and reasonable cache expiration time based on the database update interval data. When the database update trend value continues to increase, it means that the database data changes slowly, and the adaptive algorithm will increase the cache expiration time. On the contrary, it means that the database changes faster and the cache expiration time will be reduced. Thereby optimizing the performance of the cache system and reasonably maximizing the cache effect. Since the problems of avoiding cache breakdown and cache avalanche are taken into consideration, random values ​​are introduced in the calculation process, which improves the system security and is more suitable as a solution for the cache system in high concurrency scenarios.

[0154] In a specific implementation of this embodiment, Figure 7 As shown, taking Python language as an example, the method includes:

[0155] Step 1: Use the decorator app.before_request in the Flask framework to implement the interceptor function. When an incoming API is received, get the endpoint attribute, uri attribute and parameter object from the request object, concatenate the string converted from the uri and parameter queues after sorting as the redis key, recorded as ApiKey. Use the Python redis library to access the redis database. If the corresponding redis key exists, get the cached data and return; if not, go to step 2.

[0156] Step 2: Use the keyword Interval concatenated with ApiKey as the redis key, record it as ApiIntervalKey, access redis to obtain the interval data queue, and record it as data.

[0157] Step 3: Start calculating trend data according to the adaptive algorithm and record the result as RLT.

[0158] Step 3.1: Set variable initialization values

[0159] SEMA0=SEMA1=0

[0160] LEMA0=LEMA1=0

[0161] DIFF0=0

[0162] RLT0=0

[0163] Step 3.2: Calculate short-term trend data using Python

[0164]

[0165] Step 3.3: Calculate long-term trend data using Python

[0166]

[0167] Step 3.4: Calculate the average using Python

[0168]

[0169] Step 3.5: Calculate the trend difference and variance using Python

[0170]

[0171] Step 3.5: Calculate trend value using Python

[0172] RLT i =((M-1)*RLT i-1 +2*DIFF N-1 )÷(M+1)

[0173] Step 4: Start calculating trend data according to the adaptive algorithm and record the result as RLT.

[0174] Step 5: Calculate and set the cache based on the Threshold, specifically:

[0175] Rate cpu =Rate mem =Rate disk =1 / 3

[0176] CPU used , MEM used , DISK used =Python redis accesses the monitoring system to collect performance data Threshold = Max

[0177] *((1-CPU used )*Rate cpu +(1--MEM used )*Rate mem +(1--DISK used )*Rate disk )

[0178] if RLT i <THRESHOLD:

[0179] Using Python sqlalchemy library to access mysql database

[0180] Use Python redis library to write redis cache according to database return value

[0181] else

[0182] Use Python sqlalchemy to access the mysql database and return data directly

[0183] Step 6: Use the Python redis library to obtain the database update interval sampling data and record it as DB

[0184] Step 7: According to the function in step 3, use DB as the input data array to calculate the new RLT'

[0185] Step 8: Calculate the cache expiration time according to the following formula and write it into the redis cache system using the Python redis library:

[0186]

[0187] The Web cache setting optimization method provided by the implementation of the present invention is suitable for optimizing the cache system in large-scale Internet http access scenarios, can greatly improve the hit rate of the cache system and the rationality of the cache expiration time setting, and can ensure the healthy operation of the cache system and maximize its performance.

[0188] In this embodiment, a Web cache setting optimization device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0189] This embodiment provides a Web cache setting optimization device, including:

[0190] A first acquisition module, used to acquire the access time interval of each API, and write the access time interval into the first queue;

[0191] A first calculation module, used to calculate the long-term trend value, the short-term trend value and the average value respectively by using the access time interval in the first queue to determine the long-term trend array and the short-term trend array;

[0192] A first determining module, configured to perform variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine a trend difference array;

[0193] A second calculation module, used for calculating the current trend value based on the last value in the trend difference array and the previous trend value;

[0194] A second acquisition module is used to acquire current performance data of the target server to determine a hotspot threshold;

[0195] The second determination module is used to determine the target API according to the size relationship between the current trend values ​​corresponding to the various APIs and the hotspot threshold, determine the target API as hotspot data, and cache the target API.

[0196] The Web cache setting optimization device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and a memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0197] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0198] The present invention also provides an electronic device. Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the electronic device may include: at least one processor 41, such as a CPU (Central Processing Unit), at least one communication interface 43, a memory 44, and at least one communication bus 42. The communication bus 42 is used to realize the connection and communication between these components. The communication interface 43 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 43 may also include a standard wired interface and a wireless interface. The memory 44 may be a high-speed RAM memory (Random Access Memory, volatile random access memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 44 may optionally be at least one storage device located away from the aforementioned processor 41. The processor 41 calls the program code stored in the memory 44 to execute any of the above method steps.

[0199] The communication bus 42 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 42 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0200] Among them, the memory 44 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 44 may also include a combination of the above types of memory.

[0201] The processor 41 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0202] The processor 41 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0203] Optionally, the memory 44 is also used to store program instructions. The processor 41 can call the program instructions to implement the Web cache setting optimization method shown in any embodiment of the present application.

[0204] The embodiment of the present invention also provides a non-transitory computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the Web cache setting optimization method in any of the above method embodiments. Wherein, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0205] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for optimizing Web cache settings, characterized in that: include: Obtain the access time interval of each API, and write the access time interval into the first queue; Using the access time intervals in the first queue, respectively calculating the long-term trend value, the short-term trend value and the average value to determine a long-term trend array and a short-term trend array; Perform variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine a trend difference array; Calculating the current trend value based on the last value in the trend difference array and the previous trend value; Obtain current performance data of the target server to determine the hotspot threshold; The target API is determined according to the magnitude relationship between the current trend values ​​corresponding to the various APIs and the hotspot threshold, and the target API is determined as hotspot data and cached.

2. The method according to claim 1, characterized in that The current trend value is calculated using the following formula: RLT N-1 =((M-1)*RLT N-2 +2*DIFF N-1 )÷(M+1) Among them, RLT N-1 is the current trend value, RLT N-2 is the previous trend value, DIFF N-1 is the last value in the trend difference array, and M is the period constant.

3. The method according to claim 1, characterized in that The obtaining of current performance data of the target server to determine the hotspot threshold includes: Obtain the CPU usage percentage, memory usage percentage, and disk usage percentage of the target server; A weighted calculation is performed based on the CPU usage percentage, the memory usage percentage, and the disk usage percentage to determine the hotspot threshold.

4. The method according to claim 1, characterized in that: The step of determining a target API according to a magnitude relationship between a current trend value corresponding to each API and the hotspot threshold, determining the target API as hotspot data, and caching the target API includes: Corresponding to each API, determining whether the current trend value corresponding to the API is less than the hotspot threshold; When the current trend value corresponding to the api is less than the hotspot threshold, the api is determined to be the target api and the hotspot data and the target api is cached.

5. The method according to claim 4, characterized in that The method further comprises: determining a target API according to a magnitude relationship between a current trend value corresponding to each API and the hotspot threshold, determining the target API as hotspot data and caching the target API; When the current trend value corresponding to the API is greater than or equal to the hotspot threshold, the database corresponding to the API is accessed.

6. The method according to claim 1, characterized in that Before the step of obtaining the access time interval of each API, the method further includes: Get access request; Determining whether a corresponding interface keyword exists in the cache based on the access request; When the interface keyword does not exist, the step of obtaining the access time interval of each API is executed.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtaining the update time interval of each database, and writing the update time interval into the second queue; Using the update time interval in the second queue, respectively calculate the updated long-term trend value, the updated short-term trend value and the updated average value to determine the updated long-term trend array and the updated short-term trend array; Perform variance and calculation based on the updated long-term trend array, the updated short-term trend array and the updated average value to determine an updated trend difference array; The current update trend value is calculated based on the last value in the update trend difference array and the previous update trend value; The cache expiration time of the database is determined according to the current update trend value and the update average value.

8. The method according to claim 7, characterized in that The cache expiration time is calculated using the following formula: Among them, RLT′ n-1 is the current update trend value, RATE is the update weight, n is the number of update time intervals in the second queue, db i is the i-th update time interval in the second queue, and RANDOM is a random value.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the Web cache setting optimization method according to any one of claims 1 to 8 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the Web cache setting optimization method according to any one of claims 1 to 8.

Images