Self-adaptive Redis database read-write performance optimization method and device
Through adaptive optimization of the read and write performance of Redis database, dynamically adjusting the batch quantity, solving the problem of improper batch quantity setting in the existing technology, and achieving the performance improvement of Redis in high concurrency scenarios and improving system efficiency.
Patent Information
- Application Number
- CN202510629434.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-08-15
AI Technical Summary
The prior art cannot automatically adapt to different command lengths and scenarios to optimize batch processing quantity, resulting in limited performance improvement of Redis in high concurrency processing.
Through the adaptive Redis database read and write performance optimization method, the operand data per second of specific Redis commands under different data lengths and batch quantities are obtained, the batch number is dynamically optimized, batch write and read commands are generated, and the average length changes of batch compressed data are analyzed to improve the read and write performance of Redis database.
Significantly improve Redis read and write performance, adapt to dynamically changing high-concurrency scenarios, no manual intervention is required, reduce debugging costs, and improve system throughput and response efficiency.
Smart Images
Figure CN120492459A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data management, and in particular to a method and device for adaptively optimizing the read and write performance of a Redis database. Background Art
[0002] As a high-performance, in-memory database, Redis is widely used in high-concurrency scenarios such as financial transactions, interactive gaming, medical monitoring, and personalized e-commerce services, thanks to its fast read and write capabilities and flexible data structures, such as strings, hashes, lists, sets, and ordered sets. However, with the surge in the number of internet users and business volume, Redis faces significant performance challenges in high-concurrency processing, low-latency responses, and data consistency.
[0003] Existing technologies for optimizing Redis read and write performance primarily focus on improving memory management, enhancing network transmission efficiency, hardware adaptation, configuration parameter tuning, and optimizing application-layer policies. Among these, Redis pipeline technology reduces network round trips by batching commands, making it an important means of improving read and write performance. However, existing research lacks a systematic analysis of the operating mechanism, applicable scenarios, and performance boundaries of pipeline technology, and is particularly flawed in setting the number of batches. Experiments have shown that the larger the batch size, the better. Excessively large batches increase the burden on the client and server, while too small a number will not fully realize performance potential. Furthermore, the optimal value for the number of batches is closely related to the command length and the number of operations per second on the server. Existing technologies do not provide a batch size optimization method that automatically adapts to different command lengths and scenarios, resulting in limited performance improvements and difficulty meeting dynamically changing high-concurrency requirements. Summary of the Invention
[0004] The purpose of the present invention is to provide an adaptive Redis database read and write performance optimization method and device to solve the problem that the existing technology cannot automatically adapt to different command lengths and scenarios to optimize the batch number.
[0005] To achieve one of the above-mentioned objectives, an embodiment of the present invention provides an adaptive Redis database read and write performance optimization method, the method comprising:
[0006] In response to changes in data length or adjustments to system performance requirements, obtain data on the number of operations per second for a specific Redis command at different data lengths and batch sizes, and determine the optimal batch size;
[0007] Process the data length, compress the original data, generate batch write commands and read commands, and package them according to the optimal batch processing quantity and send them to the Redis server; receive the batch compressed data returned by the Redis server, decompress it, and restore the original data;
[0008] Analyze the changes in the average length of batch compressed data returned by the Redis server, draw conclusions on adjusting the batch number, and dynamically optimize the batch number to improve the read and write performance of the Redis database.
[0009] As a further improvement of an embodiment of the present invention, the method further includes, before “obtaining data on the number of operations per second of a specific Redis command under different data lengths and batch sizes in response to a change in data length or an adjustment in system performance requirements”, further including:
[0010] When the system starts, it reads the Redis connection information and data compression algorithm configuration from the configuration file. It also performs performance testing to test the number of operations per second of specific Redis commands under different data lengths and batch sizes, determines the default batch size, and completes initialization.
[0011] As a further improvement of an embodiment of the present invention, the method further includes that determining the optimal value of the batch processing quantity includes:
[0012] Using the results of the performance test, draw a trend curve with the number of batches as the horizontal axis and the number of operations per second as the vertical axis;
[0013] The inflection point of the curve is determined by calculating the slope of adjacent data points. When the slope is less than or equal to zero, the batch size of the previous data point is selected as the optimal batch size.
[0014] The Redis command and data length corresponding to the optimal batch processing quantity are stored in the query table.
[0015] As a further improvement of an embodiment of the present invention, the method further includes that compressing the original data to generate a batch write command includes:
[0016] Use data compression algorithm to compress the original data and calculate the average length of the compressed data;
[0017] Querying the optimal value of the batch processing quantity in the query table according to the average length of the compressed data;
[0018] If the optimal batch processing quantity is found, the compressed data is used as a Redis command parameter according to the optimal batch processing quantity, a write command is generated, and the compressed data is batch packaged and sent to the Redis server;
[0019] If not found, the performance test is triggered, and the new optimal value of the batch processing quantity is determined based on the performance test results. The query table is updated, and based on the new optimal value of the batch processing quantity, the compressed data is used as the Redis command parameter, a write command is generated, and the compressed data is batch packaged and sent to the Redis server.
[0020] As a further improvement of an embodiment of the present invention, the method further includes that generating a batch read command includes:
[0021] Generate read commands based on the optimal batch processing quantity, batch-pack them, and send them to the Redis server;
[0022] Receive the batch-packaged compressed data returned by the Redis server, decompress the received batch-packaged compressed data in the order of the received data, and restore the original data.
[0023] As a further improvement of an embodiment of the present invention, the method further includes: analyzing the change in the average length of the compressed data and dynamically optimizing the number of batches includes:
[0024] Calculate the average length of the compressed data of the current batch, and query the optimal value of the batch processing quantity in the query table;
[0025] If the optimal batch processing quantity is found, the default batch processing quantity is replaced by the optimal batch processing quantity and the configuration file is updated;
[0026] If not found, a performance test is triggered, a new optimal batch quantity value is determined based on the performance test result, the query table and configuration file are updated, and the default batch quantity is replaced with the new optimal batch quantity value.
[0027] As a further improvement of one embodiment of the present invention, the method also includes regularly monitoring the average length of compressed data and system performance indicators, analyzing performance trends, and regularly querying tables and optimal batch processing quantities to continuously optimize the read and write performance of the Redis database.
[0028] To achieve one of the above-mentioned objects of the invention, an embodiment of the present invention further provides an adaptive Redis database read and write performance optimization device, which is characterized by comprising a performance testing module, a data processing module and an optimization module;
[0029] The performance testing module is used to obtain the number of operations per second of a specific Redis command under different data lengths and batch processing quantities in response to changes in data length or adjustments to system performance requirements, and determine the optimal value of the batch processing quantity;
[0030] The data processing module is used to process the data length, compress the original data, generate batch write commands and read commands, and package them according to the optimal batch processing quantity and send them to the Redis server; receive the batch compressed data returned by the Redis server and decompress it to restore the original data;
[0031] The optimization module is used to analyze the changes in the average length of batch compressed data returned by the Redis server, draw conclusions on the adjustment of the batch processing quantity, and dynamically optimize the batch processing quantity to improve the read and write performance of the Redis database.
[0032] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor, characterized in that the memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the above-mentioned adaptive Redis database read and write performance optimization method are implemented.
[0033] To achieve one of the above-mentioned objects of the invention, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, the steps in the above-mentioned adaptive Redis database read and write performance optimization method are implemented.
[0034] Compared to existing technologies, this invention provides an adaptive Redis database read and write performance optimization method and device. This method automatically generates an optimal number of pipeline batches based on device and system configurations, and increases this number by shortening input data length, significantly improving Redis read and write performance. Furthermore, this automated optimization process eliminates the need for manual intervention, adapts to dynamically changing high-concurrency scenarios, effectively reduces debugging costs, and improves system throughput and response efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is an overall flow chart of the adaptive Redis database read and write performance optimization method described in the present invention.
[0036] Figure 2 This is a flow chart of automatically obtaining the optimal value of the batch processing quantity in the adaptive Redis database read and write performance optimization method described in the present invention.
[0037] Figure 3 This is a flow chart of increasing the optimal value of the batch processing quantity in the adaptive Redis database read and write performance optimization method described in the present invention to improve the Redis database read and write performance.
[0038] Figure 4 This is a flow chart of dynamically adjusting the number of batches of data sent by Redis in the adaptive Redis database read and write performance optimization method of the present invention.
[0039] Figure 5 Schematic diagram of the architecture of the adaptive Redis database read and write performance optimization device shown in the present invention. DETAILED DESCRIPTION
[0040] The present invention will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by those skilled in the art based on these embodiments are all within the scope of protection of the present invention.
[0041] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.
[0042] In the first embodiment of the present invention, the present invention provides an adaptive Redis database read and write performance optimization method, such as Figure 1 As shown, the method includes,
[0043] S1: In response to changes in data length or adjustments to system performance requirements, obtain the number of operations per second for a specific Redis command at different data lengths and batch sizes, and determine the optimal batch size;
[0044] S2: Processes the data length, compresses the original data, generates batch write commands and read commands, and packages them according to the optimal batch processing quantity and sends them to the Redis server; receives the batch compressed data returned by the Redis server, decompresses it, and restores the original data;
[0045] S3: Analyze the changes in the average length of batched compressed data returned by the Redis server, draw conclusions on how to adjust the number of batches, and dynamically optimize the number of batches to improve the read and write performance of the Redis database.
[0046] In a specific embodiment of the present invention, before “obtaining data on the number of operations per second of a specific Redis command under different data lengths and batch sizes in response to a change in data length or an adjustment in system performance requirements”, the method further includes:
[0047] When the system starts, it reads the Redis connection information and data compression algorithm configuration from the configuration file. It also performs performance testing to test the number of operations per second of specific Redis commands under different data lengths and batch sizes, determines the default batch size, and completes initialization.
[0048] Specifically, at system startup, the system reads Redis connection information and data compression algorithm configuration from the configuration file. Redis connection information includes the Redis server's IP address, port number, and authentication credentials, which are used to establish communication with the Redis server. The data compression algorithm configuration includes the compression algorithm type (such as LZ4 or Zstandard) and its parameters (such as the compression level) for subsequent data compression processing. The configuration file uses a key-value format (such as JSON or YAML) and is stored in a system-specified path.
[0049] Furthermore, using performance testing tools (such as the official Redis-benchmark), test the number of operations per second (QPS) of specific Redis commands under different data lengths and batch sizes to determine the default batch size. The specific steps are as follows:
[0050] Test configuration: Select representative Redis commands (such as LPUSH and SET), set the data length range (such as 100 to 1000 bytes, in increments of 100 bytes) and the batch number range (such as 0 to 1000, in increments of 50).
[0051] Execute the test: Use the -t parameter of redis-benchmark to specify the command, the -d parameter to specify the data length, and the -P parameter to specify the batch size. Record the QPS under each configuration.
[0052] Data analysis: For each data length, analyze the relationship between batch size and QPS. Identify the minimum batch size at which QPS no longer increases significantly or begins to fluctuate. Use this as the default batch size.
[0053] Storage results: Associate the determined default batch quantity with the corresponding Redis command and data length, and store them in the system configuration file or memory.
[0054] After reading the configuration file and determining the default batch size, the system completes initialization. Initialization results include establishing a connection with the Redis server and verifying the connection's validity. Loading the data compression algorithm ensures that subsequent data processing complies with the configuration. Setting the default batch size serves as the initial parameter for subsequent commands.
[0055] In a specific embodiment of the present invention, the optimal value of the batch processing quantity is determined as follows:
[0056] Using the results of the performance test, draw a trend curve with the number of batches as the horizontal axis and the number of operations per second as the vertical axis;
[0057] Determine the inflection point of the curve by calculating the slope of adjacent data points. When the slope is less than or equal to zero, select the batch processing quantity of the previous data point as the optimal batch processing quantity value.
[0058] Store the Redis command and data length corresponding to the optimal batch processing quantity value in the query table.
[0059] In a specific implementation scenario of the present invention, as Figure 2 shown, the steps to automatically obtain the optimal batch processing quantity value are as follows:
[0060] Step 1: Use the redis-benchmark tool in combination with an automatic test script to generate Table 1. For example, the name of the automatic test script: auto_test_redis; input parameters: -c <Redis command name>; -d <Redis input data length 1, Redis input data length 2,..., Redis input data length n>; script output: xlsx table.
[0061] The pseudo-code of the automatic test script process is as follows:
[0062] Input: -c <Redis command name>; -d <Redis input data length 1, Redis input data length 2,..., Redis input data length n>
[0063] Output: xlsx table
[0064] 1. Create and open the xlsx table "Table 1" and set the "append write" permission.
[0065] 2. Set the pipeline batch processing quantity array a_pipe_cnt[] = [10, 20, 40, 80, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000]. [[ID=2,7]]
[0066] 3. Create a header for "Table 1". The name of the first column is: "Redis command name", and the names of the second column to the nth column are: the element values of the pipeline batch processing quantity array.
[0067] 4. FOR i = 1 to the number of Redis input data lengths
[0068] 5. Write the Redis input data length [i] to the first column and the (1 + i)th row of Table 1.
[0069] 6. FOR j = 0 to the number of elements in the array a_pipe_cnt - 1
[0070] 7. redis test command =. / redis - benchmark - t <Redis command name> - d Redis input data length [i] - P a_pipe_cnt [j] - c 1
[0071] 8. Execute the redis test command to obtain the number of operations per second, ops
[0072] 9. Write ops to the second + j column of the first + i row
[0073] 10. END FOR
[0074] 11. END FOR
[0075] For example, execute the automatic test script. / auto_test_redis - c LPUSH - d 100,500,1000, and the output is shown in Table 1
[0076] Step 2: Analyze the LPUSH command in Table 1. The batch processing quantities are 10, 20, 40, etc. When the data length is 100, the number of operations per second are 259067, 442477, 68493, etc. Taking the batch processing quantity as the X - axis and the number of operations per second as the Y - axis, if the calculated slope k1=(y2 - y1) / (x2 - x1), when k1 <= 0, determine (x1, y1) as the inflection point, and the corresponding number of operations per second y1 is the optimal value of the batch processing quantity, and generate Table 2
[0077] Preferably, in the implementation scenario of the present invention, methods such as polynomial fitting, spline fitting, and sliding window analysis can also be used to determine the inflection point
[0078] The pseudo - code for the process of obtaining the optimal value of the batch processing quantity is as follows
[0079] Input: xlsx table storing batch processing quantity and number of operations per second
[0080] Output: xlsx table storing the corresponding relationship between data length and optimal batch processing value
[0081] 1. Create and open the xlsx table "Table 2" and set the "append write" permission
[0082] 2. Create a header for Table 2, the first column is "Data Length", and the second column is "Optimal Batch Processing Value"
[0083] 3. Open the xlsx table "Table 1" of the input parameters
[0084] 4.line_num = "Table 1" row number
[0085] 5.colum_num = "Table 1" column number
[0086] 6. FOR i=2 Same as line_num
[0087] 7.Data_len = value in row i, column 1 of table 1
[0088] 8.For j=2 to colum_num-1
[0089] 9.Ops1 = the value in row i, column j of Table 1
[0090] 10.Ops2 = the value in row i, column j+1 of Table 1
[0091] 11.P1 = value in row 1, column j of Table 1
[0092] 12. P2 = value in row 1, column j+1 of Table 1
[0093] 13.Gradient = (Ops2 - Ops1) / (P2 - P1)
[0094] 14.IF Gradient <= 0
[0095] 15. Data_len is written to column 1, row i of table 2
[0096] 16.Ops1 writes to table 2, column 2, row i
[0097] 17.END IF
[0098] 18.END FOR
[0099] 19.END FOR
[0100] In a specific embodiment of the present invention, the original data is compressed to generate a batch write command, specifically,
[0101] Use data compression algorithm to compress the original data and calculate the average length of the compressed data;
[0102] Querying the optimal value of the batch processing quantity in the query table according to the average length of the compressed data;
[0103] If the optimal batch processing quantity is found, the compressed data is used as a Redis command parameter according to the optimal batch processing quantity, a write command is generated, and the compressed data is batch packaged and sent to the Redis server;
[0104] If not found, the performance test is triggered, and the new optimal value of the batch processing quantity is determined based on the performance test results. The query table is updated, and based on the new optimal value of the batch processing quantity, the compressed data is used as the Redis command parameter, a write command is generated, and the compressed data is batch packaged and sent to the Redis server.
[0105] It should be noted that the present invention uses LZ4 or other fast compression algorithms to compress the original data according to the compression parameters specified in the configuration file, generating compressed data and calculating the average length of the compressed data. Based on the Redis write-side processing flow, the average length of the compressed data is used as a query condition to query the optimal batch size stored in a query table. The query table records the optimal batch size for specific Redis commands (such as LPUSH) at different data lengths. If an optimal batch size matching the current average length of the compressed data is found, the compressed data is used as a Redis command parameter to generate a complete write command. The command is then batched according to the batch size, organized into a single message, stored in the cache, and sent to the Redis server via the Redis send interface. If no matching optimal batch size is found, the default batch size defined in the configuration file is used, and the performance testing module is triggered. The redis-benchmark tool, combined with an automated testing script, tests the number of operations per second of the current Redis command at different data lengths and batch sizes. The test data is analyzed to confirm the new optimal batch size, and the query table is updated. Subsequently, based on the new optimal value of the batch processing quantity, the compressed data is used as a Redis command parameter, a write command is generated and batch packaged, organized into a message, and the Redis sending interface is called to send it to the Redis server.
[0106] In a specific embodiment of the present invention, a batch read command is generated, specifically,
[0107] Generate read commands based on the optimal batch processing quantity, batch-pack them, and send them to the Redis server;
[0108] Receive the batch-packaged compressed data returned by the Redis server, decompress the received batch-packaged compressed data in the order of the received data, and restore the original data.
[0109] It should be noted that, through the data processing module, according to the optimal value of the batch processing quantity stored in the query table, for specific Redis commands and keys, corresponding read commands are generated. According to the Redis connection information defined in the configuration file, a communication connection with the Redis server is established, and according to the optimal value of the batch processing quantity, multiple read commands are batch packaged, organized into a message and stored in the cache, and the Redis sending interface is called to send it to the Redis server at one time. The batch compressed data returned by the Redis server is received. The data is the compression result arranged in order, and the same compression algorithm as the write end is used in sequence. It is decompressed according to the compression parameters specified in the configuration file to restore the original data. The decompressed original data is processed one by one in the order of return to ensure data consistency for subsequent use by the system.
[0110] In a specific implementation scenario of the present invention, Figure 3 As shown in the figure, by increasing the optimal value of the batch number, the read and write performance of the Redis database is improved. It is divided into the Redis data writing end processing flow and the Redis data acquisition end processing flow. The Redis data writing end and the Redis data acquisition end each include three functional modules: initialization module, Redis command request module and Redis command response module. Figure 3 The write process shown includes using the initialization module to obtain the write-side information from the configuration file. When data needs to be stored in the Redis database, the Redis command request module is called to send the data, and the Redis command response module receives the response information from the Redis database. The processing flow of the Redis data acquisition side is the same as that of the write side. The difference is that the acquisition side calls the Redis command response module to receive the query data, while the write side receives the success or failure feedback information. The implementation of the three functional modules is as follows:
[0111] Initialization module: Both the Redis data writer and data retrieval end need to obtain Redis usage information and data compression information from the configuration file. The different types of Redis clients require different configuration information from the configuration file. The contents of the configuration file are shown in Table 3.
[0112]
[0113] The Redis command request module first reads the Redis configuration information, Redis public information, and data compression information for the corresponding client type from the configuration file, based on the Redis client type registered during initialization. It then connects to the Redis database, compresses the original data, packages the Redis commands, and finally sends the Redis commands to the database. The pseudo code for the Redis command request process is shown below:
[0114] Input: The value of the Redis command
[0115] Output: Command sending success or failure status
[0116] 12.IF 1 == Compressed.enable
[0117] 13.Value1 = Compress the input parameter according to Compressed.type
[0118] 14.Value2 = Replace '\0' in Value1 with Compressed.special-char
[0119] 15. Redis value = compressed data length: Value2
[0120] 16.ELSE
[0121] 17.Redis value = input parameter
[0122] 18.END IF
[0123] 19.IF client type == write end
[0124] 20.Redis command = Redis.write-cmd-type
[0125] 21.Redis key = Redis.write-key-name
[0126] 22.Batch number = Redis.pipeline.write-batch-size
[0127] 23.ELSE IF client type == reading end
[0128] 24.Redis command = Redis.read-cmd-type
[0129] 25.Redis key = Redis.read-key-name
[0130] 26.Batch number = Redis.pipeline.read-batch-size
[0131] 27.END IF
[0132] 28.FOR i=0 to batch quantity
[0133] 29.String[i] = Redis command + Redis key + Redis value
[0134] 30. Call the Redis interface to send the complete Redis command String[i]
[0135] 31.END FOR
[0136] The Redis command response module processes the response messages from the requested command according to the Redis client type registered during initialization. If the registered client type is a writer, only the success or failure feedback is processed. If the client type is a reader, the read return data is processed according to the data compression information in the configuration file. The pseudo code of the Redis command response process is as follows:
[0137] Input: Redis return message
[0138] Output: feedback information or data
[0139] 1.IF client type == write end
[0140] 2. Batch return of success or failure feedback information
[0141] 3.ELSE IF client type == reading end
[0142] 4.FOR i = 1 to Redis.pipeline.read-batch-size
[0143] 5.IF 1 == Compressed.enable
[0144] 6. Compressed data length = the content from the beginning to ":" of the input data
[0145] 7.Value1=The first character after “:” to the end
[0146] 8.Value2=Replace the special characters in Value1 with '\0' according to Compressed.special-char
[0147] 9. Data = decompress Value1 according to Compressed.type and compressed data length
[0148] 10.ELSE
[0149] 11. Data = Input Data
[0150] 12.END IF
[0151] 13.END FOR
[0152] 14. Batch return data
[0153] 15.END IF
[0154] In a specific embodiment of the present invention, the average length change of compressed data is analyzed and the number of batches is dynamically optimized, specifically:
[0155] Calculate the average length of the compressed data of the current batch, and query the optimal value of the batch processing quantity in the query table;
[0156] If the optimal batch processing quantity is found, the default batch processing quantity is replaced by the optimal batch processing quantity and the configuration file is updated;
[0157] If not found, a performance test is triggered, a new optimal batch quantity value is determined based on the performance test result, the query table and configuration file are updated, and the default batch quantity is replaced with the new optimal batch quantity value.
[0158] It should be noted that the dynamic optimization module calculates the average length of the current compressed data before each Redis data transmission and compares it with the optimal batch size stored in a lookup table. The lookup table records the optimal batch size for specific Redis commands at different data lengths. If the current average compressed data length matches a record in the lookup table, the optimal batch size corresponding to that record is directly used to organize the batch packaging of write or read commands, and the Redis send interface is called to send the command to the Redis server. If there is a mismatch, indicating that the current data length exceeds the query table coverage, the performance testing module is triggered. Using the redis-benchmark tool combined with an automated testing script, the performance testing module tests the number of operations per second at different batch sizes for the current Redis command and compressed data length. A trend curve is plotted between the batch size and the number of operations per second, and the new optimal batch size is confirmed through polynomial fitting or sliding window analysis. The new optimal batch size is stored in the lookup table along with the Redis command and data length. The default batch size in the configuration file is updated, and the adjusted batch size is applied to subsequent data transmissions, ensuring dynamic optimization of the Redis database's read and write performance.
[0159] In a specific implementation scenario of the present invention, Figure 4 As shown, the steps to dynamically adjust the number of batches of data sent by Redis include:
[0160] On the Redis write side, before each batch of data is organized and sent to Redis, the system checks whether the batch size has been appropriately set based on the compressed data length. If so, it is used directly. Otherwise, the optimal batch size is obtained by querying Table 2 based on the average compressed data length. If the optimal batch size can be obtained from Table 2, it is marked, the default batch size in the configuration file is updated, and the data is sent to Redis based on the optimal batch size. Otherwise, the automatic test script is triggered to update Table 2 based on the calculated compressed data length, and the optimal batch size is then retrieved and used in subsequent processes.
[0161] In a specific embodiment of the present invention, the average length of compressed data and system performance indicators are regularly monitored, and by analyzing performance trends, the optimal values of tables and batch processing quantities are regularly queried to continuously optimize the read and write performance of the Redis database.
[0162] It should be noted that during system operation, the dynamic optimization module regularly monitors the average compressed data length and system performance indicators, including key metrics such as the Redis server's operations per second, response latency, and throughput. The monitoring period can be executed at intervals preset in the configuration file or dynamically adjusted when changes in system load are detected. Based on the monitored data, the changing trends in the average compressed data length and performance indicator fluctuations are analyzed to determine whether the current optimal batch size is still suitable for the system's operating status. If the analysis indicates a declining performance trend or a significant deviation in the average compressed data length from the records in the lookup table, the redis-benchmark tool, combined with automated testing scripts, tests the operations per second at different batch sizes for the current Redis command and compressed data length. A trend curve is plotted between batch size and operations per second, and a new optimal batch size is determined using polynomial fitting or sliding window analysis. The new optimal batch size, along with the Redis command and data length, is stored in the lookup table. The default batch size in the configuration file is updated, and the adjusted batch size is used to organize subsequent data transmissions. This continuously optimizes Redis database read and write performance, ensuring efficient system operation in high-concurrency scenarios.
[0163] In the second embodiment of the present invention, the present invention provides an adaptive Redis database read and write performance optimization device, such as Figure 5 As shown, the device includes a performance testing module 1, a data processing module 2 and an optimization module 3;
[0164] The performance testing module 1 is used to obtain the number of operations per second of a specific Redis command under different data lengths and batch processing quantities in response to changes in data length or adjustments to system performance requirements, and determine the optimal value of the batch processing quantity;
[0165] The data processing module 2 is used to process the data length, compress the original data, generate batch write commands and read commands, and package them according to the optimal batch processing quantity and send them to the Redis server; receive the batch compressed data returned by the Redis server and decompress it to restore the original data;
[0166] The optimization module 3 is used to analyze the changes in the average length of the batch compressed data returned by the Redis server, draw conclusions on the adjustment of the batch processing quantity, and dynamically optimize the batch processing quantity to improve the read and write performance of the Redis database.
[0167] In a third embodiment of the present invention, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps of the adaptive Redis database read and write performance optimization method described above are implemented.
[0168] In a fourth embodiment of the present invention, the present invention provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described adaptive Redis database read and write performance optimization method.
[0169] In summary, the present invention provides an adaptive Redis database read and write performance optimization method and device. This method automatically generates an optimal pipeline batch size based on device and system configuration, and increases this value by shortening input data length, significantly improving Redis read and write performance. Furthermore, this automated optimization process eliminates the need for manual intervention, adapting to dynamically changing high-concurrency scenarios, effectively reducing debugging costs, and improving system throughput and response efficiency.
[0170] It should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation method can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.
[0171] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the modules described above can refer to the corresponding process in the aforementioned method implementation, and will not be repeated here.
[0172] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment according to actual needs.
[0173] In addition, the functional modules in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0174] The above-mentioned integrated modules implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above-mentioned software functional modules are stored in a storage medium and include a number of instructions for causing a computer system (which may be a personal computer, server, or network system, etc.) or a processor to execute some of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application.
Claims
1. An adaptive Redis database read and write performance optimization method, characterized by: include, In response to changes in data length or adjustments to system performance requirements, obtain data on the number of operations per second for a specific Redis command at different data lengths and batch sizes, and determine the optimal batch size; Process the data length, compress the original data, generate batch write commands and read commands, and package them according to the optimal batch processing quantity and send them to the Redis server; receive the batch compressed data returned by the Redis server, decompress it, and restore the original data; Analyze the changes in the average length of batch compressed data returned by the Redis server, draw conclusions on adjusting the batch number, and dynamically optimize the batch number to improve the read and write performance of the Redis database.
2. The adaptive Redis database read and write performance optimization method according to claim 1, characterized in that: Before "Getting the number of operations per second for a specific Redis command at different data lengths and batch sizes in response to changes in data length or adjustments to system performance requirements", it also includes: When the system starts, it reads the Redis connection information and data compression algorithm configuration from the configuration file. It also performs performance testing to test the number of operations per second of specific Redis commands under different data lengths and batch sizes, determines the default batch size, and completes initialization.
3. The adaptive Redis database read and write performance optimization method according to claim 2, characterized in that: Determining the optimal batch processing quantity includes: Using the results of the performance test, draw a trend curve with the number of batches as the horizontal axis and the number of operations per second as the vertical axis; The inflection point of the curve is determined by calculating the slope of adjacent data points. When the slope is less than or equal to zero, the batch size of the previous data point is selected as the optimal batch size. The Redis command and data length corresponding to the optimal batch processing quantity are stored in the query table.
4. The adaptive Redis database read and write performance optimization method according to claim 3, characterized in that: The compressing original data to generate a batch write command includes: Use data compression algorithm to compress the original data and calculate the average length of the compressed data; Querying the optimal value of the batch processing quantity in the query table according to the average length of the compressed data; If the optimal batch processing quantity is found, the compressed data is used as a Redis command parameter according to the optimal batch processing quantity, a write command is generated, and the compressed data is batch packaged and sent to the Redis server; If not found, the performance test is triggered, and the new optimal value of the batch processing quantity is determined based on the performance test results. The query table is updated, and based on the new optimal value of the batch processing quantity, the compressed data is used as the Redis command parameter, a write command is generated, and the compressed data is batch packaged and sent to the Redis server.
5. The adaptive Redis database read and write performance optimization method according to claim 4, characterized in that: Generating a batch read command includes: Generate read commands based on the optimal batch processing quantity, batch-pack them, and send them to the Redis server; Receive the batch-packaged compressed data returned by the Redis server, decompress the received batch-packaged compressed data in the order of the received data, and restore the original data.
6. The adaptive Redis database read and write performance optimization method according to claim 5, characterized in that: The analysis of the average length change of compressed data and the dynamic optimization of the batch processing quantity include: Calculate the average length of the compressed data of the current batch, and query the optimal value of the batch processing quantity in the query table; If the optimal batch processing quantity is found, the default batch processing quantity is replaced by the optimal batch processing quantity and the configuration file is updated; If not found, a performance test is triggered, a new optimal batch quantity value is determined based on the performance test result, the query table and configuration file are updated, and the default batch quantity is replaced with the new optimal batch quantity value.
7. The adaptive Redis database read and write performance optimization method according to claim 1, characterized in that: Also includes, Regularly monitor the average length of compressed data and system performance indicators, analyze performance trends, and regularly query the optimal values of tables and batch quantities to continuously optimize the read and write performance of the Redis database.
8. An adaptive Redis database read and write performance optimization device, characterized by: Including performance testing module, data processing module and optimization module; The performance testing module is used to obtain the number of operations per second of a specific Redis command under different data lengths and batch processing quantities in response to changes in data length or adjustments to system performance requirements, and determine the optimal value of the batch processing quantity; The data processing module is used to process the data length, compress the original data, generate batch write commands and read commands, and package them according to the optimal batch processing quantity and send them to the Redis server; receive the batch compressed data returned by the Redis server and decompress it to restore the original data; The optimization module is used to analyze the changes in the average length of batch compressed data returned by the Redis server, draw conclusions on the adjustment of the batch processing quantity, and dynamically optimize the batch processing quantity to improve the read and write performance of the Redis database.
9. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the adaptive Redis database read and write performance optimization method according to any one of claims 1 to 7 are implemented.
10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the adaptive Redis database read and write performance optimization method as described in any one of claims 1 to 7 are implemented.