Method for smoothly reducing burr probability of server based on tide algorithm

Abstract

The invention is suitable for the technical field of database servers, and provides a method for smoothly reducing the burr probability of a server based on a tidal algorithm, and the method comprises the steps: S1, creating a mongoDB table, and setting a reference time field; s2, carrying out mark fragmentation and mapping on the peak time period and the low peak time period according to a set time value; s3, acquiring expiration reference time of each piece of data, and for the data of the expiration reference time falling into the peak period, adjusting the expiration reference time into the corresponding low peak period according to the set mapping relation; and S4, when the expiration reference time is reached, deleting the data after the corresponding expiration time. According to the technical scheme, the execution time of the mongoDB active data cleaning thread can be changed, so that the time occupied by mongoDB data cleaning due to expiration and failure of a large amount of data is shortened, server burrs can be reduced, and the stability of a server is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of database servers, and in particular relates to a method for smoothly reducing the probability of server glitches based on a tidal algorithm. Background Art

[0002] In some business scenarios, data is often time-sensitive and becomes useless after a certain period of time. If not cleaned up in time, these expired data will take up a lot of storage space, increase database maintenance costs, and even affect query efficiency.

[0003] MongoDB is a database based on distributed file storage. When generating data, a TTL (Time To Live) index is set. When a large amount of data fails at the same time, MongoDB will start a thread to process the failed data. If the amount of data is large, the worker thread will be busy processing the failed data, resulting in a significant increase in the percentage of server CPU usage, reducing the server's ability to provide services to the outside world. When the server CPU is idle, the percentage of usage is very low, and some expired data cannot be processed at this time. This situation greatly reduces the effective utilization of the CPU and the stability of the server.

[0004] Server glitches are common in servers, usually referring to sudden and short-term abnormal fluctuations in performance indicators or resource usage. These fluctuations may manifest as dramatic changes in CPU usage, memory usage, network traffic, or disk I / O. These irregular peaks, like voltage spikes or glitches in a circuit, may have a negative impact on system stability, performance, and user experience. This may also occur if expired server data is not cleaned up in a timely manner.

[0005] The current solutions to this problem are: first, adjust the memory usage plan; second, optimize server performance, such as index optimization, regular index rebuilding, and adjustment of write operation security level. These two solutions are generally applicable, but when it comes to a large amount of data in the database, because it needs to expire at the same time, no matter how the memory usage is adjusted, it cannot change the fact that a large amount of data needs to be processed; since it is a deletion operation, index rebuilding is meaningless, and mongoDB will start another thread to clean up expired data, which will cause the thread to be executed all the time due to the large amount of data, and then cause the server CPU usage percentage to reach almost 100%, which greatly weakens the server's ability to provide external services or even makes it unavailable.

[0006] Currently, the existing technology focuses more on the impact of write operations and query operations on the server, but does not pay attention to the fact that when MongoDB cleans up a large number of data configured with TTL indexes at the same time, the server becomes unstable. Summary of the invention

[0007] In view of the above problems, the purpose of the present invention is to provide a method for smoothly reducing the probability of server glitches based on a tidal algorithm, aiming to solve the technical problem that when a large amount of data fails, the CPU usage percentage of the existing mongoDB database server is high due to mongoDB cleaning up expired data.

[0008] The present invention adopts the following technical solution:

[0009] The method for reducing the probability of server burrs based on tidal algorithm smoothing includes the following steps:

[0010] Step S1, create a mongoDB table and set a reference time field, use the expiration reference time of the reference time field as the designated index field, and set a TTL index of the expiration time for the reference time field;

[0011] Step S2: According to the peak and off-peak period characteristics of the current business data, the peak period and the off-peak period are marked and fragmented with set time values, and then the fragments of the peak period and the off-peak period are mapped;

[0012] Step S3: for the data to be stored, obtain the expiration reference time of each piece of data, and for the data whose expiration reference time falls into the peak period, adjust the expiration reference time to the corresponding off-peak period according to the set mapping relationship;

[0013] Step S4: For each piece of data, when the expiration reference time is reached, the data is deleted after the corresponding expiration time has passed.

[0014] Furthermore, in step S2, the slices are marked with unit hour time values.

[0015] Furthermore, the mapping relationship set in step S2 is a fixed mapping relationship, and each shard during a peak period is mapped to a fixed shard during a low-peak period.

[0016] Furthermore, the mapping relationship set in step S2 is a dynamically adjusted mapping relationship, and the mapping relationship is set as follows:

[0017] Set a data traffic upper limit and count the data traffic during peak hours by seconds. If the current peak time data traffic exceeds the upper limit, the current data is mapped to each off-peak time period modulo the number of off-peak time periods by taking the remainder per second.

[0018] Furthermore, step S3 uses the creation time as the expiration reference time of each piece of data.

[0019] The beneficial effects of the present invention are as follows: the present invention makes full use of the mechanism of mongoDB to clean up expired data and the TTL index working mechanism, marks and fragments the peak time periods and the off-peak time periods, sets the mapping relationship between the peak time periods and the off-peak time periods, can reasonably break up the expiration time of expired data, reduce the execution time of mongoDB's active data cleaning thread, thereby reducing the time occupied by mongoDB to clean up data due to the expiration and invalidation of large quantities of data, and increase the stability of the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a method for reducing the probability of server burrs based on tidal algorithm smoothing provided by an embodiment of the present invention;

[0021] Figure 2 This is an example of a fixed mapping relationship provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0023] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0024] like Figure 1 As shown, the method for reducing the probability of server glitch based on tidal algorithm smoothing provided in this embodiment includes the following steps:

[0025] Step S1, create a mongoDB table and set a reference time field, use the expiration reference time of the reference time field as the designated index field, and set a TTL index of the expiration time for the reference time field.

[0026] The mongoDB table stores the information of each created data. Before creating the table, you need to set the format of the base time field. The base time field is of date type. For example, the expiration base time is 1:00 on March 10, 2024, and the expiration time is a whole day, that is, N*24*3600 seconds, where N is the number of expiration days. Under normal circumstances, starting at 1:00, the data needs to be cleaned up after N*24*3600 seconds. Use the mongoDB createIndex({"expireBaseTime":1},{expireAfterSeconds:N*24*3600}) method to use the expiration base time expireBaseTime in the base time field as the specified index field, and set the expiration time TTL index for the base time field. In the above example, 3600 seconds is set, that is, 1 hour, and the expiration time is in seconds. In actual production scenarios, you need to calculate the expiration time and set it according to the actual situation. The expiration time is generally a whole day.

[0027] Step S2: According to the peak and off-peak period characteristics of the current business data, the peak period and the off-peak period are marked and fragmented with set time values, and then the fragments of the peak period and the off-peak period are mapped.

[0028] The peak and low peak of business data can be reflected by the CPU load. A high CPU load indicates that more normal business needs to be processed. Conversely, a high CPU load indicates that the CPU is relatively idle. The ideal state is to start a thread to clean up expired data when the CPU is idle to avoid processing expired data when the CPU is busy.

[0029] Therefore, this step needs to analyze the peak and off-peak time characteristics according to the processing characteristics of normal business data to determine the peak and off-peak time periods. Of course, the time periods other than the peak and off-peak time periods are off-peak time periods. Then, the shards are marked according to the set time value. In this embodiment, the shards are marked with unit hour time values. The specific time granularity can be set according to the actual situation.

[0030] For example, if 12 noon is the peak time for the current normal business, then the marked slice is 12, and similarly, 13 o'clock is marked as slice 13, and then the current business off-peak period is determined, and the off-peak period is also marked with the hourly time value. For example, if 0 o'clock in the morning is the off-peak time, it is marked as 0, and 1 o'clock in the morning is marked as 1. Finally, the peak period slices and the off-peak period slices are mapped. This embodiment does not limit the specific mapping relationship. For example, slice 12 and slice 0, slice 13 and slice 1 are a mapping relationship between a peak period and a off-peak period. If 10 o'clock is a flat period, then no mapping is required.

[0031] This embodiment is applied in a payment scenario, the peak time period and the off-peak time period are fixed, and the mapping relationship has a fixed mode and a dynamically adjusted mapping mode.

[0032] Fixed mapping means that each shard during peak hours is mapped to a fixed shard during off-peak hours. Figure 2 As shown in the mapping relationship table, the daily business peak time is 7:00-8:00, and the business off-peak time is 0:00-1:00. Each data of shard 7 is mapped to shard 0.

[0033] Dynamically adjust the mapping means that the mapping relationship is uncertain. In this embodiment, after configuring the mapping of peak time periods and off-peak time periods, set a data flow upper limit, such as 600 lines / second, and the data of normal peak time periods is 300 lines / second. If the data flow is abnormally high (more than 600 lines / second, and generally continuous), the number of off-peak time periods is used as a modulus, and the abnormally high data is mapped to each off-peak time period by taking the remainder per second. This can avoid the abnormally high data from being concentrated in one off-peak time period, thereby increasing the data dispersion effect.

[0034] Of course, taking into account the needs of different application scenarios, for application scenarios with uncertain peak and off-peak periods, the peak and off-peak periods can be confirmed through prediction models, and the mapping relationship can be tested using an automatic adjustment model. The details are not elaborated here.

[0035] Step S3: for the data to be stored, obtain the expiration reference time of each piece of data, and for the data whose expiration reference time falls into the peak period, adjust the expiration reference time to the corresponding off-peak period according to the set mapping relationship.

[0036] The business service calculates the base time of each piece of data to be stored, usually using the creation time of each piece of data as the expiration base time, and then adjusts the base time of the peak period data to the corresponding off-peak period time based on the mapping relationship set in step S2.

[0037] For example, a piece of data was generated at 2024-03-10 07:25:50, using Figure 2 In the fixed mapping relationship shown, shard 7 is mapped to shard 0, and the values ​​of minutes and seconds remain unchanged. Then the value of the expireBaseTime field in the corresponding mongoDB table is set to 2024-03-1100:25:50, which ensures that the data is valid within the specified time, and adjusts the data created during the peak period to be processed during the off-peak period, so as to break up the data expiration time, thereby avoiding the problem that a large amount of data expires at the same time, causing the mongoDB data cleaning thread to be occupied all the time, resulting in high CPU usage or mongoDB unavailability. Therefore, the CPU can handle normal business during peak hours, achieving CPU load balancing.

[0038] Step S4: For each piece of data, when the expiration reference time is reached, the data is deleted after the corresponding expiration time has passed.

[0039] After reaching the expiration benchmark time, the expiration time of the data is obtained according to the TTL index, and the expiration time is started, such as N*24*3600 seconds. After the timing reaches this execution time, the mongoDB database CPU creates a data cleaning thread to clean up the expired data. It should be noted that since the expiration time is all day, after N days, when cleaning up expired data, it is still during the off-peak period.

[0040] In summary, the present invention marks and segments peak time periods and off-peak time periods, sets a mapping relationship between peak time periods and off-peak time periods, and can reasonably break up the expiration time of expired data, change the execution time of mongoDB's active data cleaning thread, and thereby reduce the time occupied by mongoDB for data cleaning due to the expiration and invalidation of large quantities of data, thereby reducing server glitches and increasing server stability.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for reducing the probability of server glitches based on a tidal algorithm, characterized in that: The method comprises the following steps: Step S1, create a mongoDB table and set a reference time field, use the expiration reference time of the reference time field as the designated index field, and set a TTL index of the expiration time for the reference time field; Step S2: According to the peak and off-peak period characteristics of the current business data, the peak period and the off-peak period are marked and fragmented with set time values, and then the fragments of the peak period and the off-peak period are mapped; Step S3: for the data to be stored, obtain the expiration reference time of each piece of data, and for the data whose expiration reference time falls into the peak period, adjust the expiration reference time to the corresponding off-peak period according to the set mapping relationship; Step S4: For each piece of data, when the expiration reference time is reached, the data is deleted after the corresponding expiration time has passed.

2. The method for reducing the probability of server glitches based on tidal algorithm smoothing as claimed in claim 1, characterized in that: In step S2, the slices are marked with time values ​​in units of hours.

3. The method for reducing the probability of server glitches based on tidal algorithm smoothing as claimed in claim 2, characterized in that: The mapping relationship set in step S2 is a fixed mapping relationship, and each shard during a peak period is mapped to a fixed shard during a low-peak period.

4. The method for reducing the probability of server glitches based on tidal algorithm smoothing as claimed in claim 2, characterized in that: The mapping relationship set in step S2 is a dynamically adjusted mapping relationship, and the mapping relationship is set as follows: Set a data traffic upper limit and count the data traffic during peak hours by seconds. If the current peak time data traffic exceeds the upper limit, the current data is mapped to each off-peak time period modulo the number of off-peak time periods by taking the remainder per second.

5. The method for reducing the probability of server glitches based on tidal algorithm smoothing as described in any one of claims 1 to 4, characterized in that: Step S3 uses the creation time as the expiration reference time for each piece of data.

Images