Database large key scanning method and device, equipment, medium and program product

Abstract

The present disclosure provides a scanning method of a database large key, relates to the technical field of data processing, and can be applied to the financial field or other fields. The scanning method comprises the following steps: acquiring comprehensive load information, wherein the comprehensive load information comprises a plurality of continuous time intervals and comprehensive loads of a database in each time interval; determining a target time interval according to the comprehensive load information, wherein the target time interval comprises time intervals with a comprehensive load lower than a first target load in the plurality of continuous time intervals; and scanning a large key in the database when the target time interval overlaps with a current time. In the process of scanning the large key in the database, an initial scanning frequency is used in a starting stage, and the scanning frequency is gradually adjusted according to a current comprehensive load and a preset second target load in an intermediate stage until the comprehensive load of the database reaches an expected load. The present disclosure also provides a scanning device of a database large key, equipment, a storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to a method, apparatus, electronic device, storage medium, and program product for scanning database keys. Background Technology

[0002] Redis (Remote Dictionary Server) is a high-performance key-value in-memory database characterized by high performance, rich data types, and good scalability, and is widely used in the Internet field.

[0003] In Redis, if the number of files (fields) or the storage capacity of a key is too large, this key is called a big key. Deletion or data migration may cause service blockage, leading to access timeouts and affecting business processing. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a method, apparatus, electronic device, storage medium, and program product for scanning database keys.

[0005] According to a first aspect of this disclosure, a method for scanning large keys in a database is provided, comprising:

[0006] Obtain comprehensive load information, which includes multiple consecutive time intervals and the comprehensive load of the database in each time interval;

[0007] Based on the comprehensive load information, a target time interval is determined, which includes multiple consecutive time intervals in which the comprehensive load is lower than a first target load.

[0008] When the target time interval overlaps with the current time, scan the major keys in the database;

[0009] Specifically, when scanning the major keys in the database, an initial scanning frequency is used during the startup phase, and during the intermediate phase, the scanning frequency is gradually adjusted according to the current overall load and the preset second target load until the overall load of the database reaches the expected load.

[0010] According to embodiments of this disclosure, the time interval includes multiple sub-intervals, and obtaining comprehensive load information includes:

[0011] For at least one of the aforementioned time intervals

[0012] Based on network load information and processor load information, calculate the comprehensive load metric of the database on each of the multiple sub-intervals;

[0013] The overall load over the time interval is calculated based on the overall load metric values ​​over the multiple sub-intervals.

[0014] According to embodiments of this disclosure, the network load information includes network bandwidth and queries per second, the processor load information includes processor utilization, and the step of calculating a comprehensive load metric for the database on each of the plurality of sub-intervals based on the network load information and the processor load information includes:

[0015] For at least one of the said sub-intervals

[0016] Get the current processor utilization rate;

[0017] Obtain the ratio of the current network bandwidth to the maximum physical bandwidth to obtain the network bandwidth utilization rate;

[0018] The network traffic utilization rate is obtained by comparing the current query rate per second with the maximum query rate per second.

[0019] The comprehensive load metric for the sub-interval is calculated based on the current processor utilization, network bandwidth utilization, and network traffic utilization.

[0020] According to embodiments of this disclosure, the second target load includes a target value, and the expected load is greater than or equal to the target value; the step of gradually adjusting the scanning frequency based on the current overall load and the preset second target load until the overall load of the database reaches the expected load includes:

[0021] If the current overall load is less than the target value, then the increase in the scanning frequency is determined based on the difference between the current overall load and the target value;

[0022] The scanning frequency is increased according to the determined increase until the overall load of the database reaches the expected load.

[0023] According to embodiments of this disclosure, the second target load further includes a warning value, wherein the target value is lower than the warning value; the step of gradually adjusting the scanning frequency based on the current overall load and the preset second target load until the overall load of the database reaches the expected load further includes:

[0024] If the current overall load is greater than the warning value, the reduction in the scanning frequency is determined based on the ratio between the current overall load and the warning value.

[0025] The scanning frequency is reduced according to the determined reduction rate until the overall load of the database reaches the expected load;

[0026] According to embodiments of this disclosure, the increase is positively correlated with the difference; and / or, the decrease is positively correlated with the ratio.

[0027] A second aspect of this disclosure provides a database key scanning apparatus, comprising:

[0028] The acquisition module is used to acquire comprehensive load information, which includes multiple consecutive time intervals and the comprehensive load of the database in each time interval;

[0029] The processing module is configured to determine a target time interval based on the comprehensive load information, wherein the target time interval includes a number of consecutive time intervals in which the comprehensive load is lower than a first target load;

[0030] The scanning module is used to scan the major keys in the database when the target time interval overlaps with the current time.

[0031] Specifically, when scanning the major keys in the database, an initial scanning frequency is used during the startup phase, and during the intermediate phase, the scanning frequency is gradually adjusted according to the current overall load and the preset second target load until the overall load of the database reaches the expected load.

[0032] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the database key scanning method described above.

[0033] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the aforementioned database key scanning method.

[0034] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described database key scanning method.

[0035] The above one or more embodiments have the following advantages or benefits:

[0036] The database key scanning method according to the present disclosure monitors the historical overall load of the database and can automatically start key scanning when the load is lowest each day. At the same time, it adaptively adjusts the scanning frequency according to the current overall load and the preset second target load, which can improve the efficiency of key scanning without affecting normal business processing. Attached Figure Description

[0037] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0038] Figure 1 The illustrations depict application scenarios of database key scanning methods, apparatus, electronic devices, storage media, and program products according to embodiments of the present disclosure.

[0039] Figure 2 A flowchart illustrating a database key scanning method according to an embodiment of the present disclosure is shown schematically.

[0040] Figure 3 A flowchart illustrating the calculation of the overall load according to an embodiment of the present disclosure is shown schematically;

[0041] Figure 4 A flowchart illustrating the calculation of a comprehensive load metric value according to an embodiment of the present disclosure is shown schematically;

[0042] Figure 5A and Figure 5B A schematic diagram of a comprehensive load variation curve according to an embodiment of the present disclosure is shown;

[0043] Figure 6 One of the flowcharts illustrating the dynamic adjustment of the scan frequency according to an embodiment of the present disclosure is shown schematically;

[0044] Figure 7 A second flowchart illustrating the dynamic adjustment of the scanning frequency according to an embodiment of the present disclosure is shown schematically.

[0045] Figure 8 This schematic diagram illustrates a structural block diagram of a database key scanning apparatus according to an embodiment of the present disclosure;

[0046] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a database key scanning method according to an embodiment of the present disclosure. Detailed Implementation

[0047] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0048] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0049] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0050] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0051] It should be noted that the database key scanning method, apparatus, electronic device, storage medium, and program product provided in the embodiments of this disclosure relate to the field of data processing technology. The database key scanning method, apparatus, electronic device, storage medium, and program product provided in the embodiments of this disclosure can be applied to the financial field or any field other than the financial field. For example, the database key scanning method, apparatus, electronic device, storage medium, and program product provided in the embodiments of this disclosure can be applied to testing operations in the financial field. The embodiments of this disclosure do not limit the application field of the database key scanning method, apparatus, electronic device, storage medium, and program product.

[0052] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0053] In one example, the entire Redis database is scanned periodically to identify and alert users to large keys early. This is achieved by optimizing the data structure and splitting large keys to reduce the risk of blocking. This example typically employs two methods to scan for large keys: online mode and offline mode.

[0054] In online mode, Redis provides a command to scan large keys, but it only supports execution at a specific rate. This scanning method will preempt the processor (CPU) and network resources of the Redis server. In order to avoid affecting the processing of normal business, the scanning frequency cannot be set too high. This results in a long scanning time when facing Redis clusters with large storage or many nodes, which may take several hours.

[0055] In offline mode, Redis supports using RDB or AOF files as persistent storage, preventing complete data loss due to process crashes in abnormal situations. RDB, or Redis DataBase, saves a snapshot of Redis's full memory at a given moment in binary format to disk. AOF, or Append Only File, appends the command and its parameters to the end of the AOF file each time Redis executes a data modification command, storing it in plaintext. This allows for data recovery and is also easily readable, facilitating the analysis of historical operation records.

[0056] Offline analysis of RDB or AOF files can determine the presence of large keys. This method does not preempt processor and network resources on the Redis server, but it does preempt disk I / O resources, which affects master-slave synchronization and persistent file writing. Therefore, the scan speed cannot be set too high, and the aforementioned time-consuming problem persists.

[0057] Furthermore, some Redis clusters, designed with multiple replicas, already ensure fault domain isolation and high availability. In pursuit of high performance, they may disable persistent storage such as RDB and AOF, which renders offline mode unavailable.

[0058] In view of this, embodiments of the present disclosure provide a method for scanning large keys in a database, comprising: acquiring comprehensive load information, the comprehensive load information including multiple consecutive time intervals and the comprehensive load of the database in each time interval; determining a target time interval based on the comprehensive load information, the target time interval including time intervals in the multiple consecutive time intervals where the comprehensive load is lower than a first target load; scanning large keys in the database when the target time interval overlaps with the current time; wherein, when scanning large keys in the database, in the startup phase, an initial scanning frequency is used, and in the intermediate phase, the scanning frequency is gradually adjusted according to the current comprehensive load and a preset second target load until the comprehensive load of the database reaches the expected load.

[0059] The database key scanning method according to the embodiments of this disclosure can automatically start key scanning when the daily load is at its lowest by monitoring the overall load of the database. At the same time, it can adaptively adjust the scanning frequency according to the current overall load and the preset second target load, which can improve the efficiency of key scanning without affecting normal business processing.

[0060] Figure 1 The illustrations depict application scenarios of database key scanning methods, apparatus, electronic devices, storage media, and program products according to embodiments of the present disclosure.

[0061] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0062] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0063] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0064] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0065] It should be noted that the database key scanning method provided in this embodiment can generally be executed by server 105. Correspondingly, the database key scanning device provided in this embodiment can generally be located in server 105. The database key scanning method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the database key scanning device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0066] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0067] The following will be based on Figure 1 The described scene, through Figures 2-7 The database key scanning method of the disclosed embodiments is described in detail.

[0068] Figure 2 A flowchart illustrating a database key scanning method according to an embodiment of the present disclosure is shown schematically.

[0069] like Figure 2 As shown, the scanning method of this embodiment includes steps S210 to S230. It should be noted that, although Figure 2 The steps in the diagram are shown sequentially as indicated by the arrows; however, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0070] In step S210, comprehensive load information is obtained, which includes multiple consecutive time intervals and the comprehensive load of the database in each time interval.

[0071] In the embodiments of this disclosure, the database may specifically refer to a Reids database, and the overall load of the database may refer to comprehensive data formed by combining multiple load indicators of the database. For example, the overall load may include comprehensive data formed by combining load indicators such as processor load and network load. Optionally, multiple load indicators may be processed using methods such as weighted averaging to obtain the overall load. Here, unlike the "current overall load" mentioned below, the overall load in each time interval of the overall load information specifically refers to the historical overall load, that is, the overall load collected and recorded before the current moment, while the "current overall load" mentioned below specifically refers to the overall load collected in real time at the current moment.

[0072] In embodiments of this disclosure, time intervals can be set according to the frequency of large key scans required by the database. For example, large key scans can be performed on a "day" basis, meaning a large key scan is performed on the database once a day. In this case, a continuous time interval can refer to a continuous period of time within a day. For example, a day can be divided into 24 time intervals, each lasting one hour, so that multiple consecutive time intervals can cover the entire day.

[0073] It should be noted that the embodiments disclosed herein are not limited to the above-described division method. For example, a day can also be divided into 48 time intervals, each time interval lasting half an hour, etc.

[0074] In step S220, a target time interval is determined based on the comprehensive load information. The target time interval includes the time intervals in which the comprehensive load is lower than the first target load among multiple consecutive time intervals.

[0075] In embodiments of this disclosure, the overall load information includes the overall load of the database in each time interval, thereby determining how the overall load of the database changes over time. Combining this change with a first target load, a time interval where the overall load is favorable for scanning large keys at a higher scan frequency can be selected, and this time interval can then be used as the target time interval.

[0076] Optionally, the first target load can be a fixed value or a dynamic value. For example, the fixed value can be a lower overall load preset based on historical load data of the database; the dynamic value can be a lower value dynamically calculated based on multiple overall loads over multiple consecutive time intervals. For example, the dynamic value can be the lower overall load of the current day.

[0077] When the overall load of the database is lower than the first target load, it means that the overall load at this time is lower than the overall load of other time intervals on this day. In other words, when the overall load of the database is lower than the first target load, it means that the database is in a relatively idle period of the day. Therefore, the time interval to which the overall load belongs is the target time interval.

[0078] In step S230, when the target time interval overlaps with the current time, the major keys in the database are scanned. Specifically, during the scanning of major keys in the database, an initial scanning frequency is used in the initial phase, and in the intermediate phase, the scanning frequency is gradually adjusted based on the current overall load and the preset second target load until the overall load of the database reaches the expected load.

[0079] For example, if 1 a.m. is determined as the target time interval using the above method, in step S230, if the current time is also 1 a.m., scanning the major keys in the database can begin; if the current time is any other time, the database can enter a waiting state until the current time is 1 a.m. before scanning the major keys in the database.

[0080] As described above, the selected target time interval is a relatively idle period of the day for the database. Therefore, scanning the large keys in the database during this time can consume more server resources, allowing for a higher scanning frequency and thus improving scanning efficiency.

[0081] In embodiments of this disclosure, the initial scan frequency can be a low value to ensure that routine business processing is not affected when scanning large keys in the database begins. The second target load can refer to the value that the overall database load should not exceed when scanning large keys. The expected load can be less than or equal to the second target load. Thus, when the current overall database load is less than the second target load, the scan frequency can be gradually increased until the current overall database load reaches the expected load; once the overall database load is greater than or equal to the second target load, increasing the scan frequency can be stopped, or even appropriately reduced, thereby maintaining the current overall database load at the expected load.

[0082] The database key scanning method according to the present disclosure monitors the historical overall load of the database and can automatically start key scanning when the load is lowest each day. At the same time, it adaptively adjusts the scanning frequency according to the current overall load and the preset second target load, which can improve the efficiency of key scanning without affecting normal business processing.

[0083] The following is combined Figures 2 to 7The scanning method of this disclosure embodiment will be further described. For clarity, the multiple time intervals mentioned below refer to multiple time intervals obtained by dividing a day into equal parts. For example, dividing a day into 24 equal parts yields 24 consecutive time intervals; similarly, dividing a day into 48 equal parts yields 48 consecutive time intervals.

[0084] Figure 3 A flowchart illustrating the calculation of the comprehensive load according to an embodiment of the present disclosure is shown schematically.

[0085] The following example uses the Nth time interval out of multiple time intervals, combined with... Figure 3 As shown, the process of calculating the comprehensive load in the embodiments of this disclosure is explained, where N is a positive integer.

[0086] It should be noted that the Nth time interval can refer to any one of a series of consecutive time intervals. That is, in the embodiments of this disclosure, steps S211 and S212 can be executed for any one of these multiple time intervals. Furthermore, steps S211 and S212 can be executed for each of these multiple time intervals.

[0087] In some specific embodiments, the time interval includes multiple sub-intervals, and step S210 includes steps S211 and S212.

[0088] In step S211, based on network load information and processor load information, the comprehensive load metric (CLM) of the database is calculated for each of the multiple sub-intervals.

[0089] In the embodiments of this disclosure, both network load information and processor load information refer to network load information and processor load information generated in the Nth time interval. The network load information may include information reflecting network usage, such as network bandwidth and queries per second. Similarly, the processor load information may include information reflecting processor usage, such as processor utilization.

[0090] Figure 4 A flowchart illustrating the calculation of a comprehensive load metric value according to an embodiment of the present disclosure is shown schematically.

[0091] In the embodiments of this disclosure, the Nth time interval can be divided into M sub-intervals, where M is a positive integer. The following example demonstrates the calculation of the comprehensive load metric for the Mth sub-interval, combined with... Figure 4 The process of calculating the comprehensive load metric value in an embodiment of this disclosure is described as shown below.

[0092] In some specific embodiments, step S211 includes steps S2111 to S2114.

[0093] In step S2111, the current processor utilization rate is obtained.

[0094] Processor utilization can be obtained directly from the running parameters in the database. The obtained processor utilization is denoted as Pc (0≤Pc≤1.0).

[0095] In step S2112, the ratio of the current network bandwidth to the maximum physical bandwidth is obtained to determine the network bandwidth utilization rate.

[0096] Network bandwidth utilization is denoted as Pn, and can be calculated using the following formula:

[0097]

[0098] Where B1 represents the network bandwidth at the current moment, and B2 represents the maximum physical bandwidth.

[0099] In step S2113, the ratio of the current query rate per second to the maximum query rate per second is obtained to obtain the network traffic utilization rate.

[0100] In some specific embodiments, the network traffic utilization rate Pq can be calculated using the following formula:

[0101]

[0102] Where Q1 represents the query rate per second at the current moment, and Q2 represents the maximum query rate per second.

[0103] In step S2114, the comprehensive load metric value on the sub-interval is calculated based on the current processor utilization, network bandwidth utilization, and network traffic utilization.

[0104] In embodiments of this disclosure, the Comprehensive Load Metric (CLM) can be calculated by averaging. Specifically, the CLM can be calculated using the following formula:

[0105]

[0106] For example, for a gigabit network (i.e., a maximum physical bandwidth of 1000 Mbps), if the maximum query rate per second is measured to be 50,000, and in the Mth sub-interval, the processor utilization Pc is 50%, the network bandwidth is 500 Mbps, and the query rate per second at the current moment is 10,000, then the overall load metric CLM = 0.4. If the processor utilization Pc is 80%, the network bandwidth is 900 Mbps, and the query rate per second at the current moment is 50,000, then the overall load metric CLM = 0.9.

[0107] In step S212, the comprehensive load over the time interval is calculated based on the comprehensive load metric values ​​over multiple sub-intervals.

[0108] In the embodiments of this disclosure, the above-mentioned comprehensive load metric value CLM can be obtained by using a timer with the length of the sub-interval as the acquisition period, and the comprehensive load of each time interval can be calculated.

[0109] For example, two timers can be configured, namely timer T1 and timer T2.

[0110] The acquisition period can be configured according to the length of the sub-interval using timer T1 to obtain the comprehensive load metric CLM mentioned above.

[0111] Optionally, the sampling period of timer T1 can be the same as the length of the sub-interval. For example, the sampling period Ta of timer T1 can be set to a range of 1 second to 3 seconds.

[0112] Using timer T2, the acquisition period can be configured according to the length of multiple consecutive sub-intervals to obtain multiple adjacent composite load metric (CLM) values ​​and calculate their average (hereinafter also referred to as the composite load metric average). Then, based on this composite load metric average, the composite load of the Nth time interval can be determined. For example, based on the obtained composite load metric average, the average can be further calculated to obtain the composite load of the Nth time interval; or, the extreme value (e.g., the lowest or highest value) among the obtained composite load metric averages can be used as the composite load of the Nth time interval.

[0113] Optionally, the acquisition period of timer T2 can be greater than or equal to the length of multiple consecutive sub-intervals. For example, the acquisition period Tb of timer T2 can be set to a range of 1 minute to 5 minutes.

[0114] In some specific embodiments, step S210 further includes step S213.

[0115] In step S213, a comprehensive load change curve is generated based on the comprehensive load over multiple time intervals to obtain comprehensive load information.

[0116] For example, in the embodiments of this disclosure, a timer T3 can also be configured. The acquisition period can be configured according to the length of the time interval through the timer T3 to obtain the comprehensive load of each time interval, thereby generating a comprehensive load change curve. Through the comprehensive load change curve, the comprehensive load that matches the first target load can be found.

[0117] Optionally, the acquisition period of timer T3 can be the same as the length of the time interval. For example, the acquisition period Tc of timer T3 can be set to a range of 30 minutes to 1 hour, that is, dividing a day into 24 time intervals or dividing a day into 48 time intervals.

[0118] In some specific embodiments, if the combined load values ​​of multiple time intervals are the same on a given day, the earliest time interval on that day is taken as the target time interval.

[0119] Figure 5A and Figure 5B A schematic diagram of a comprehensive load variation curve according to an embodiment of the present disclosure is shown, wherein, Figure 5A and Figure 5B The curves shown represent the changes in the overall load over a 24-hour period, with the horizontal axis representing time and the vertical axis representing the value of the overall load.

[0120] like Figure 5A As shown, the overall load is lowest at 1 AM, and the time is the earliest. Therefore, around 1 AM is the target time range, meaning that the large keys should be scanned around 1 AM. Figure 5B As shown, there are batch synchronization and backup operations between 1 and 2 a.m., so the overall load value is high during this time period. This time period should be avoided. The overall load is lowest at 3 a.m. and the time is the earliest. Therefore, around 3 a.m. is the target time period, that is, scan the big keys around 3 a.m.

[0121] In some specific embodiments, large keys can be scanned by iterating through the number of elements or storage capacity of each key in the database. Taking storage capacity as an example, when the storage capacity of a certain key exceeds a preset threshold, the key is confirmed as a large key, its name is recorded, and an alert log is generated.

[0122] Meanwhile, during the scanning process, the current overall load can be obtained in real time. For example, the current overall load metric CLM can be obtained in real time through timer T1. Combined with the second target load, the scanning frequency can be adaptively adjusted to ensure high scanning efficiency without affecting normal business processing.

[0123] Figure 6 One of the flowcharts illustrating the dynamic adjustment of the scan frequency according to an embodiment of the present disclosure is shown schematically.

[0124] In some specific embodiments, the second target load includes a target value, whereby the expected load is greater than or equal to the target value. For example... Figure 6 As shown, step S230 includes steps S231 and S232.

[0125] In step S231, if the current overall load is less than the target value, the increase in scanning frequency is determined based on the difference between the current overall load and the target value. Optionally, the increase is positively correlated with the difference; that is, the larger the difference, the larger the increase, thereby rapidly increasing the scanning frequency when the overall load is low.

[0126] The difference between the current overall load and the target value specifically refers to the difference obtained by subtracting the current overall load from the target value and minuending it.

[0127] In the embodiments of this disclosure, timer T1 collects the comprehensive load metric value according to a preset period. Therefore, the comprehensive load metric value collected by timer T1 that is closest to the current time can be obtained as the current comprehensive load.

[0128] The target value can include: the optimal overall load on the database when scanning large keys, without affecting the database's regular business operations.

[0129] In the embodiments of this disclosure, the increase in scanning frequency can be dynamically adjusted based on the difference between the current comprehensive load (i.e., the most recently collected comprehensive load metric) and the target value. For example, when the difference is positive and large, the increase in scanning frequency can be large, and when the difference is positive and small, the increase in scanning frequency can be small.

[0130] In step S232, the scanning frequency is increased according to the determined increase until the overall load of the database reaches the expected load.

[0131] In the embodiments of this disclosure, the expected load can be a numerical range, with its lower limit being the same as the target value. Thus, when the current overall load is significantly lower than the expected load, the difference between the current overall load and the target value is large. In this case, the scan frequency can be increased rapidly to bring the current overall load closer to the expected load as quickly as possible. When the current overall load is close to the lower limit of the expected load, the difference between the current overall load and the target value is small. In this case, the increase in the scan frequency can be slowed down, allowing the current overall load to gradually approach the expected load, which is beneficial for maintaining database stability.

[0132] Optionally, if the difference between the current overall load and the target value is 0 or negative, it means that the current overall load is equal to or greater than the target value. In this case, the increment can be set to 0, thereby stopping the increase in the scanning frequency, so that the current overall load is kept as close as possible to the expected load.

[0133] In some specific embodiments, the increase P1 can be calculated using the following formula:

[0134]

[0135] Where CLMopt is the target value and CLMopt is the current total load.

[0136] Let the current scanning frequency be F, then the adjusted scanning frequency F′ can be calculated using the following formula:

[0137] F′=(P1*Δp+1)*F

[0138] Wherein, Δp is the reference amplitude. Optionally, the magnitude of the reference increase Δp can be determined according to actual needs, but its maximum value should be 0.5. Optionally, the reference increase Δp = 0.05.

[0139] For example, assuming the current overall load CLMcur = 0.1 and the target value CLMopt = 0.5, then the increase P = 1.2, that is, the adjusted scan frequency F′ is 1.2 times the current scan frequency F.

[0140] Assuming the current overall load CLMcur = 0.3 and the target value CLMopt = 0.5, then the increase P = i.033, that is, the adjusted scan frequency F′ is 1.033 times the current scan frequency F.

[0141] Therefore, it can be seen that when the current overall load CLMcur is much smaller than the target value CLMopt, the scanning frequency increases by a higher factor to increase the scanning frequency as quickly as possible. As the overall load CLMcur gets closer and closer to the target value CLMopt, the increase in scanning frequency also becomes smaller and smaller, thus preventing jitter.

[0142] Figure 7 A second flowchart illustrating the dynamic adjustment of the scanning frequency according to an embodiment of the present disclosure is shown schematically.

[0143] In some specific embodiments, the second target load also includes a warning value, where the target value is lower than the warning value. Optionally, the expected load is less than or equal to the warning value. Figure 7 As shown, step S230 also includes steps S233 and S234.

[0144] In step S233, if the current overall load is greater than the warning value, the reduction in scanning frequency is determined based on the ratio between the current overall load and the warning value. Optionally, the reduction is positively correlated with the ratio; that is, the larger the ratio, the larger the reduction, thereby rapidly reducing the scanning frequency when the overall load is high.

[0145] The ratio between the current overall load and the warning value specifically refers to the ratio obtained by using the current overall load as the numerator and the warning value as the denominator. The warning value can include: the maximum value that the overall load of the database can reach when scanning large keys, that is, the value that the overall load of the database should not exceed when scanning large keys.

[0146] In embodiments of this disclosure, the reduction rate of the scanning frequency can be dynamically adjusted based on the ratio between the current comprehensive load (i.e., the most recently collected comprehensive load metric) and the warning value. For example, when the ratio is positive and large, the reduction rate of the scanning frequency can be large, and when the ratio is positive and small, the reduction rate of the scanning frequency can be small.

[0147] In step S234, the scanning frequency is reduced according to the determined reduction rate until the overall load of the database reaches the expected load.

[0148] As mentioned earlier, the expected load can be a numerical range, with the lower limit being the target value mentioned above, and the lower limit can also be the warning value. Thus, if the current overall load is higher than the warning value, the current overall load is also higher than the expected load. If the current overall load is significantly higher than the warning value, then the ratio of the current overall load to the warning value is large. In this case, the scanning frequency can be quickly reduced, thereby rapidly reducing the current overall load to free up resources for handling routine business and preventing the processing of routine business from being affected.

[0149] In some specific embodiments, the reduction P2 can be calculated in the following way:

[0150]

[0151] CLMwarn is the warning value.

[0152] Let the current scanning frequency be F, the adjusted scanning frequency F″ can be calculated using the following formula:

[0153] F″=P2*F

[0154] For example, assuming the current overall load CLMcur = 0.6 and the warning value CLMwarn = 0.5, then the reduction P2 = 1 / 1.44, that is, the adjusted scan frequency F′ is 1 / 1.44 of the current scan frequency F.

[0155] Assuming the current overall load CLMcur = 0.9 and the warning value CLMwarn = 0.5, then the reduction P2 = 1 / 3.24, that is, the adjusted scan frequency F′ is 1 / 3.24 of the current scan frequency F.

[0156] Therefore, when the current overall load CLMcur is greater than the warning value CLMwarn, the larger the overall load CLMcur is, the higher the multiple by which the scanning frequency is reduced, thus significantly reducing the scanning frequency to free up system resources for normal business processing.

[0157] Optionally, when the above ratio is 0 or negative, it indicates that the current overall load is equal to or less than the warning value. In this case, the current overall load can be kept unchanged so that the current overall load remains at the expected load.

[0158] Based on the above-described method for scanning large database keys, this disclosure also provides a device for scanning large database keys. The following will be combined with... Figure 8 The device is described in detail.

[0159] Figure 8 A schematic block diagram of a database key scanning apparatus according to an embodiment of the present disclosure is shown.

[0160] like Figure 8 As shown, the database key scanning device 800 of this embodiment includes an acquisition module 810, a processing module 820, and a scanning module 830.

[0161] The acquisition module 810 is used to acquire comprehensive load information, which includes multiple consecutive time intervals and the comprehensive load of the database in each time interval. In one embodiment, the acquisition module 810 can be used to execute step S210 described above, which will not be repeated here.

[0162] The processing module 820 is used to determine a target time interval based on the comprehensive load information. The target time interval includes multiple consecutive time intervals in which the comprehensive load is lower than a first target load. In one embodiment, the processing module 820 can be used to execute step S220 described above, which will not be repeated here.

[0163] The scanning module 830 is used to scan the large keys in the database when the target time interval overlaps with the current time. In one embodiment, the scanning module 830 can be used to perform step S230 described above, which will not be repeated here.

[0164] Specifically, when scanning the major keys in the database, an initial scanning frequency is used during the startup phase, and during the intermediate phase, the scanning frequency is gradually adjusted according to the current overall load and the preset second target load until the overall load of the database reaches the expected load.

[0165] The database key scanning device according to the embodiments of this disclosure can automatically start key scanning when the load is lowest each day by monitoring the historical comprehensive load of the database. At the same time, it can adaptively adjust the scanning frequency according to the current comprehensive load and the preset second target load, which can improve the efficiency of key scanning without affecting normal business processing.

[0166] According to embodiments of this disclosure, any plurality of modules among the acquisition module 810, processing module 820, and scanning module 830 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 810, processing module 820, and scanning module 830 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module 810, processing module 820, and scanning module 830 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0167] In some specific embodiments, the time interval includes multiple sub-intervals, and the acquisition module 810 is specifically used to perform the following steps:

[0168] For at least one time interval

[0169] Based on network load information and processor load information, calculate the comprehensive load metric for the database in each of the multiple sub-intervals.

[0170] The overall load over the time interval is calculated based on the overall load metric values ​​across multiple sub-intervals.

[0171] In some specific embodiments, network load information includes network bandwidth and queries per second, and processor load information includes processor utilization. The acquisition module 810 is specifically used to perform the following steps:

[0172] For at least one subinterval,

[0173] Get the current processor usage.

[0174] Obtain the ratio of the current network bandwidth to the maximum physical bandwidth to get the network bandwidth utilization rate.

[0175] Get the ratio of the current queries per second to the maximum queries per second to obtain the network traffic utilization.

[0176] Calculate the overall load metric for the sub-interval based on the current processor utilization, network bandwidth utilization, and network traffic utilization.

[0177] In some specific embodiments, the second target load includes a target value, whereby the expected load is greater than or equal to the target value. The scanning module 830 is specifically used to perform the following steps:

[0178] If the current overall load is less than the target value, the increase in the scanning frequency is determined based on the difference between the current overall load and the target value.

[0179] Increase the scanning frequency according to the determined increase until the overall database load reaches the expected load.

[0180] In some specific embodiments, the second target load also includes a warning value, where the target value is lower than the warning value. The scanning module 830 is further configured to perform the following steps:

[0181] If the current overall load is greater than the warning value, the reduction in scanning frequency is determined based on the ratio between the current overall load and the warning value.

[0182] Reduce the scanning frequency according to the determined reduction rate until the overall database load reaches the expected load.

[0183] In some specific embodiments, the increase is positively correlated with the difference. And / or, the decrease is positively correlated with the ratio.

[0184] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a database key scanning method according to an embodiment of the present disclosure.

[0185] like Figure 9As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0186] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0187] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0188] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the database key scanning method according to embodiments of this disclosure.

[0189] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0190] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the database key scanning method provided in embodiments of this disclosure.

[0191] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0192] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0193] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0194] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0195] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0196] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0197] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for scanning large keys in a database, characterized in that, include: Obtain comprehensive load information, which includes multiple consecutive time intervals and the comprehensive load of the database in each time interval; Based on the comprehensive load information, a target time interval is determined, which includes multiple consecutive time intervals in which the comprehensive load is lower than a first target load. When the target time interval overlaps with the current time, scan the large keys in the database; Specifically, when scanning the major keys in the database, an initial scanning frequency is used during the startup phase, and during the intermediate phase, the scanning frequency is gradually adjusted according to the current overall load and the preset second target load until the overall load of the database reaches the expected load; the expected load is less than or equal to the second target load. The time interval includes multiple sub-intervals, and the acquisition of comprehensive load information includes: For at least one of the aforementioned time intervals Based on network load information and processor load information, calculate the comprehensive load metric of the database on each of the multiple sub-intervals; The overall load over the time interval is calculated based on the overall load metric values ​​over the multiple sub-intervals; The second target load includes a target value, and the expected load is greater than or equal to the target value; the step of gradually adjusting the scanning frequency based on the current overall load and the preset second target load until the overall load of the database reaches the expected load includes: If the current overall load is less than the target value, the increase in the scanning frequency is determined based on the difference between the current overall load and the target value; the increase is positively correlated with the difference. The scanning frequency is increased according to the determined increase until the overall load of the database reaches the expected load.

2. The scanning method according to claim 1, characterized in that, The network load information includes network bandwidth and queries per second, and the processor load information includes processor utilization. The step of calculating a comprehensive load metric for the database on each of the multiple sub-intervals based on the network load information and processor load information includes: For at least one of the said sub-intervals Get the current processor utilization rate; Obtain the ratio of the current network bandwidth to the maximum physical bandwidth to obtain the network bandwidth utilization rate; The network traffic utilization rate is obtained by comparing the current query rate per second with the maximum query rate per second. The comprehensive load metric for the sub-interval is calculated based on the current processor utilization, network bandwidth utilization, and network traffic utilization.

3. The scanning method according to claim 1, characterized in that, The second target load also includes a warning value, wherein the target value is lower than the warning value; the step of gradually adjusting the scanning frequency based on the current overall load and the preset second target load until the overall load of the database reaches the expected load further includes: If the current overall load is greater than the warning value, the reduction in the scanning frequency is determined based on the ratio between the current overall load and the warning value. The scanning frequency is reduced according to the determined reduction rate until the overall load of the database reaches the expected load.

4. The scanning method according to claim 3, characterized in that, The increase is positively correlated with the difference; and / or, the decrease is positively correlated with the ratio.

5. A database key scanning device, characterized in that, include: The acquisition module is used to acquire comprehensive load information, which includes multiple consecutive time intervals and the comprehensive load of the database in each time interval; the time interval includes multiple sub-intervals. The processing module is configured to determine a target time interval based on the comprehensive load information, wherein the target time interval includes a number of consecutive time intervals in which the comprehensive load is lower than a first target load; The scanning module is used to scan the major keys in the database when the target time interval overlaps with the current time. Specifically, when scanning the major keys in the database, an initial scanning frequency is used during the startup phase, and during the intermediate phase, the scanning frequency is gradually adjusted according to the current overall load and the preset second target load until the overall load of the database reaches the expected load; the expected load is less than or equal to the second target load. The second target load includes a target value, and the expected load is greater than or equal to the target value; the step of gradually adjusting the scanning frequency based on the current overall load and the preset second target load until the overall load of the database reaches the expected load includes: If the current overall load is less than the target value, then the increase in the scanning frequency is determined based on the difference between the current overall load and the target value; the increase is positively correlated with the difference; the scanning frequency is increased according to the determined increase until the overall load of the database reaches the expected load; The acquisition module is further configured to, for at least one of the time intervals, calculate a comprehensive load metric value of the database on each of the plurality of sub-intervals based on network load information and processor load information; and calculate the comprehensive load on the time interval based on the comprehensive load metric values ​​on the plurality of sub-intervals.

6. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the scanning method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to perform the scanning method according to any one of claims 1 to 4.

8. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the scanning method according to any one of claims 1 to 4.

Images