A method for quickly obtaining topic lists in large-scale Kafka clusters based on distributed cache
By regularly obtaining and caching the topic metadata of Kafka clusters, adjusting the data structure and backup delay of the distributed cache system, the problems of low Topic list acquisition rate and insufficient stability in large-scale Kafka clusters are solved, and more efficient and stable are achieved Get the topic list.
Patent Information
- Application Number
- CN202410659442.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-27
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-05-27
AI Technical Summary
In large-scale Kafka clusters, the Topic list acquisition rate is low and the stability is insufficient, resulting in data processing lag. The existing technology's distributed cache data structure types are incomplete and cannot effectively solve this problem.
It regularly obtains real-time topic metadata of Kafka cluster and caches it into a distributed cache system using the data structure of key-value pairs. Through verification and backup of topic information call requests, the metadata update method and cache data structure type are adjusted. Adjust the backup delay time according to the fluctuation amplitude and call rate of the generation time to achieve rapid acquisition and stability improvement of the topic list.
Through the multi-step acquisition and caching mechanism, the storage effectiveness of topic list information is improved, the probability of lag is reduced, the comprehensiveness and stability of cache is enhanced, the efficiency and stability of topic list acquisition is improved, and through encryption and Adjust the backup delay time, improving information security and data recovery capabilities.
Smart Images

Figure CN118612298B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cache technology, and in particular to a method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache. Background Art
[0002] As data scale continues to expand and processing requirements increase, Kafka is widely used as a high-performance, distributed stream processing platform. However, in a large-scale Kafka cluster, the number of topics is huge, and how to quickly obtain the topic list has become an important technical performance issue.
[0003] Chinese Patent Publication No.: CN117499410A discloses a Kafka cluster and its message synchronization method, device, electronic device, and storage medium, the method comprising: obtaining a topic filter list saved in the target topic of the source Kafka cluster; wherein the source Kafka cluster includes the target topic and at least one source topic, and the topic filter list includes the identifier of the source topic that is allowed to be synchronized and / or the identifier of the source topic that is not allowed to be synchronized; according to the topic filter list, the identifier of the source topic that is allowed to be synchronized is obtained from the identifier of the source topic in the source Kafka cluster and / or the identifier of the source topic that is not allowed to be synchronized is filtered out to obtain the identifier of the source topic to be synchronized; the message in the source topic corresponding to the identifier of the source topic to be synchronized is synchronized to the target Kafka cluster, so as to back up the message in the source topic to be synchronized to the Kafka cluster. According to the embodiment of the present disclosure, the problem of interruption of the synchronization channel caused by restarting the synchronization component when the synchronization component configuration is changed can be solved. It can be seen that the Kafka cluster and its message synchronization method, device, electronic device, and storage medium have problems such as incomplete data structure types of distributed cache, easy jams when processing Kafka cluster data, low topic list acquisition rate, and insufficient stability. Summary of the invention
[0004] To this end, the present invention provides a method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache, so as to overcome the problems in the prior art such as incomplete data structure types of distributed cache, easy jamming when processing Kafka cluster data, low topic list acquisition rate and insufficient stability.
[0005] To achieve the above-mentioned purpose, the present invention provides a method for quickly obtaining a large-scale Kafka cluster Topic list based on distributed cache, comprising: step S1, regularly obtaining real-time topic metadata in the Kafka cluster; step S2, caching the topic metadata in a distributed cache system using a key-value pair data structure type; step S3, sending a topic information call request to the distributed cache system, the distributed cache system verifies the topic information call request and caches and backs up the topic information call request and the topic information respectively; step S4, when the topic information call request passes the verification, the distributed cache system queries the corresponding topic metadata according to the topic information, and sends the topic metadata to the distributed cache system. Sent to the calling requester to generate a topic list; step S5, obtaining the generation time of several topic lists, and judging whether the stability of topic list acquisition meets the requirements according to the fluctuation range of the generation time of the topic lists; step S6, when it is judged that it does not meet the requirements, adjusting the update method of metadata, or adjusting the number of cached data structure types according to the linear fit between the byte amount of the value in the key value and the cache delay time, or initially adjusting the backup delay time according to the metadata call rate; step S7, when completing the initial adjustment of the delay time of the backup, performing a secondary adjustment on the delay time of the backup according to the data recovery time of the distributed cache system; wherein, the metadata update method includes a timed update method and an event-triggered update method.
[0006] Furthermore, the subject information includes a subject name and subject prefix information.
[0007] Furthermore, in the step S3, before sending the subject information calling request, the subject information calling request is also compressed and encrypted.
[0008] Further, in step S6, if the fluctuation range of the generation time of the topic list is within the preset first fluctuation range condition, the preset second fluctuation range condition, or the preset third fluctuation range condition, it is determined that the stability of the topic list acquisition does not meet the requirements, wherein,
[0009] If the fluctuation range of the generation time of the topic list only meets the preset first fluctuation range condition, it is preliminarily determined that the comprehensiveness of the cache does not meet the requirements, and the comprehensiveness of the cache is secondarily determined based on the linear fit between the byte amount of the value in the key value and the cache delay time;
[0010] If the fluctuation range of the generation time of the topic list only meets the preset second fluctuation range condition, it is preliminarily determined that the stability of the distributed cache system does not meet the requirements, and the stability of the distributed cache system is secondarily determined according to the call rate of the metadata;
[0011] If the fluctuation range of the generation time of the topic list only meets the preset third fluctuation range condition, it is determined to adjust the update mode of the metadata from the scheduled update mode to the scheduled update plus event-triggered update mode;
[0012] If the fluctuation range of the generation time of the topic list is within the preset fourth fluctuation range condition, it is determined that the stability of the topic list acquisition meets the requirements.
[0013] Furthermore, the preset first fluctuation range condition is that the fluctuation range of the generation time of the topic list is greater than the preset first fluctuation range and less than or equal to the preset second fluctuation range;
[0014] The preset second fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is greater than the preset second fluctuation amplitude and less than or equal to the preset third fluctuation amplitude;
[0015] The preset third fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is greater than the preset third fluctuation amplitude; the preset fourth fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is less than or equal to the preset first fluctuation amplitude.
[0016] Further, if the linear fit between the byte amount of the value in the key value and the cache delay duration is less than a preset fit, it is determined that the comprehensiveness of the cache does not meet the requirements, and the number of cached data structure types is increased;
[0017] The increase range of the number of cached data structure types is determined by the difference between the preset fitness and the linear fitness of the byte quantity of the value in the key value and the cache delay duration.
[0018] Furthermore, if the call rate of the metadata is less than the preset call rate, it is determined that the stability of the distributed cache system does not meet the requirements, and the initial adjustment process of increasing the delay time of the backup is performed.
[0019] The increase of the backup delay time is determined by the difference between the preset call rate and the metadata call rate; the backup delay time is the interval between the backup time and the cache time of the distributed cache system;
[0020] The calculation formula of the metadata call rate is:
[0021] Q=E / T
[0022] Among them, Q is the call rate of metadata, E is the number of bytes of metadata called in a single call cycle, and T is the duration of a single call cycle.
[0023] Furthermore, the timed update method is to update the metadata according to a predetermined period; the event-triggered update method is to update the metadata when a change occurs in the subject metadata.
[0024] Furthermore, the calculation formula for the fluctuation range of the generation time of the topic list is:
[0025] U=Umax-Umin
[0026] Among them, U is the fluctuation range of the generation time of the topic list, Umax is the maximum generation time among several topic lists, and Umin is the minimum generation time among several topic lists; the generation time of the topic list is the time taken by the requesting party from issuing a topic information call request to generating the topic list.
[0027] Further, the step S7 comprises:
[0028] Step S71, when the initial adjustment of the delay time for the backup is completed, the data recovery time of the distributed cache system is obtained;
[0029] Step S72, determining whether the data recovery performance meets the requirements according to the data recovery time of the distributed cache system;
[0030] Step S73, if it is determined that the data recovery performance does not meet the requirements, a secondary adjustment process is performed to reduce the increased backup delay time;
[0031] Among them, if the data recovery time of the distributed cache system is longer than the preset recovery time, it is determined that the data recovery performance does not meet the requirements; the reduction extent of the increased backup delay time is determined by the difference between the data recovery time of the distributed cache system and the preset recovery time.
[0032] Compared with the prior art, the beneficial effects of the present invention are that, through the multiple acquisition steps set, the method of the present invention obtains the latest topic metadata in the Kafka cluster by utilizing the metadata management mechanism of the Kafka cluster, and caches the obtained topic metadata in the distributed cache system. By using the distributed cache system, the storage effectiveness of the topic list information is improved; by adopting the data cache and backup mechanism and adjusting the backup delay time in the backup process, the conflict with the cache process in the backup process is avoided, thereby reducing the probability of jamming in the cache process and the metadata call process; by adjusting the number of cached data structure types according to the linear fit between the byte amount of the value in the key value and the cache delay time, the comprehensiveness of the cache is improved, and the lack of cache comprehensiveness caused by a single data structure and the impact on stability are avoided; by adjusting the metadata update method, the timeliness and effectiveness of the metadata update are improved, and the efficiency and stability of the acquisition of the topic list are improved.
[0033] Furthermore, the method of the present invention improves the encryption level of the subject information call request by compressing and encrypting the subject information call request, improves the security of the information, and thus improves the stability of obtaining the subject list.
[0034] Furthermore, the method of the present invention increases the backup delay time by setting a preset call rate when it is determined for the second time that the stability of the distributed cache system does not meet the requirements, that is, the interval time between the backup time and the cache time of the distributed cache system is increased. The increase in the interval time between the backup time and the cache time reduces the running memory pressure of the distributed cache system when caching data, and reduces the usage of backup space in the case of a large number of short-term calls, thereby increasing the calling efficiency of the calling process, avoiding invalid space usage, and improving the efficiency of obtaining the topic list.
[0035] Furthermore, the method of the present invention sets a preset recovery duration. After the backup delay duration is initially adjusted, that is, the backup delay duration is increased. If the increase is too large, the amount of backup data may be insufficient, thereby causing a decrease in data recovery capability in the event of a data disaster or emergency. Therefore, by making a secondary adjustment to the backup delay duration, a balance is achieved between the comprehensive performance improvement of the cache and backup process and the data recovery capability, thereby further improving the efficiency and stability of obtaining the subject list. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is an overall flow chart of a method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache according to an embodiment of the present invention;
[0037] Figure 2This is a specific flow chart of step S6 of the method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache in an embodiment of the present invention;
[0038] Figure 3 This is a specific flow chart of step S7 of the method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache in an embodiment of the present invention. DETAILED DESCRIPTION
[0039] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0040] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.
[0041] See also Figure 1 and Figure 2 As shown, Figure 1 The flowchart of the method for quickly obtaining a large-scale Kafka cluster Topic list based on distributed cache according to an embodiment of the present invention, the specific flowchart of step S6, and the specific flowchart of step S7 are shown below. The method for quickly obtaining a large-scale Kafka cluster Topic list based on distributed cache according to this embodiment includes:
[0042] Step S1, regularly obtain real-time topic metadata in the Kafka cluster;
[0043] Step S2, caching the subject metadata into a distributed cache system using a key-value pair data structure type;
[0044] Step S3, sending a subject information call request to the distributed cache system, the distributed cache system verifies the subject information call request and caches and backs up the subject information call request and the subject information respectively;
[0045] Step S4, when the subject information call request passes the verification, the distributed cache system queries the corresponding subject metadata according to the subject information, and sends the subject metadata to the call requester to generate a subject list;
[0046] Step S5, obtaining the generation time of several topic lists, and judging whether the stability of topic list acquisition meets the requirements according to the fluctuation range of the generation time of the topic lists;
[0047] Step S6, when it is determined that the requirements are not met, the updating method of the metadata is adjusted, or the number of cached data structure types is adjusted according to the linear fit between the byte amount of the value in the key value and the cache delay time, or the backup delay time is initially adjusted according to the metadata call rate;
[0048] Step S7, when the initial adjustment of the delay time of the backup is completed, the delay time of the backup is secondarily adjusted according to the data recovery time of the distributed cache system;
[0049] The metadata update method includes a timed update method and an event-triggered update method.
[0050] Specifically, the key of the key value is the subject name, and the value is the metadata information of the subject.
[0051] Specifically, metadata mainly refers to information about topics, partitions, replicas, and Kafka servers.
[0052] Specifically, the information of the Kafka server includes log information, performance indicators, and configuration information.
[0053] Specifically, the log information includes the log size of each partition, the start and end offsets of the log, and the compression parameters of the log.
[0054] Specifically, performance metrics include the number of messages processed per second, network traffic, and request response time.
[0055] Specifically, the configuration information includes message retention strategy, compression algorithm, and replica synchronization strategy.
[0056] Specifically, the distributed cache system verifies the call request for subject information, including verifying the identity of the requester and performing access control. The access control specifically requires that the number of requests made by the requester in a unit of time must be within a reasonable range. For example, when the number of requests made by the requester within 10 seconds is more than 30 times, the requester is judged to have an access security risk, and the call request from the requester is rejected and a risk warning notification is issued.
[0057] Specifically, the subject prefix information includes professional terms, industry terms, codes, and identifiers.
[0058] Specifically, the step S6 includes:
[0059] Step S61, calculating the fluctuation range of the generation time of the topic list according to the generation time of several topic lists;
[0060] Step S62, comparing the fluctuation range of the generation time of the topic list with the preset first fluctuation range, the preset second fluctuation range and the preset third fluctuation range respectively to determine whether the stability of topic list acquisition meets the requirements;
[0061] Step S63, when it is determined that the requirements are not met, adjust the metadata update method, or adjust the number of cached data structure types according to the linear fit between the byte size of the value in the key value and the cache delay duration, or adjust the backup delay duration according to the metadata call rate.
[0062] Specifically, topic metadata includes the topic name, number of partitions, and number of replicas.
[0063] Specifically, the specific method of obtaining the real-time topic metadata in the Kafka cluster can be achieved through the API interface or management tool provided by the Kafka cluster.
[0064] Specifically, the distributed cache system may be Redis or Memcached.
[0065] Specifically, the format and protocol of the topic information call request can be adaptively set according to actual conditions. For example, protocols such as RESTful API or gRPC can be used to send and receive requests.
[0066] In implementation, the method of the present invention obtains the latest topic metadata in the Kafka cluster by setting multiple acquisition steps and utilizing the metadata management mechanism of the Kafka cluster, and caches the obtained topic metadata in a distributed cache system. By using the distributed cache system, the storage effectiveness of the topic list information is improved; by adopting a data cache and backup mechanism and adjusting the backup delay duration during the backup process, conflicts with the cache process during the backup process are avoided, thereby reducing the probability of jamming during the cache process and the metadata call process; by adjusting the number of cached data structure types according to the linear fit between the byte amount of the value in the key value and the cache delay duration, the comprehensiveness of the cache is improved, and the lack of cache comprehensiveness that may be caused by a single data structure and the impact on stability is avoided; by adjusting the metadata update method, the timeliness and effectiveness of metadata updates are improved, and the efficiency and stability of acquiring the topic list are improved.
[0067] Specifically, the subject information includes a subject name and subject prefix information.
[0068] Specifically, in the step S3, before sending the subject information calling request, the subject information calling request is also compressed and encrypted.
[0069] In implementation, the method of the present invention improves the encryption level of the subject information call request by compressing and encrypting the subject information call request, improves the security of the information, and thus improves the stability of obtaining the subject list.
[0070] Specifically, in step S6, if the fluctuation range of the generation time of the topic list is within the preset first fluctuation range condition, the preset second fluctuation range condition, or the preset third fluctuation range condition, it is determined that the stability of the topic list acquisition does not meet the requirements, wherein,
[0071] If the fluctuation range of the generation time of the topic list only meets the preset first fluctuation range condition, it is preliminarily determined that the comprehensiveness of the cache does not meet the requirements, and the comprehensiveness of the cache is secondarily determined based on the linear fit between the byte amount of the value in the key value and the cache delay time;
[0072] If the fluctuation range of the generation time of the topic list only meets the preset second fluctuation range condition, it is preliminarily determined that the stability of the distributed cache system does not meet the requirements, and the stability of the distributed cache system is secondarily determined according to the call rate of the metadata;
[0073] If the fluctuation range of the generation time of the topic list only meets the preset third fluctuation range condition, it is determined to adjust the update mode of the metadata from the scheduled update mode to the scheduled update plus event-triggered update mode;
[0074] If the fluctuation range of the generation time of the topic list is within the preset fourth fluctuation range condition, it is determined that the stability of the topic list acquisition meets the requirements.
[0075] Specifically, the preset first fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is greater than the preset first fluctuation amplitude and less than or equal to the preset second fluctuation amplitude;
[0076] The preset second fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is greater than the preset second fluctuation amplitude and less than or equal to the preset third fluctuation amplitude;
[0077] The preset third fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is greater than the preset third fluctuation amplitude; the preset fourth fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is less than or equal to the preset first fluctuation amplitude.
[0078] In implementation, the preset first fluctuation amplitude is generally set to 1.6 seconds to 2.2 seconds, the preset second fluctuation amplitude is set to 3.5 seconds to 4.3 seconds, and the preset third fluctuation amplitude is set to 4.8 seconds to 5.1 seconds. It can be understood that the smaller the fluctuation amplitude of the generation time of the topic list, the higher the stability of the topic list acquisition process.
[0079] Specifically, the preferred embodiment of the preset first fluctuation amplitude is 2 seconds, the preferred embodiment of the preset second fluctuation amplitude is 4 seconds, and the preferred embodiment of the preset third fluctuation amplitude is 5 seconds.
[0080] Specifically, if the linear fit between the byte amount of the value in the key value and the cache delay duration is less than a preset fit, it is determined that the comprehensiveness of the cache does not meet the requirements, and the number of cached data structure types is increased;
[0081] The increase range of the number of cached data structure types is determined by the difference between the preset fitness and the linear fitness of the byte quantity of the value in the key value and the cache delay duration.
[0082] Specifically, the cache delay duration refers to the delay duration of the distributed cache system in a single cache process.
[0083] Specifically, the linear fit between the byte amount of the value in the key value and the cache delay time means that in a rectangular coordinate system with the byte amount of the value in the key value as the horizontal axis and the cache delay time as the vertical axis, the horizontal axis corresponds to a number of values, and the vertical axis also corresponds to a number of values, so a number of points in the coordinate system are constituted, and a linear regression function is formed by linear regression fitting of these points, and the straight-line distances from the several points to the linear regression function are calculated, and the average and reciprocal of the straight-line distances are obtained to obtain the linear fit between the byte amount of the value in the key value and the cache delay time, which represents the degree of correlation between the parameter of the byte amount of the value in the key value and the parameter of the cache delay time. The formation process of the linear regression function is a conventional technical means well known to those skilled in the art, and therefore the formation process and method of the linear regression function will not be repeated here.
[0084] In implementation, the number of cached data structure types increases by one for every 0.1 that the linear fit between the amount of bytes in the key value and the cache delay duration is less than the preset fit; for example, when the linear fit between the amount of bytes in the key value and the cache delay duration is 0.3 and the number of cached data structure types is 1, the number of cached data structure types increases by two to become 3 compared with the original number of cached data structure types; when the difference between the preset fit and the linear fit between the amount of bytes in the key value and the cache delay duration is less than 0.1, the number of cached data structure types is not increased.
[0085] Specifically, a preferred embodiment of the preset fitting degree is 0.5.
[0086] Specifically, data structure types include lists, sets, and hash tables.
[0087] Specifically, if the call rate of the metadata is lower than the preset call rate, it is determined that the stability of the distributed cache system does not meet the requirements, and the initial adjustment process of increasing the delay time of the backup is performed.
[0088] The increase of the backup delay time is determined by the difference between the preset call rate and the metadata call rate; the backup delay time is the interval between the backup time and the cache time of the distributed cache system;
[0089] The calculation formula of the metadata call rate is:
[0090] Q=E / T
[0091] Among them, Q is the call rate of metadata, E is the number of bytes of metadata called in a single call cycle, and T is the duration of a single call cycle.
[0092] In implementation, the method of the present invention increases the backup delay time by setting a preset call rate when it is determined for the second time that the stability of the distributed cache system does not meet the requirements, that is, the interval time between the backup time and the cache time of the distributed cache system is increased. The increase in the interval time between the backup time and the cache time reduces the running memory pressure of the distributed cache system when caching data, and reduces the usage of backup space in the case of a large number of short-term calls, thereby increasing the calling efficiency of the calling process, avoiding invalid space usage, and improving the efficiency of obtaining the topic list.
[0093] In practice, the preset call rate is generally set at 180MB / s to 210MB / s. It is understandable that the smaller the metadata call rate is, the lower the performance stability of the distributed cache system is.
[0094] Preferably, for every 5 MB / s that the metadata call rate is lower than the preset call rate, the backup delay duration is increased by 5 seconds based on the current backup delay duration; for example, when the metadata call rate is 180 MB / s and the current backup delay duration is 10 seconds, the preset call rate minus the metadata call rate = 20 MB / s, and the backup delay duration is increased by 4 × 5 seconds = 20 seconds based on the current backup delay duration, and the increased backup delay duration is 10 seconds + 20 seconds = 30 seconds.
[0095] Preferably, the preferred embodiment of the preset call rate is 200 MB / s.
[0096] Specifically, the scheduled update method is to update the metadata according to a predetermined period; the event-triggered update method is to update the metadata when an event involving a topic, partition, or replica occurs.
[0097] Specifically, events involving topics include the creation or deletion of topics.
[0098] Specifically, events involving partitions include the addition or deletion of partitions to a topic.
[0099] Specifically, events involving replicas include changes in replica allocation. It is understandable that the replica allocation of a partition may change due to the addition, exit, or manual adjustment of brokers (Kafka servers).
[0100] Specifically, the calculation formula for the fluctuation range of the generation time of the topic list is:
[0101] U=Umax-Umin
[0102] Among them, U is the fluctuation range of the generation time of the topic list, Umax is the maximum generation time among several topic lists, and Umin is the minimum generation time among several topic lists; the generation time of the topic list is the time taken by the requesting party from issuing a topic information call request to generating the topic list.
[0103] Specifically, the step S7 includes:
[0104] Step S71, when the initial adjustment of the delay time for the backup is completed, the data recovery time of the distributed cache system is obtained;
[0105] Step S72, determining whether the data recovery performance meets the requirements according to the data recovery time of the distributed cache system;
[0106] Step S73, if it is determined that the data recovery performance does not meet the requirements, a secondary adjustment process is performed to reduce the increased backup delay time;
[0107] Among them, if the data recovery time of the distributed cache system is longer than the preset recovery time, it is determined that the data recovery performance does not meet the requirements; the reduction extent of the increased backup delay time is determined by the difference between the data recovery time of the distributed cache system and the preset recovery time.
[0108] Specifically, the data recovery time of the distributed cache system is the time it takes to restore the original data when an abnormal situation such as a freeze or crash occurs in the distributed cache system.
[0109] In practice, the preset recovery time is generally set at 6 seconds to 15 seconds. It is understandable that the longer the data recovery time of the distributed cache system is, the worse the data recovery performance and stability of the distributed cache system are.
[0110] Preferably, the preferred embodiment of the preset recovery time length is 10 seconds.
[0111] In implementation, for every second that the data recovery time of the distributed cache system is higher than the preset recovery time, the backup delay time is reduced by 3 seconds based on the backup delay time after the initial adjustment. For example, when the initial adjustment of the backup delay time is completed and the data recovery time of the distributed cache system is 12 seconds, the backup delay time after the initial adjustment is reduced by (12-10)×3 seconds=6 seconds, and the backup delay time after the second adjustment is 30 seconds-6 seconds=24 seconds.
[0112] In implementation, the method of the present invention sets a preset recovery time. After the backup delay time is initially adjusted, that is, the backup delay time is increased. If the increase is too large, the amount of backup data may be insufficient, thereby causing a decrease in data recovery capability in the event of a data disaster or emergency. Therefore, by making a secondary adjustment to the backup delay time, a balance is achieved between the comprehensive performance improvement of the cache and backup process and the data recovery capability, thereby further improving the efficiency and stability of obtaining the subject list.
[0113] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.
Claims
1. A method for quickly obtaining a large-scale Kafka cluster Topic list based on distributed cache, characterized in that: include: Step S1, regularly obtain real-time topic metadata in the Kafka cluster; Step S2, caching the subject metadata into a distributed cache system using a key-value pair data structure type; Step S3, sending a subject information call request to the distributed cache system, the distributed cache system verifies the subject information call request and caches and backs up the subject information call request and the subject information respectively; Step S4, when the subject information call request passes the verification, the distributed cache system queries the corresponding subject metadata according to the subject information, and sends the subject metadata to the call requester to generate a subject list; Step S5, obtaining the generation time of several topic lists, and judging whether the stability of topic list acquisition meets the requirements according to the fluctuation range of the generation time of the topic lists; Step S6, when it is determined that the requirements are not met, the updating method of the metadata is adjusted, or the number of cached data structure types is adjusted according to the linear fit between the byte amount of the value in the key value and the cache delay time, or the backup delay time is initially adjusted according to the metadata call rate; Step S7, when the initial adjustment of the delay time of the backup is completed, the delay time of the backup is secondarily adjusted according to the data recovery time of the distributed cache system; The metadata update method includes a timed update method and an event-triggered update method.
2. According to the method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache in claim 1, it is characterized in that: The subject information includes a subject name and subject prefix information.
3. According to claim 2, the method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache is characterized in that: In the step S3, before sending the subject information calling request, the subject information calling request is also compressed and encrypted.
4. According to claim 3, the method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache is characterized in that: In step S6, if the fluctuation range of the generation time of the topic list is within the preset first fluctuation range condition, the preset second fluctuation range condition, or the preset third fluctuation range condition, it is determined that the stability of the topic list acquisition does not meet the requirements, wherein: If the fluctuation range of the generation time of the topic list only meets the preset first fluctuation range condition, it is preliminarily determined that the comprehensiveness of the cache does not meet the requirements, and the comprehensiveness of the cache is secondarily determined based on the linear fit between the byte amount of the value in the key value and the cache delay time; If the fluctuation range of the generation time of the topic list only meets the preset second fluctuation range condition, it is preliminarily determined that the stability of the distributed cache system does not meet the requirements, and the stability of the distributed cache system is secondarily determined according to the call rate of the metadata; If the fluctuation range of the generation time of the topic list only meets the preset third fluctuation range condition, it is determined to adjust the update mode of the metadata from the scheduled update mode to the scheduled update plus event-triggered update mode; If the fluctuation range of the generation time of the topic list is within the preset fourth fluctuation range condition, it is determined that the stability of the topic list acquisition meets the requirements.
5. According to claim 4, the method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache is characterized in that: The preset first fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is greater than the preset first fluctuation amplitude and less than or equal to the preset second fluctuation amplitude; The preset second fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is greater than the preset second fluctuation amplitude and less than or equal to the preset third fluctuation amplitude; The preset third fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is greater than the preset third fluctuation amplitude; The preset fourth fluctuation amplitude condition is that the fluctuation amplitude of the generation time of the topic list is less than or equal to the preset first fluctuation amplitude.
6. According to claim 5, the method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache is characterized in that: If the linear fit between the byte amount of the value in the key value and the cache delay duration is less than a preset fit, it is determined that the comprehensiveness of the cache does not meet the requirements, and the number of cached data structure types is increased; The increase range of the number of cached data structure types is determined by the difference between the preset fitness and the linear fitness of the byte quantity of the value in the key value and the cache delay duration.
7. The method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache according to claim 6 is characterized in that: If the call rate of the metadata is lower than the preset call rate, it is determined that the stability of the distributed cache system does not meet the requirements, and the initial adjustment process of increasing the delay time of the backup is performed. The increase of the backup delay time is determined by the difference between the preset call rate and the metadata call rate; the backup delay time is the interval between the backup time and the cache time of the distributed cache system; The calculation formula of the metadata call rate is: Q=E / T Among them, Q is the call rate of metadata, E is the number of bytes of metadata called in a single call cycle, and T is the duration of a single call cycle.
8. The method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache according to claim 7 is characterized in that: The scheduled update method is to update the metadata according to a predetermined period; the event-triggered update method is to update the metadata when a change occurs in the subject metadata.
9. The method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache according to claim 8 is characterized in that: The calculation formula for the fluctuation range of the generation time of the topic list is: U=Umax-Umin Among them, U is the fluctuation range of the generation time of the topic list, Umax is the maximum generation time among several topic lists, and Umin is the minimum generation time among several topic lists; the generation time of the topic list is the time taken by the requesting party from issuing a topic information call request to generating the topic list.
10. The method for quickly acquiring a large-scale Kafka cluster Topic list based on distributed cache according to claim 9 is characterized in that: The step S7 comprises: Step S71, when the initial adjustment of the delay time for the backup is completed, the data recovery time of the distributed cache system is obtained; Step S72, determining whether the data recovery performance meets the requirements according to the data recovery time of the distributed cache system; Step S73, if it is determined that the data recovery performance does not meet the requirements, a secondary adjustment process is performed to reduce the increased backup delay time; Among them, if the data recovery time of the distributed cache system is longer than the preset recovery time, it is determined that the data recovery performance does not meet the requirements; the reduction extent of the increased backup delay time is determined by the difference between the data recovery time of the distributed cache system and the preset recovery time.
