Kafka-based resource processing method and apparatus, device, and storage medium

Abstract

The application discloses a Kafka-based resource processing method and device, equipment and a storage medium, and the method comprises the following steps: determining resource usage information of a Kafka resource requester; predicting future resource usage demand of the Kafka resource requester based on the resource usage information and a preset resource usage demand model; and dynamically adjusting the resources of the Kafka resource requester based on the future resource usage demand; wherein the resources of the Kafka are allocated to the Kafka resource requester based on topic logic and tenant logic. In the application, resource and data isolation between different businesses is ensured, a preset resource usage demand model is used, that is, a machine learning algorithm is used for resource usage demand prediction and analysis, and the resources of each Kafka resource requester are dynamically adjusted, so that the resource utilization rate is maximized.

Description

Technical Field

[0001] This application relates to the field of communication and computer technology, and in particular to a resource processing method, apparatus, device and storage medium based on Kafka. Background Technology

[0002] Kafka is the most widely used open-source message middleware or high-throughput distributed publish-subscribe messaging system. It is frequently used in high-throughput scenarios, such as distributed message subscription-publishing and stream processing.

[0003] Multiple business units often need to use Kafka resources in production or processing. Currently, different business units need to share Kafka resources. However, it is difficult to achieve data isolation when different business units share Kafka resources, which leads to data security risks. In addition, there are problems of uneven resource allocation and resource contention when different business units share Kafka resources, resulting in low utilization of Kafka resources.

[0004] In other words, existing Kafka resources suffer from low utilization. Summary of the Invention

[0005] The main purpose of this application is to provide a resource processing method, apparatus, device and storage medium based on Kafka, which aims to solve the technical problem of low resource utilization when using Kafka resources in existing businesses.

[0006] To achieve the above objectives, this application provides a Kafka-based resource processing method, which includes:

[0007] Determine the resource usage information of the Kafka resource requester;

[0008] Based on the resource usage information and the preset resource usage demand model, predict the future resource usage demand of the Kafka resource requester;

[0009] Based on the future resource usage requirements, the resources of the Kafka resource requesters are dynamically adjusted.

[0010] The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic.

[0011] Optionally, before the step of dynamically adjusting the resources of the Kafka resource requester based on the future resource usage needs, the method includes:

[0012] Upon receiving a tenant request instruction from a Kafka resource requester, determine the request information in the tenant request instruction;

[0013] Based on the application information and the preset multi-tenant adaptation model, the tenant logic and topic logic of the Kafka resource requester are adapted to obtain the target tenant and target topic respectively.

[0014] Configure permissions and partitions for the target tenant to obtain the target permissions and target partitions for the target tenant, respectively.

[0015] The step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage needs includes:

[0016] Based on the target tenant, target permissions, target topic, target partition, and future resource usage requirements, the resources of the Kafka resource requester are dynamically adjusted.

[0017] Optionally, the step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage needs includes:

[0018] The preset permission controller is invoked to dynamically adjust the resources of the Kafka resource requester based on the future resource usage requirements and the preset permission controller.

[0019] The preset permission controller allocates Kafka resources to the Kafka resource requester based on topic logic and tenant logic.

[0020] Optionally, the step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage requirements and the preset permission controller includes:

[0021] Based on the preset permission controller, the fine-grained permissions of the target tenant are determined;

[0022] Based on the fine-grained permissions of the target tenant, determine the adjustment subject that will dynamically adjust the resources of the Kafka resource requester;

[0023] Based on the aforementioned adjustment subject and the future resource usage requirements, the resources of the Kafka resource requester are dynamically adjusted.

[0024] Optionally, before the step of predicting the future resource usage needs of the Kafka resource requester based on the resource usage information and a preset resource usage demand model, the method includes:

[0025] Obtain training data with preset resource requirement labels;

[0026] Based on the training data, the preset training model is iteratively trained until a model that meets the preset training completion conditions is obtained. The model that meets the preset training completion conditions is then used as the preset resource usage requirement model.

[0027] Optionally, after the step of determining the resource usage information of the Kafka resource requester, the method includes:

[0028] Based on the resource usage information, determine whether the resource value currently used by the Kafka resource requester is within a preset threshold;

[0029] If the resource usage is not within the preset threshold, a resource usage alarm is generated for the Kafka resource requester.

[0030] Optionally, the step of determining the resource usage information of the Kafka resource requester includes:

[0031] Obtain the topic logic lifecycle of the Kafka resource requester recorded by the Kafka-based resource processing platform, wherein the topic logic lifecycle is determined based on the storage time of the corresponding data of the topic logic.

[0032] Based on the topic's logical lifecycle, the resource usage information of the Kafka resource requester is determined.

[0033] This application also provides a Kafka-based resource processing device, which includes:

[0034] The first determination module is used to determine the resource usage information of the Kafka resource requester;

[0035] The prediction module is used to predict the future resource usage needs of the Kafka resource requester based on the resource usage information and a preset resource usage demand model.

[0036] The adjustment module is used to dynamically adjust the resources of the Kafka resource requester based on the future resource usage requirements.

[0037] The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic.

[0038] This application also provides a Kafka-based resource processing device, which is a physical node device. The Kafka-based resource processing device includes: a memory, a processor, and a program of the Kafka-based resource processing method stored in the memory and executable on the processor. When the program of the Kafka-based resource processing method is executed by the processor, it can implement the steps of the Kafka-based resource processing method as described above.

[0039] This application also provides a storage medium storing a program that implements the above-described Kafka-based resource processing method. When the program is executed by a processor, it implements the steps of the above-described Kafka-based resource processing method.

[0040] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the Kafka-based resource processing method described above.

[0041] This application provides a resource processing method, apparatus, device, and storage medium based on Kafka. Compared to existing technologies where different services share Kafka resources, resulting in low resource utilization, this application determines the resource usage information of Kafka resource requesters; based on the resource usage information and a preset resource usage demand model, it predicts the future resource usage demand of the Kafka resource requesters; and based on the future resource usage demand, it dynamically adjusts the resources of the Kafka resource requesters. The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic. This application quickly maps or allocates Kafka resources to or allocates them to different topic logics and tenant logics (different services use different tenant logics, and different services use different topic logics), thereby ensuring resource and data isolation between different services. Furthermore, this application uses a preset resource usage demand model, i.e., uses machine learning algorithms to predict and analyze resource usage demand, dynamically adjusting the resources of each Kafka resource requester to maximize resource utilization. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the first embodiment of the resource processing method based on Kafka in this application;

[0045] Figure 2 This is a flowchart illustrating the process prior to step S20 in the Kafka-based resource processing method of this application.

[0046] Figure 3 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application;

[0047] Figure 4 This is a schematic diagram of the first scenario involving the resource processing method based on Kafka in this application;

[0048] Figure 5 This is a schematic diagram of the second scenario involved in the resource processing method based on Kafka in this application.

[0049] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0051] This application provides a resource processing method based on Kafka. In the first embodiment of this Kafka-based resource processing method, refer to... Figure 1 The Kafka-based resource processing method includes:

[0052] Step S10: Determine the resource usage information of the Kafka resource requester;

[0053] Step S20: Based on the resource usage information and the preset resource usage demand model, predict the future resource usage demand of the Kafka resource requester;

[0054] Step S30: Based on the future resource usage requirements, dynamically adjust the resources of the Kafka resource requester;

[0055] The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic.

[0056] In this embodiment, it should be noted that the Kafka-based resource processing method can be applied to a Kafka-based resource processing device, which belongs to a Kafka-based resource processing system, which belongs to a Kafka-based resource processing platform, which belongs to a Kafka-based resource processing equipment.

[0057] In this embodiment, the Kafka-based resource processing system includes not only a Kafka-based resource processing device but also a Kafka resource requester, which requests resources from the Kafka-based resource processing device.

[0058] In this embodiment, the application background is:

[0059] First, multiple businesses often need to use Kafka resources in production or processing. Currently, different businesses need to share Kafka resources, but it is difficult to achieve data isolation when different businesses share Kafka resources, which leads to data security risks. In addition, there are problems of uneven resource allocation and resource contention among different Kafka resource requesters, resulting in low resource utilization of Kafka resource requesters.

[0060] Secondly, in order to achieve data isolation between different businesses, different businesses need to use their own independent Kafka resources. The use of independent Kafka resources by different businesses increases the operational and maintenance difficulty of deploying and managing Kafka resources, and makes it difficult to maximize resource utilization.

[0061] In this embodiment, Kafka resources are quickly mapped or allocated to different topic logic and tenant logic (different services use different tenant logic, and different services use different topic logic), thereby ensuring resource and data isolation between different services. In addition, in this application, a preset resource usage demand model is used, that is, machine learning algorithms are used to predict and analyze resource usage demand, and the resources of each Kafka resource requester are dynamically adjusted to maximize resource utilization.

[0062] The specific steps are as follows:

[0063] Step S10: Determine the resource usage information of the Kafka resource requester;

[0064] In this embodiment, it should be noted that the Kafka resource requester can be any business party. Specifically, for example, the Kafka resource requester can be business party a1, business party a2, and business party a3, etc., without any specific limitation.

[0065] One way to determine the resource usage information of a Kafka resource requester is as follows:

[0066] Method 1: Directly obtain the historical resource usage of the Kafka resource requester, and thus obtain the resource usage information of the Kafka resource requester;

[0067] Method 2: Directly call the resource allocation information allocated to the Kafka resource requester, and then obtain the resource usage information of the Kafka resource requester.

[0068] The resource usage information includes resource utilization information and the historical task execution status of Kafka resource requesters.

[0069] Step S20: Based on the resource usage information and the preset resource usage demand model, predict the future resource usage demand of the Kafka resource requester;

[0070] In this embodiment, the preset resource usage demand model is a pre-trained model that can accurately predict future resource usage demands. Therefore, by inputting resource usage information into the preset resource usage demand model and performing prediction processing on the resource usage information based on the preset resource usage demand model, the future resource usage demands of the Kafka resource requester can be obtained.

[0071] Prior to the step of predicting the future resource usage needs of the Kafka resource requester based on the resource usage information and a preset resource usage demand model, the method includes:

[0072] Step S01: Obtain training data with preset resource requirement labels;

[0073] Step S02: Based on the training data, iteratively train the preset training model until a model that meets the preset training completion conditions is obtained, and use the model that meets the preset training completion conditions as the preset resource usage requirement model.

[0074] This embodiment specifically explains how to obtain a preset resource usage demand model.

[0075] Specifically, firstly, training data with preset resource requirement labels is obtained, and then the training data is input into the preset model to be trained.

[0076] The preset training model obtains the training prediction result of the training data based on the training data and the model parameters in the preset training model;

[0077] The training prediction results are compared with the preset resource requirement labels to calculate the difference, and the training error results are obtained.

[0078] Determine whether the training error result meets the error standard indicated by the preset error threshold range;

[0079] If the training error result does not meet the error standard indicated by the preset error threshold range, then update the current model parameters of the preset model to be trained, and return to the step of obtaining the training prediction result of the training data based on the training data and the model parameters using the preset model to be trained, until the training error result meets the error standard indicated by the preset error threshold range and then stop training.

[0080] The preset model to be trained, which has stopped training, is used as the preset resource usage requirement model with model parameters that meet the accuracy requirements.

[0081] The training error result includes the mean squared error result, the preset error threshold range includes the preset mean squared error threshold range, and the error criterion for determining whether the training error result meets the preset error threshold range includes: determining whether the mean squared error result is less than the preset mean squared error threshold range.

[0082] Step S30: Based on the future resource usage requirements, dynamically adjust the resources of the Kafka resource requester;

[0083] The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic.

[0084] In this embodiment, it should be noted that the Kafka resources are allocated to the Kafka resource requester based on topic logic and tenant logic.

[0085] It should be noted that the allocation of Kafka resources to the Kafka resource requester based on topic logic and tenant logic involves (multi)tenant technology. Currently, multi-tenant technology is most widely used in the field of cloud computing, but with the rapid development of the big data field, the concept of multi-tenant related to big data has also been proposed.

[0086] Among them, multi-tenant mode is divided into physical multi-tenant and logical multi-tenant, etc. The allocation based on topic logic and tenant logic belongs to the category of logical multi-tenant.

[0087] In this context, a tenant (tenant logic) is the specific carrier for a Kafka resource requester to request resources. A tenant can be a team, that is, a team of Kafka resource requesters. In this embodiment, a tenant can be expanded into a tree structure, that is, a tenant can have different levels. A tree structure of tenants can not only intuitively show the inheritance relationship between tenants, but also flexibly expand tenant nodes.

[0088] In this embodiment, a tenant can have multiple users; that is, if a tenant is a team, then the users are the members of the team.

[0089] In this embodiment, a topic is a tenant entity. Specifically, a topic is used as a tenant entity to implement a Kafka logical tenant instance. In this embodiment, a topic can specifically be represented as: what role can perform what operation on which resource.

[0090] In this embodiment, one Kafka tenant can correspond to multiple Kafka topics, and one Kafka tenant can correspond to multiple users. In this embodiment, while ensuring data isolation between Kafka resource requesters, different users are allowed to share Kafka topics.

[0091] In this embodiment, the resources of the Kafka resource requester are dynamically adjusted based on the future resource usage requirements.

[0092] Specifically, in this embodiment, after obtaining future resource usage requirements, resources are allocated to these requirements through Kafka topics and Kafka tenants, thereby obtaining the resource allocation results.

[0093] like Figure 2 As shown, before the step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage needs, the method includes:

[0094] Step S03: Upon receiving a tenant request instruction from a Kafka resource requester, determine the request information in the tenant request instruction;

[0095] like Figure 4 As shown, the tenant is obtained by the Kafka resource requester. When the tenant request instruction is received from the Kafka resource requester, the request information in the tenant request instruction is determined. The request information includes the request topic, the number of resources requested, the number of users, and other information.

[0096] Step S04: Based on the application information and the preset multi-tenant adaptation model, adapt the tenant logic and topic logic of the Kafka resource requester to obtain the target tenant and target topic respectively.

[0097] Step S05: Configure permissions and partitions for the target tenant to obtain the target permissions and target partitions for the target tenant.

[0098] In this embodiment, the Kafka-based resource processing device approves the Kafka resource requester and the application information. After approval, based on the application information and a preset multi-tenant adaptation model (pre-trained), the Kafka resource requester is adapted for tenant logic and topic logic to obtain the target tenant and target topic. In this embodiment, the target tenant is also configured with permissions and partitions to obtain the target permissions and target partitions of the target tenant.

[0099] Specifically, Kafka creates or allocates tenant instances and topic instances based on request information. After the tenant and topic instances are created or allocated, the resource requester connects to Kafka. Then, Kafka adds an entry point for permissions. Specifically, Kafka adds an entry point for permissions (permission configuration) in the service instances (tenant instances and topic instances), thereby controlling the permissions of Kafka resource requesters. If permissions are not configured, the tenant instance and topic instance will not be able to bind users and will not be accessible.

[0100] After configuring tenant permissions, it also supports configuring permissions for multiple users. The configured permissions for multiple users support permissions such as publish, consume, configure, create, delete, etc.

[0101] In this embodiment, the target tenant is also partitioned to obtain the target partition of the target tenant. In this embodiment, the partition is used to process throughput in parallel. That is, the partition is used as an instance attribute of the resource to set up the partition to meet high throughput.

[0102] It's important to note that each partition is an ordered and immutable set of records, continuously appended to a structured commit log file. Each record in a partition is assigned an ID number to indicate its order, called the offset, which uniquely identifies each record within the partition.

[0103] The step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage needs includes:

[0104] Step A1: Based on the target tenant, the target permissions, the target topic, the target partition, and the future resource usage requirements, dynamically adjust the resources of the Kafka resource requester.

[0105] In this embodiment, dynamic resource adjustment for the Kafka resource requester is specifically implemented based on the target tenant, the target permissions, the target topic, the target partition, and the future resource usage requirements.

[0106] This application provides a Kafka-based resource processing method, apparatus, device, and storage medium. Compared to existing technologies where different services share Kafka resources, leading to low resource utilization for Kafka resource requesters, this application determines the resource usage information of Kafka resource requesters; based on this information and a preset resource usage demand model, it predicts the future resource usage demands of the Kafka resource requesters; and based on these future demands, it dynamically adjusts the resources allocated to the Kafka resource requesters. The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic. This application quickly maps or allocates Kafka resources to or allocates them to different topic logics and tenant logics (different services use different tenant logics, and different services use different topic logics), thereby ensuring resource and data isolation between different services. Furthermore, this application uses a preset resource usage demand model, i.e., machine learning algorithms, to predict and analyze resource usage demands, dynamically adjusting the resources of each Kafka resource requester to maximize resource utilization.

[0107] Furthermore, based on the first embodiment of this application, another embodiment of this application is provided. In this embodiment, the step of dynamically adjusting the resources of the Kafka resource requester based on the future resource usage requirements includes:

[0108] Step B1: Invoke the preset permission controller to dynamically adjust the resources of the Kafka resource requester based on the future resource usage requirements and the preset permission controller;

[0109] The preset permission controller allocates Kafka resources to the Kafka resource requester based on topic logic and tenant logic.

[0110] In this embodiment, the access controller is responsible for determining who belongs to which tenant, and with what role can perform what operation on which resource. The access controller achieves data isolation between tenants through access control. That is, in this embodiment, a unified device (access controller) is set up to dynamically adjust resources for the Kafka resource requesters.

[0111] The step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage requirements and the preset permission controller includes:

[0112] Step C1: Based on the preset permission controller, determine the fine-grained permissions of the target tenant;

[0113] Step C2: Based on the fine-grained permissions of the target tenant, determine the adjustment subject that will dynamically adjust the resources of the Kafka resource requester;

[0114] Step C3: Based on the adjustment subject and the future resource usage requirements, dynamically adjust the resources of the Kafka resource requester.

[0115] In this embodiment, based on the preset permission controller, fine-grained permissions for the target tenant are determined. Fine-grained permissions refer to the level of granularity in permission control, such as determining which entity can perform permission control. Based on the target tenant's fine-grained permissions, the entity responsible for dynamically adjusting resources for the Kafka resource requester (platform administrator or tenant administrator) is determined. Based on the adjusting entity and the future resource usage requirements, dynamic resource adjustments are made for the Kafka resource requester. In this application, different fine-grained permissions can be used to determine the adjusting entity, thus providing adjustment options and improving user experience.

[0116] Furthermore, based on the first and second embodiments of this application, another embodiment of this application is provided. In this embodiment, after the step of sharing the target sharing account with the first user so that the first user can play the first cloud game based on the target sharing account, the method includes:

[0117] After the step of determining the resource usage information of the Kafka resource requester, the method includes:

[0118] Step D1: Based on the resource usage information, determine whether the resource value currently used by the Kafka resource requester is within a preset threshold;

[0119] Step D2: If the resource usage is not within the preset threshold, generate a resource usage alarm for the Kafka resource requester.

[0120] In this embodiment, a threshold can be configured to determine whether the resource value currently used by the Kafka resource requester is within a preset threshold. If it is not within the preset threshold, a resource usage alarm for the Kafka resource requester is generated to ensure that the service will not be down due to resource usage.

[0121] Furthermore, based on the first, second, and third embodiments of this application, another embodiment of this application is provided. In this embodiment, the step of determining the resource usage information of the Kafka resource requester includes:

[0122] Step S11: Obtain the topic logic lifecycle of the Kafka resource requester recorded by the Kafka-based resource processing platform, wherein the topic logic lifecycle is determined based on the storage time of the corresponding topic logic data.

[0123] Step S12: Based on the topic's logical lifecycle, determine the resource usage information of the Kafka resource requester.

[0124] In this embodiment, the Kafka topic time to live (TTL) is also used as the data storage time of the instance to manage the data lifecycle. That is, the topic logical lifecycle management manages the entire process from topic generation to release for subsequent querying.

[0125] Reference Figure 3 , Figure 3 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0126] like Figure 3 As shown, the Kafka-based resource processing device may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to establish communication between the processor 1001 and the memory 1005. The memory 1005 may be high-speed RAM or stable, non-volatile memory, such as disk storage. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0127] Optionally, the Kafka-based resource processing device may also include a rectangular user interface, a network interface, a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, etc. The rectangular user interface may include a display screen and an input submodule such as a keyboard. Optionally, the rectangular user interface may also include standard wired or wireless interfaces. The network interface may optionally include standard wired or wireless interfaces (such as a Wi-Fi interface).

[0128] Those skilled in the art will understand that Figure 3 The Kafka-based resource processing device architecture shown in the diagram does not constitute a limitation on Kafka-based resource processing devices. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0129] like Figure 3As shown, the memory 1005, serving as a storage medium, may include an operating system, a network communication module, and a Kafka-based resource processing program. The operating system is a program that manages and controls the hardware and software resources of the Kafka-based resource processing device, supporting the operation of the Kafka-based resource processing program and other software and / or programs. The network communication module is used to enable communication between the various components within the memory 1005, as well as communication with other hardware and software in the Kafka-based resource processing system.

[0130] exist Figure 3 In the Kafka-based resource processing device shown, the processor 1001 is used to execute the Kafka-based resource processing program stored in the memory 1005 to implement the steps of the Kafka-based resource processing method described in any of the above claims.

[0131] The specific implementation of the Kafka-based resource processing device in this application is basically the same as the embodiments of the Kafka-based resource processing method described above, and will not be repeated here.

[0132] This application also provides a Kafka-based resource processing device, which includes:

[0133] The first determination module is used to determine the resource usage information of the Kafka resource requester;

[0134] The prediction module is used to predict the future resource usage needs of the Kafka resource requester based on the resource usage information and a preset resource usage demand model.

[0135] The adjustment module is used to dynamically adjust the resources of the Kafka resource requester based on the future resource usage requirements.

[0136] The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic.

[0137] Optionally, the Kafka-based resource processing device further includes:

[0138] The second determining module is used to determine the application information in the tenant application instruction when it receives the tenant application instruction from the Kafka resource requester;

[0139] The adaptation module is used to adapt the Kafka resource requester's tenant logic and topic logic based on the application information and the preset multi-tenant adaptation model, so as to obtain the target tenant and target topic respectively.

[0140] The configuration module is used to configure permissions and partitions for the target tenant, thereby obtaining the target permissions and target partitions for the target tenant.

[0141] The adjustment module includes:

[0142] The dynamic adjustment unit is used to dynamically adjust the resources of the Kafka resource requester based on the target tenant, the target permissions, the target topic, the target partition, and the future resource usage requirements.

[0143] Optionally, the dynamic adjustment unit is used to achieve:

[0144] The preset permission controller is invoked to dynamically adjust the resources of the Kafka resource requester based on the future resource usage requirements and the preset permission controller.

[0145] The preset permission controller allocates Kafka resources to the Kafka resource requester based on topic logic and tenant logic.

[0146] Optionally, the dynamic adjustment unit is used to achieve:

[0147] Based on the preset permission controller, the fine-grained permissions of the target tenant are determined;

[0148] Based on the fine-grained permissions of the target tenant, determine the adjustment subject that will dynamically adjust the resources of the Kafka resource requester;

[0149] Based on the aforementioned adjustment subject and the future resource usage requirements, the resources of the Kafka resource requester are dynamically adjusted.

[0150] Optionally, the device further includes:

[0151] Obtain training data with preset resource requirement labels;

[0152] Based on the training data, the preset training model is iteratively trained until a model that meets the preset training completion conditions is obtained. The model that meets the preset training completion conditions is then used as the preset resource usage requirement model.

[0153] Optionally, the device further includes:

[0154] Based on the resource usage information, determine whether the resource value currently used by the Kafka resource requester is within a preset threshold;

[0155] If the resource usage is not within the preset threshold, a resource usage alarm is generated for the Kafka resource requester.

[0156] Optionally, the first determining module is used to implement:

[0157] Obtain the topic logic lifecycle of the Kafka resource requester recorded by the Kafka-based resource processing platform, wherein the topic logic lifecycle is determined based on the storage time of the corresponding data of the topic logic.

[0158] Based on the topic's logical lifecycle, the resource usage information of the Kafka resource requester is determined.

[0159] The specific implementation of the Kafka-based resource processing device in this application is basically the same as the embodiments of the Kafka-based resource processing method described above, and will not be repeated here.

[0160] This application provides a storage medium that stores one or more programs, which can be executed by one or more processors to implement the steps of the Kafka-based resource processing method described above.

[0161] The specific implementation of the storage medium in this application is basically the same as the embodiments of the resource processing method based on Kafka described above, and will not be repeated here.

[0162] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the Kafka-based resource processing method described above.

[0163] The specific implementation of the computer program product in this application is basically the same as the embodiments of the resource processing method based on Kafka described above, and will not be repeated here.

[0164] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0165] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0167] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A resource processing method based on Kafka, characterized in that, The Kafka-based resource processing method includes: Determine the resource usage information of the Kafka resource requester; Based on the resource usage information and the preset resource usage demand model, predict the future resource usage demand of the Kafka resource requester; Upon receiving a tenant request instruction from a Kafka resource requester, determine the request information in the tenant request instruction; Based on the application information and the preset multi-tenant adaptation model, the tenant logic and topic logic of the Kafka resource requester are adapted to obtain the target tenant and target topic respectively. The target tenant is configured with permissions and partitions to obtain the target permissions and target partitions of the target tenant, respectively. After configuring permissions for the target tenant as a whole, permissions are configured for multiple users in the target tenant. Based on the future resource usage requirements, the resources of the Kafka resource requesters are dynamically adjusted. The step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage needs includes: Based on the target tenant, target permissions, target topic, target partition, and future resource usage requirements, the resources of the Kafka resource requester are dynamically adjusted. The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic.

2. The resource processing method based on Kafka as described in claim 1, characterized in that, The step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage needs includes: The preset permission controller is invoked to dynamically adjust the resources of the Kafka resource requester based on the future resource usage requirements and the preset permission controller. The preset permission controller allocates Kafka resources to the Kafka resource requester based on topic logic and tenant logic.

3. The resource processing method based on Kafka as described in claim 2, characterized in that, The step of dynamically adjusting resources for the Kafka resource requester based on the future resource usage requirements and the preset permission controller includes: Based on the preset permission controller, the fine-grained permissions of the target tenant are determined; Based on the fine-grained permissions of the target tenant, determine the adjustment subject that will dynamically adjust the resources of the Kafka resource requester; Based on the aforementioned adjustment subject and the future resource usage requirements, the resources of the Kafka resource requester are dynamically adjusted.

4. The resource processing method based on Kafka as described in claim 1, characterized in that, Before the step of predicting the future resource usage needs of the Kafka resource requester based on the resource usage information and a preset resource usage demand model, the method includes: Obtain training data with preset resource requirement labels; Based on the training data, the preset training model is iteratively trained until a model that meets the preset training completion conditions is obtained. The model that meets the preset training completion conditions is then used as the preset resource usage requirement model.

5. The resource processing method based on Kafka as described in claim 1, characterized in that, After the step of determining the resource usage information of the Kafka resource requester, the method includes: Based on the resource usage information, determine whether the resource value currently used by the Kafka resource requester is within a preset threshold; If the resource usage is not within the preset threshold, a resource usage alarm is generated for the Kafka resource requester.

6. The resource processing method based on Kafka as described in claim 1, characterized in that, The steps for determining the resource usage information of the Kafka resource requester include: Obtain the topic logic lifecycle of the Kafka resource requester recorded by the Kafka-based resource processing platform, wherein the topic logic lifecycle is determined based on the storage time of the corresponding data of the topic logic. Based on the topic's logical lifecycle, the resource usage information of the Kafka resource requester is determined.

7. A resource processing device based on Kafka, characterized in that, The Kafka-based resource processing device includes: The first determination module is used to determine the resource usage information of the Kafka resource requester; The prediction module is used to predict the future resource usage needs of the Kafka resource requester based on the resource usage information and a preset resource usage demand model. The adjustment module is used to dynamically adjust the resources of the Kafka resource requester based on the future resource usage requirements. The Kafka resources are allocated to the Kafka resource requesters based on topic logic and tenant logic. The second determining module is used to determine the application information in the tenant application instruction when it receives the tenant application instruction from the Kafka resource requester; The adaptation module is used to adapt the Kafka resource requester's tenant logic and topic logic based on the application information and the preset multi-tenant adaptation model, so as to obtain the target tenant and target topic respectively. The configuration module is used to configure permissions and partitions for the target tenant, thereby obtaining the target permissions and target partitions of the target tenant. After configuring permissions for the target tenant as a whole, permissions are configured for multiple users within the target tenant. The adjustment module includes: The dynamic adjustment unit is used to dynamically adjust the resources of the Kafka resource requester based on the target tenant, the target permissions, the target topic, the target partition, and the future resource usage requirements.

8. A resource processing device based on Kafka, characterized in that, The Kafka-based resource processing device includes: a memory, a processor, and a program stored in the memory for implementing the Kafka-based resource processing method. The memory is used to store programs that implement Kafka-based resource processing methods; The processor is configured to execute a program that implements the Kafka-based resource processing method to implement the steps of the Kafka-based resource processing method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a program that implements a Kafka-based resource processing method, which is executed by a processor to implement the steps of the Kafka-based resource processing method as described in any one of claims 1 to 6.

Images