Data partition management method and apparatus, and electronic device

Abstract

The present disclosure provides a data partition management method and apparatus, and an electronic device. The data partition management method comprises: determining a target message topic of a message to be allocated, and a plurality of data groups corresponding to the target message topic, wherein each of the plurality of data groups corresponds to one quota parameter, and the quota parameter is used for determining the allocation proportion corresponding to the data groups; on the basis of the quota parameters, allocating to the data groups a plurality of data partitions corresponding to the target message topic; and on the basis of the quota parameters, allocating to the data groups a plurality of data consumers corresponding to the target message topic.

Description

Data partition management method, device and electronic device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is based on and claims priority to an application with CN application number 202311030285.9 and filing date August 15, 2023. The disclosure content of this CN application is hereby incorporated into this disclosure as a whole. Technical Field

[0003] The present disclosure relates to the field of big data, and in particular to a data partition management method, device, and electronic device. Background Art

[0004] In related technologies, when using Kafka to implement a priority queue, a separate topic is created for each priority. On the user side, these separate topics need to be logically encapsulated into one topic, and then combined and pulled using the configured ratio to achieve a priority effect.

[0005] Summary of the Invention

[0006] According to one aspect of an embodiment of the present disclosure, a data partition management method is provided, including: determining a target message topic of a message to be allocated, and multiple data groups corresponding to the target message topic, wherein each data group in the multiple data groups corresponds to a quota parameter, and the quota parameter is used to determine the allocation ratio corresponding to each data group; based on the quota parameter, allocating multiple data partitions corresponding to the target message topic to each data group; based on the quota parameter, allocating multiple data consumption objects corresponding to the target message topic to each data group.

[0007] In some embodiments, determining a target message topic and multiple data groups corresponding to the target message topic includes: establishing a custom partitioner, wherein the custom partitioner is used to divide the data partitions; obtaining and parsing configuration parameters through the custom partitioner to obtain the target message topic, the message key prefix of each data group, and the quota parameters corresponding to the message key prefix; and determining the data groups corresponding to each message key prefix through the custom partitioner.

[0008] In some embodiments, allocating multiple data partitions corresponding to the target message topic to each data group based on quota parameters includes: determining the number of first data partitions corresponding to the target message topic; determining the allocation ratio corresponding to each data group based on the quota parameters; determining the number of second data partitions allocated to each data group based on the allocation ratio and the number of first data partitions; and dividing the second number of data partitions into data groups through a custom partitioner.

[0009] In some embodiments, the allocation ratio corresponding to each data group is equal to the ratio of the quota parameter corresponding to each data group to the sum of the quota parameters corresponding to all data groups.

[0010] In some embodiments, each data group uniquely corresponds to a message key prefix, and different data groups correspond to different message key prefixes.

[0011] In some embodiments, after allocating multiple data partitions corresponding to the target message topic to each data group based on quota parameters, the data partition management method also includes: receiving a message to be allocated that matches the target message topic; determining a message key prefix of the message to be allocated; and allocating the message to be allocated to the data partition in the target data group based on the message key prefix through a partition interface constructed by a custom partitioner, wherein the message key prefix of the target data group is consistent with the message key prefix of the message to be allocated.

[0012] In some embodiments, allocating multiple data consumption objects corresponding to the target message topic to each data group based on quota parameters includes: establishing a custom partition matcher, wherein the custom partition matcher is used to allocate corresponding partitions to data consumption objects; determining the number of data consumption objects allocated to each data group based on the quota parameters through the custom partition matcher; and allocating the number of data consumption objects corresponding to each data group to each data group through the custom partition matcher.

[0013] In some embodiments, determining the number of data consumption objects allocated to each data group based on quota parameters through a custom partition matcher includes: determining the allocation ratio of each data group based on quota parameters through a custom partition matcher; determining the first number of data consumption objects of the data consumption objects corresponding to the target message topic; and determining the second number of data consumption objects corresponding to each data group based on the allocation ratio and the first number of data consumption objects.

[0014] In some embodiments, the allocation ratio of each data group is equal to the ratio of the quota parameter corresponding to each data group to the sum of the quota parameters corresponding to all data groups.

[0015] In some embodiments, determining the number of data consumption objects allocated to each data group based on quota parameters through a custom partition matcher includes: determining the number of third data consumption objects corresponding to each data group based on the quota parameters, wherein the larger the quota parameter corresponding to the array, the greater the number of third data consumption objects corresponding to the data group.

[0016] In some embodiments, the number of data partitions corresponding to the same data group is greater than or equal to the number of corresponding data consumption objects.

[0017] According to another aspect of an embodiment of the present disclosure, a data partition management device is also provided, including: a first processing module, used to determine a target message topic, and multiple data groups corresponding to the target message topic, wherein each of the multiple data groups corresponds to a quota parameter, and the quota parameter is used to determine the allocation ratio corresponding to each data group; a second processing module, used to allocate multiple data partitions corresponding to the target message topic to each data group based on the quota parameter; a third processing module, used to allocate multiple data consumption objects corresponding to the target message topic to each data group based on the quota parameter.

[0018] According to another aspect of an embodiment of the present disclosure, a non-volatile storage medium is provided, including: a program stored in the non-volatile storage medium, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute a data partition management method.

[0019] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the data partition management method is executed when the program is run.

[0020] According to another aspect of an embodiment of the present disclosure, a computer program is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, the method as described in any one of the above embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0022] FIG1 is a schematic structural diagram of an optional computer terminal (mobile terminal) according to an embodiment of the present disclosure;

[0023] FIG2 is a flow chart of an optional data partition management method according to an embodiment of the present disclosure;

[0024] FIG3 is a schematic diagram of data partitions included in each data group and consumer objects corresponding to the data group according to an embodiment of the present disclosure;

[0025] FIG4 is a module diagram of an optional data partition management device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In order to better understand the embodiments of the present disclosure, the technical terms involved in the embodiments of the present disclosure are explained as follows:

[0029] Kafka is a distributed stream processing platform and a high-throughput, persistent, and fault-tolerant publish-subscribe messaging system. Kafka's goal is to process real-time streaming data through the parallel processing of Hadoop, an open-source distributed computing platform for storing and processing large datasets. Kafka's design takes into account the following key aspects: persistence, high throughput, and distribution. Kafka messages are organized into topics. Producers send messages to a specific topic, and consumers can subscribe to one or more topics and read messages from them. A topic is a category or subject of messages. Kafka topics can be divided into multiple partitions, each replicated across different brokers to ensure data reliability and fault tolerance. A broker is a server node running in a Kafka cluster. Consumers can work in consumer groups. Consumers in a consumer group consume messages from a topic together, with each consumer consuming only a subset of messages within that partition, thus achieving load balancing. Kafka also provides persistent message storage, saving messages to disk for subsequent data analysis and playback.

[0030] A priority queue is a specialized queue data structure in which each element has an associated priority. Elements with higher priorities are prioritized within the queue and are therefore processed first. The main operations of a priority queue are insertion and deletion. Insertion inserts an element into the queue at the appropriate position based on its priority, while deletion removes the element with the highest priority. Priority queues have a wide range of applications. For example, in operating systems, priority queues can be used to schedule tasks so that higher-priority tasks are executed more quickly. In graph algorithms, priority queues can be used to implement algorithms such as Dijkstra's algorithm and Prim's algorithm.

[0031] Map: A common data structure, a collection of key-value pairs, where each key is unique. Maps allow the corresponding value to be found, inserted, or deleted based on the key. Map data structures can be implemented in a variety of ways, including hash tables and balanced binary search trees. Each implementation has different performance and characteristics, and choosing the right one depends on the specific application scenario.

[0032] The inventor noticed that in the related art, when using Kafka to implement a priority queue, a separate topic is created for each priority. On the user side, these separate topics need to be logically encapsulated into one topic, and then combined and pulled using the configured ratio to achieve a priority effect. This method requires logical encapsulation in the outer layer and the creation of a topic for each priority will occupy a lot of system resources. Therefore, there are problems such as high logical encapsulation cost, waste of system resources and high maintenance cost.

[0033] Based on this, the present disclosure provides a data partition management solution to at least solve the technical problem that when implementing priority queues based on Kafka, a separate topic is created for each priority and logically encapsulated to achieve the priority effect, resulting in high implementation cost and high operation and maintenance cost.

[0034] According to an embodiment of the present disclosure, a method embodiment of a data partition management method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] The method embodiments provided in the embodiments of the present disclosure can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data partition management method. As shown in Figure 1, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used in the figure to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that the structure shown in Figure 1 is only illustrative and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in Figure 1, or have a configuration different from that shown in Figure 1.

[0036] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present disclosure, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data partition management method in the embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0039] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0040] In the above operating environment, an embodiment of the present disclosure provides a data partition management method, as shown in FIG2 , which includes the following steps:

[0041] Step S202: determining a target message topic and multiple data groups (pgroups) corresponding to the target message topic, wherein each of the multiple data groups corresponds to a quota parameter, and the quota parameter is used to determine an allocation ratio corresponding to each data group;

[0042] In the technical solution provided in step S202, the target message topic and the multiple data groups corresponding to the target message topic can be determined by the following steps: establishing a custom partitioner (CustomPriorityPartitioner), wherein the custom partitioner is used to divide the data partitions; obtaining and parsing configuration parameters through the custom partitioner to obtain proprietary parameters, wherein the proprietary parameters include the target message topic, the prefix of each data group, and the quota parameters corresponding to each data group; and determining the target message topic and the multiple data groups based on the proprietary parameters through the custom partitioner. The data group is used to store or record the data partitions corresponding to the message topic, that is, the elements in the data group are the data partitions corresponding to the message topic.

[0043] In some embodiments of the present disclosure, the above-mentioned quota parameters can be regarded as the allocation weight or allocation ratio corresponding to each data group. When allocating corresponding data partitions or consumption objects to each data group, the quota parameters of each data group can be referred to. The data group with a larger quota parameter is allocated more data partitions and consumption objects, and the data group with a smaller quota parameter is allocated fewer data partitions and consumption objects.

[0044] As an optional implementation, the aforementioned configuration parameters refer to the initial settings, typically recorded in a configuration file. The target message topic, prefix of each data group, and quota parameters corresponding to each data group obtained after parsing the configuration parameters can be used as parameters of the custom partitioner. The extracted parameters can then be applied to specific scenarios or to implement specific functions.

[0045] Taking Kafka as an example, in the scenario of implementing priority queues based on Kafka's partitions, the above-mentioned own parameters can be determined in the following way: use the method provided by Kafka's Partitioner interface to build a custom partitioner. This custom partitioner is used for custom partition division and can also store custom configurations, such as topic name, distribution ratio, etc., and is stored using a Map data structure. The key of the Map stores the logical identifier of the data group, that is, each data group uniquely corresponds to a message key prefix, and the value corresponding to the key is the partition list corresponding to the data group. The configuration parameters are converted into own parameters during initialization; then a custom partition matcher is defined to assign corresponding partitions to data consumer objects.

[0046] The Partitioner interface is used to distribute messages to different partitions. It calculates the partition to which the message should be distributed based on the incoming message's subject, key, value, key byte array, value byte array, and cluster information.

[0047] Step S204: allocating multiple data partitions corresponding to the target message topic to each data group according to the quota parameters;

[0048] In the technical solution provided in step S204, the step of allocating multiple data partitions corresponding to the target message topic to each data group based on the quota parameters includes: determining the number of first data partitions corresponding to the target message topic; determining the allocation ratio corresponding to each data group based on the quota parameters, wherein the allocation ratio corresponding to each data group is equal to the ratio of the quota parameter corresponding to each data group to the sum of the quota parameters corresponding to all data groups; determining the number of second data partitions allocated to each data group based on the allocation ratio and the number of first data partitions; and dividing the data partitions of the second data partition number to the data groups through a custom partitioner.

[0049] In some embodiments of the present disclosure, the number of partitions corresponding to each data group can be allocated according to the storage ratio of the self-configuration, as shown in the following formula:

[0050] Among them, S1 represents the number of partitions of the target data group, S represents the total number of partitions of the topic to which the target data group belongs, N1 represents the quota parameter of the target data group, and N represents the sum of the quota parameters of all data groups in the topic to which the target data group belongs.

[0051] As an optional implementation, each data group uniquely corresponds to a message key prefix, and different data groups correspond to different message key prefixes; after the step of allocating multiple data partitions corresponding to the target message topic to each data group based on the quota parameters, the data partition management method also includes: receiving messages to be allocated that match the target message topic; determining the message key prefix of the message to be allocated; and allocating the message to be allocated to the data partition in the target data group based on the message key prefix through the partition interface constructed by the custom partitioner, wherein the message key prefix of the target data group is consistent with the message key prefix of the message to be allocated.

[0052] In some embodiments of the present disclosure, when receiving a message, all partitions contained in the matching topic are first obtained, a partition corresponding to the quota is allocated to each data group according to the allocation ratio, and the data group and its corresponding partition are stored together in a Map structure. After obtaining the mapping relationship between the data group and the partition, when a message sent by a production object is received, as long as the corresponding topic name is matched and the message key prefix matches the data group logical identifier (i.e., the value of the key in the Map), the partition list (i.e., the value of the value corresponding to the current key in the Map) can be retrieved according to the stored relationship and the message loop can be allocated to it.

[0053] Step S206: Allocate multiple data consumer objects corresponding to the target message topic to each data group according to the quota parameter;

[0054] In the technical solution provided in step S206, the step of allocating multiple data consumption objects corresponding to the target message topic to each data group based on the quota parameters includes: establishing a custom partition matcher (CustomPriorityPartitionAssignor), wherein the custom partition matcher is used to allocate corresponding partitions to data consumption objects; determining the number of data consumption objects allocated to each data group based on the quota parameters through the custom partition matcher; and allocating the number of data consumption objects corresponding to each data group to each data group through the custom partition matcher.

[0055] Specifically, a CustomPriorityPartitionAssignor object is constructed and the Configurable interface is implemented to complete the self-configuration construction operation. The CustomPriorityPartitionAssignor is used to store the topic name that needs to be matched, the partition allocation ratio, etc., and to construct a mapping relationship (Map) collection. The Map collection is used to store the relationship between data groups and partitions. When performing partition assignment, the list of consumer objects corresponding to the configured topic is first found, and the corresponding number of consumer objects is allocated to the partition of the corresponding data group according to the allocation ratio. Here, the number of consumer objects in the data group must be less than or equal to the number of partitions in the data group. The assigned consumer objects are polled for their designated partitions to ensure that the consumer objects of the current data group are evenly distributed among the partitions of the current data group.

[0056] There are two ways to determine the number of data consumer objects allocated to each data group based on quota parameters through a custom partition matcher. One is to determine the allocation ratio of each data group, and the other is to allocate a fixed number of consumer objects to each data group.

[0057] Determining the allocation ratio of each data group refers to determining the allocation ratio of each data group based on quota parameters through a custom partition matcher, where the allocation ratio of each data is equal to the ratio of the quota parameters corresponding to each data group to the sum of the quota parameters corresponding to all data groups; determining the number of first data consumer objects of the data consumer objects corresponding to the target message topic; and determining the number of second data consumer objects corresponding to each data group based on the allocation ratio and the number of first data consumer objects.

[0058] The specific formula for allocating the corresponding quota consumption objects to the partitions of the corresponding data group according to the configured proportion is as follows:

[0059] Among them, C1 represents the number of consumer objects of the target data group, C represents the total number of consumer objects of the topic to which the target data group belongs, N1 represents the quota parameter of the target data group, and N represents the sum of the quota parameters of all data groups in the topic to which the target data group belongs.

[0060] Allocating a fixed number of consumer objects to each data group refers to determining the number of third data consumer objects corresponding to each data group based on a quota parameter, wherein the larger the quota parameter corresponding to the data group, the greater the number of third data consumer objects corresponding to the data group.

[0061] Through the above steps, it is possible to store messages in a corresponding number of data partitions according to the specified ratio, consume messages according to the specified ratio, and achieve a priority effect with high ratio priority consumption.

[0062] The following describes in detail the principles used in managing data partitions in conjunction with Figure 3. Based on the principle shown in Figure 3, the message sent by the producer is received by the custom partitioner, and then logical matching is performed, including initializing configuration information, calculating and storing the allocation ratio, storing the topic name to be matched, etc. According to the initialized self-configuration information, the partitions of the topic are grouped according to the configured message key prefix. Each group is recorded as a pgroup. The number of scores corresponding to each pgroup is allocated according to the self-configured storage ratio. The storage ratio is obtained by dividing the data group's own quota by the quota of all data groups. When the topic name is completely matched and the message key prefix matches, the messages under the topic are cyclically stored in the partition of the corresponding pgroup, so that a high quota obtains a large number of partitions, giving priority to message storage. Accordingly, the data group is matched with a consumer object with the same number of partitions as the data group, achieving a high-ratio priority consumption and a priority effect.

[0063] Take the topic as an example, combined with Figure 3, the topic has three priorities and the quota parameters are {"one": 50, "two": 30, "three": 20}, then the data partitions can be divided into three groups: pgroup-one, pgroup-two, and pgroup-three. The allocation ratio corresponding to each data group is obtained according to the quota parameters, where the allocation ratio is equal to the ratio of the quota parameter corresponding to each data group to the sum of the quota parameters corresponding to all data groups, then pgroup-one: pgroup-two: pgroup-three = 0.5:0.3:0.2; at this time, the total number of data partitions corresponding to the target message topic is set to 10, that is, the number of first data partitions is 10, and the second number of each data group is determined according to the allocation ratio and the number of first data partitions. The number of data partitions is the number of data partitions allocated to each data group based on the allocation ratio. In this case, pgroup-one is allocated 5 partitions, pgroup-two is allocated 3 partitions, and pgroup-three is allocated 2 partitions. The total number of consumers is set to 10, meaning the first number of consumers for the target message topic is 10. The allocation ratio for each data group is determined based on the quota parameters. The second number of consumers for each data group is determined based on the allocation ratio and the first number of consumers. The second number of consumers is the number of consumers for each data group. In this case, pgroup-one is allocated 5 consumers, pgroup-two is allocated 3 consumers, and pgroup-three is allocated 2 consumers. When a message with the topic name "order" and the message key prefix "one" is received, it is sequentially placed into the five partitions of pgroup-one. Five consumers will then be responsible for consuming these five partitions. The same applies to the remaining groups.

[0064] Another implementation method for determining the number of data consumer objects allocated to each data group based on quota parameters is to assign a fixed number of consumer objects to each data group. Specifically, the number of third data consumer objects corresponding to each data group is determined based on the quota parameters. The larger the quota parameter for a data group, the greater the number of third data consumer objects corresponding to that data group. This means that the fixed number of consumer objects is allocated based solely on the size of the allocation ratio, rather than based on a specific allocation ratio. Specifically, according to the above embodiment, pgroup-one can be assigned 6 consumer objects, pgroup-two can be assigned 3 consumer objects, and pgroup-three can be assigned 1 consumer object.

[0065] In addition, it is necessary to specify that after allocating the corresponding quota of data partitions to each data group according to the allocation ratio, the message key prefix and the allocated data partition of the corresponding data group are stored in the Map, and the number of data partitions of the topic at that time is recorded. If the number of data partitions changes subsequently, the data partitions need to be reallocated and the Map updated. If there is no change, the Map will not be changed.

[0066] An embodiment of the present disclosure provides a data partition management device, and Figure 4 is a structural schematic diagram of the device. As shown in Figure 4, the device includes: a first processing module 40, used to determine a target message topic, and multiple data groups corresponding to the target message topic, wherein each data group in the multiple data groups corresponds to a quota parameter, and the quota parameter is used to determine the allocation ratio corresponding to each data group; a second processing module 42, used to allocate multiple data partitions corresponding to the target message topic to each data group based on the quota parameter; a third processing module 44, used to allocate multiple data consumption objects corresponding to the target message topic to each data group based on the quota parameter.

[0067] According to a specific embodiment of the present disclosure, it is necessary to explain in detail that the first processing module 40 is used to determine the target message topic and the multiple data groups corresponding to the target message topic, specifically including: establishing a custom partitioner to divide the data partitions, and obtaining and parsing the configuration parameters through the custom partitioner to obtain own parameters, wherein the own parameters include the target message topic, the prefix of each data group, and the quota parameters corresponding to each data group; determining the target message topic and multiple data groups based on the own parameters through the custom partitioner. Among them, determining the target message topic, that is, determining the topic of the message sent by the producer, determining the number of data groups corresponding to the target message topic, each data group has a different logical identifier, and when storing messages, it is necessary to match the logical identifier of the data group according to the message key prefix. Each data group has a quota parameter, and the quota parameter is used to determine the allocation ratio corresponding to each data group.

[0068] According to a specific embodiment of the present disclosure, it is necessary to explain in detail that the second processing module 42 is used to determine the number of first data partitions corresponding to the target message topic; determine the allocation ratio corresponding to each data group based on the quota parameter, wherein the allocation ratio corresponding to each data group is equal to the ratio of the quota parameter corresponding to each data group to the sum of the quota parameters corresponding to all data groups; determine the number of second data partitions allocated to each data group based on the allocation ratio and the number of first data partitions, i.e., the number of partitions allocated to each data group according to the allocation ratio; and divide the data partitions of the second data partition into the data groups using a custom partitioner. Each data group uniquely corresponds to a message key prefix, and different data groups correspond to different message key prefixes.

[0069] After the step of allocating multiple data partitions corresponding to the target message topic to each data group according to the quota parameters, the second processing module 42 is also used to: receive the message to be allocated that matches the target message topic, then determine the message key prefix of the message to be allocated, and then allocate the message to be allocated to the data partition in the target data group according to the message key prefix through the partition interface constructed by the custom partitioner, wherein the message key prefix of the target data group is consistent with the message key prefix of the message to be allocated.

[0070] According to a specific embodiment of the present disclosure, it is necessary to explain in detail that the third processing module 44 is used to allocate multiple data consumption objects corresponding to the target message topic to each data group based on the quota parameters. The step of allocating multiple data consumption objects corresponding to the target message topic to each data group based on the quota parameters includes: establishing a custom partition matcher, the custom partition matcher is used to allocate corresponding partitions to the data consumption objects, determining the number of data consumption objects allocated to each data group based on the quota parameters by the custom partition matcher, and then allocating the data consumption objects corresponding to the number of data consumption objects of each data group to each data group by the custom partition matcher. The step of determining the number of data consumption objects allocated to each data group based on the quota parameters by the custom partition matcher includes: determining the allocation ratio of each data group based on the quota parameters by the custom partition matcher, wherein the allocation ratio in each data is equal to the ratio of the quota parameters corresponding to each data group to the sum of the quota parameters corresponding to all data groups, then determining the first number of data consumption objects of the data consumption objects corresponding to the target message topic based on the allocation ratio, and finally determining the second number of data consumption objects corresponding to each data group based on the allocation ratio and the first number of data consumption objects. The number of third data consumer objects corresponding to each data group is determined based on a quota parameter, wherein the larger the quota parameter corresponding to a data group, the greater the number of third data consumer objects corresponding to that data group. The first number of data consumer objects is the total number of consumer objects for the subject, the second number of data consumer objects is the number of consumer objects allocated to each data group according to an allocation ratio, and the third number of data consumer objects is a fixed number of consumer objects set for each data group based on the allocation ratio. The higher the allocation ratio, the greater the number of consumer objects allocated, and the sum of the number of consumer objects contained in each data group of the subject does not exceed the total number of consumer objects for the subject.

[0071] It should be noted that the various modules in the above-mentioned data partition management device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0072] According to an embodiment of the present disclosure, an embodiment of a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following data partition management method: determining a target message topic, and multiple data groups corresponding to the target message topic, wherein each of the multiple data groups corresponds to a quota parameter, and the quota parameter is used to determine the allocation ratio corresponding to each data group; based on the quota parameter, allocating multiple data partitions corresponding to the target message topic to each data group; based on the quota parameter, allocating multiple data consumption objects corresponding to the target message topic to each data group.

[0073] According to an embodiment of the present disclosure, an electronic device is provided. The electronic device includes a memory and a processor, the processor being configured to run a program stored in the memory, wherein when the program is run, the following data partition management method is executed: determining a target message topic and multiple data groups corresponding to the target message topic, wherein each of the multiple data groups corresponds to a quota parameter, and the quota parameter is used to determine an allocation ratio corresponding to each data group; allocating multiple data partitions corresponding to the target message topic to each data group based on the quota parameter; and allocating multiple data consumption objects corresponding to the target message topic to each data group based on the quota parameter.

[0074] In the above embodiments of the present disclosure, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0075] In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0076] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0077] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0079] The above is only a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications should also be regarded as within the scope of protection of the present disclosure.

Claims

1. A data partition management method, comprising: Determine a target message subject of a message to be allocated, and a plurality of data groups corresponding to the target message subject, wherein each of the plurality of data groups corresponds to a quota parameter, and the quota parameter is used to determine an allocation ratio corresponding to each of the data groups; Allocating a plurality of data partitions corresponding to the target message topic to each of the data groups according to the quota parameters; According to the quota parameters, multiple data consumption objects corresponding to the target message topic are allocated to each of the data groups.

2. The data partition management method according to claim 1, wherein: The determining of the target message subject and the multiple data groups corresponding to the target message subject include: Establishing a custom partitioner, wherein the custom partitioner is used to divide the data partition; Acquire and parse configuration parameters through the custom partitioner to obtain the target message topic, the message key prefix of each data group, and the quota parameter corresponding to the message key prefix; The data groups corresponding to the respective message key prefixes are determined by the custom partitioner.

3. The data partition management method according to claim 2, wherein: The allocating the multiple data partitions corresponding to the target message topic to each of the data groups according to the quota parameter comprises: Determine the number of first data partitions corresponding to the target message topic; Determining the allocation ratio corresponding to each of the data groups according to the quota parameters; Determining the number of second data partitions allocated to each of the data groups according to the allocation ratio and the number of the first data partitions; The data partitions of the second number of data partitions are divided into the data groups by the custom partitioner.

4. The data partition management method according to claim 2, wherein: The allocation ratio corresponding to each of the data groups is equal to the ratio of the quota parameter corresponding to each of the data groups to the sum of the quota parameters corresponding to all of the data groups.

5. The data partition management method according to claim 2, wherein: Each of the data groups uniquely corresponds to a message key prefix, and different data groups correspond to different message key prefixes.

6. The data partition management method according to claim 5, further comprising: After allocating the multiple data partitions corresponding to the target message topic to each of the data groups according to the quota parameters, Receiving the message to be assigned that matches the subject of the target message; Determining the message key prefix of the message to be assigned; The partition interface constructed by the custom partitioner distributes the message to be distributed to the data partition in the target data group according to the message key prefix, wherein the message key prefix of the target data group is consistent with the message key prefix of the message to be distributed.

7. The data partition management method according to any one of claims 1 to 6, wherein: The allocating the multiple data consumption objects corresponding to the target message topic to each of the data groups according to the quota parameter comprises: Establishing a custom partition matcher, wherein the custom partition matcher is used to assign corresponding partitions to the data consumption object; Determine the number of the data consumption objects allocated to each of the data groups according to the quota parameters by the custom partition matcher; The data consumption objects of the number of data consumption objects corresponding to each of the data groups are allocated to each of the data groups through the custom partition matcher.

8. The data partition management method according to claim 7, wherein: Determining the number of data consumption objects allocated to each of the data groups according to the quota parameters by the custom partition matcher includes: Determining the allocation ratio of each of the data groups according to the quota parameters by the custom partition matcher; Determine the first data consumer object quantity of the data consumer object corresponding to the target message topic; The number of second data consumer objects corresponding to each of the data groups is determined according to the allocation ratio and the number of the first data consumer objects.

9. The data partition management method according to claim 8, wherein: The allocation ratio in each of the data is equal to the ratio of the quota parameter corresponding to each of the data groups to the sum of the quota parameters corresponding to all of the data groups.

10. The data partition management method according to claim 8, wherein: Determining the number of data consumption objects allocated to each of the data groups according to the quota parameters by the custom partition matcher includes: The number of third data consumer objects corresponding to each of the data groups is determined according to the quota parameter, wherein the larger the quota parameter corresponding to the data group is, the greater the number of third data consumer objects corresponding to the data group is.

11. The data partition management method according to any one of claims 1 to 10, wherein: The number of the data partitions corresponding to the same data group is greater than or equal to the number of the corresponding data consumption objects.

12. A data partition management device, comprising: A first processing module is used to determine a target message subject and a plurality of data groups corresponding to the target message subject, wherein each of the plurality of data groups corresponds to a quota parameter, and the quota parameter is used to determine an allocation ratio corresponding to each of the data groups; A second processing module, configured to allocate a plurality of data partitions corresponding to the target message topic to each of the data groups according to the quota parameter; The third processing module is used to allocate multiple data consumption objects corresponding to the target message topic to each of the data groups according to the quota parameters.

13. A non-volatile storage medium, wherein a program is stored in the non-volatile storage medium, wherein: When the program is running, the device where the non-volatile storage medium is located is controlled to execute the data partition management method described in any one of claims 1 to 11.

14. An electronic device comprising: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the data partition management method described in any one of claims 1 to 11 when running.

15. A computer program comprising computer instructions, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images