Integrated order information sorting method and system based on Kafka

Abstract

The invention provides a Kafka-based integrated order message sorting method and system, an electronic device and a storage medium, and aims to solve the problem that messages in different partitions of Kafka messages may be sent and received out of order, so that the priority of an integrated order cannot be considered in-topic or cross-topic sequencing. The method comprises the following steps: receiving an integrated order and a topic adapted to a collection service; constructing a Kafka priority queue model and a service lock model; and combining a Kafka priority queue with a service lock strategy, establishing a message processing transmission strategy considering priority and sorting, solving a multi-objective optimization problem, generating a message control mechanism according to a solving result, and distributing an integrated order message. According to the method and the device, the priority of the Kafka message can be realized, the in-topic or cross-topic sequence is considered, and the balance among the priority, the service sequence, the service growth and the system performance is obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a Kafka-based integrated order message sorting method, a Kafka-based integrated order message sorting system, an electronic device and a computer-readable storage medium. Background Art

[0002] The telecom operator integrated order system is an important business support system for telecom operators. Its core function is to receive business orders from the business acceptance system, adapt the orders to business activation orders that can be operated on the back-end platform based on the relationship between the orders and customer products and services, split the orders into executable instructions and send them to each network element platform for activation and other business activation work.

[0003] Kafka excels in real-time processing, high throughput, and scalability. It is an indispensable component in modern data architecture and can provide enterprises with powerful data stream processing capabilities and flexible data integration solutions. It is adopted by telecom operators' integrated order systems.

[0004] Kafka is a distributed stream processing platform designed for high throughput, durability, scalability, replicability, real-time processing capabilities, publish / subscribe mode, seamless integration with the big data ecosystem, and a rich ecosystem. It is widely used to build real-time data pipelines and streaming applications.

[0005] 1) Kafka was designed with high throughput and real-time processing capabilities in mind, and is suitable for application scenarios that require large-scale data processing and real-time data processing; 2) Kafka's distributed architecture allows the cluster to be expanded horizontally by adding nodes, which can easily cope with the growth of data volume and traffic. Topics can be divided into multiple partitions, and partitions can be distributed on different servers to improve parallel processing capabilities; 3) Kafka writes messages to partition files in the form of logs, writes in an append manner, and ensures high data availability through a replication mechanism; 4) Kafka supports the production and consumption of real-time data streams. Combined with stream processing tools such as Kafka Streams and KSQL, complex real-time data processing and analysis can be achieved; 5) Kafka implements a flexible publish / subscribe message model. Producers are responsible for sending messages to specified topics, and consumers consume messages through subscriptions. It also supports multiple consumption modes such as consumer groups to achieve load balancing and high availability;

[0006] Kafka messages are stored in partitions. Each topic can have multiple partitions. It only guarantees the order within the partition, that is, messages within the same partition are sent and received in the order of Offset. However, the current Kafka message sending does not guarantee the order within or across topics, that is, messages in different partitions may be sent and received out of order. This enables it to execute read and write request processing for each partition, achieve high scalability of the system, and achieve a large number of consumer subscriptions, but the pull model based on Offset is in conflict with the priority attribute requirement of the integrated order. Therefore, how to achieve the priority of the integrated order while taking into account the order within or across topics has become a problem that needs to be solved urgently. Summary of the invention

[0007] In order to at least solve the problem in the prior art that messages in different partitions of Kafka messages may be sent and received out of order, and it is impossible to achieve the problem that the priority of integrated orders can take into account the order within or across topics. The present disclosure provides a method for sorting integrated order messages based on Kafka, a system for sorting integrated order messages based on Kafka, an electronic device, and a computer-readable storage medium, which can achieve the priority of Kafka messages taking into account the order within or across topics, and achieve a balance between priority, business order, business growth, and system performance.

[0008] In a first aspect, the present disclosure provides a method for sorting integrated order messages based on Kafka, the method comprising:

[0009] Receive integrated orders and collect topics adapted for business;

[0010] Build the Kafka priority queue model and business lock model;

[0011] Combine the Kafka priority queue with the business lock strategy to establish a message processing and transmission strategy that takes priority and sorting into account, solve multi-objective optimization problems, and generate a message control mechanism according to the solution results to distribute integrated order messages.

[0012] Furthermore,

[0013] The integrated order includes a letter control integrated order and a business integrated order;

[0014] The topics adapted by the collection service include cities, order types, telecommunication networks, and special attribute flags.

[0015] Furthermore, the working process of the Kafka priority queue model includes:

[0016] Create multiple topics, one topic represents one priority, and each topic is divided into multiple partitions;

[0017] Construct a kjPriorityConsumer that aggregates multiple priority consumers;

[0018] According to the business situation, set the number of messages pulled by kjPriorityConsumer each time, so that the consumer consumes the data in the corresponding partition in order from high to low priority.

[0019] Furthermore, the business lock model working process includes:

[0020] S1: Set key values ​​for redis: number key value SERIAL_NUM, number lock key value MSISDN_LOCK;

[0021] S2: parse the received message and determine whether there is a lock in redis based on the number key value SERIAL_NUM;

[0022] S3: If the number is not locked, lock the number and then send a message; and,

[0023] S4: After the entire work order is executed, redis is unlocked;

[0024] S5: If there is a lock, send the message to the first delay queue; and,

[0025] S6: Parse the message received by the first delay queue and check the redis result. If it is not unlocked, the message will continue to be thrown into the first delay queue and wait; if it is unlocked, continue S3 operation.

[0026] Furthermore,

[0027] The key value set by redis also includes the number office key value MSISDN_SWITCH_LOCK;

[0028] The business lock model working process also includes:

[0029] In S6, check the redis result. If it is unlocked, query the order to be executed according to the number and send it to the acquisition queue;

[0030] Parse the message received by the second delay queue, detect the redis lock, and send it to the waiting queue if it is unlocked, otherwise throw it back to the queue and wait;

[0031] Parse the data of the message received by the queue to be sent, check whether there is a direction lock in the redis node according to the number key value and the number direction key value, and send it to the second delay queue if there is one;

[0032] If not, lock the number key value and the number office key value, and execute the work order to interact with the exchange side;

[0033] Parse the interactive return results, update the work order status, and unlock the number key value and number office key value in redis.

[0034] Furthermore, the establishment of a message processing and transmission strategy taking into account priorities and sorting, and solving a multi-objective optimization problem, includes:

[0035] For the received integrated order message, check whether there is a lock in redis based on the number key value SERIAL_NUM;

[0036] If there is a lock, the message is sent to the first delay queue and the redis result is checked;

[0037] If the redis result is not unlocked, it will continue to be thrown into the first delay queue and wait; if it is unlocked or not locked, the matched business adaptation topic will be obtained, the number will be locked and sent to the corresponding priority topic; the consumer consumes the data in the corresponding partition in order from high to low priority according to the number of messages pulled each time by the kjPriorityConsumer set according to the business situation, and unlocks the number after completing the business logic processing.

[0038] In a second aspect, the present disclosure provides an integrated order message sorting system based on Kafka, the system comprising:

[0039] A receiving module, which is configured to receive integrated orders and collect topics adapted for business;

[0040] Construction module, which is set to build Kafka priority queue model and business lock model;

[0041] The sorting module is configured to combine the Kafka priority queue with the business lock strategy, establish a message processing and transmission strategy that takes priority and sorting into account, solve multi-objective optimization problems, and generate a message control mechanism according to the solution results to distribute integrated order messages.

[0042] Furthermore,

[0043] The working process of the Kafka priority queue model includes:

[0044] Create multiple topics, one topic represents one priority, and each topic is divided into multiple partitions;

[0045] Construct a kjPriorityConsumer that aggregates multiple priority consumers;

[0046] Set the number of messages pulled by kjPriorityConsumer each time according to the business situation, so that the consumer consumes the data in the corresponding partition in order from high to low priority;

[0047] The working process of the business lock model includes:

[0048] S1: Set key values ​​for redis: number key value SERIAL_NUM, number lock key value MSISDN_LOCK;

[0049] S2: parse the received message and determine whether there is a lock in redis based on the number key value SERIAL_NUM;

[0050] S3: If the number is not locked, lock the number and then send a message; and,

[0051] S4: After the entire work order is executed, redis is unlocked;

[0052] S5: If there is a lock, send the message to the first delay queue; and,

[0053] S6: Parse the message received by the first delay queue and check the redis result. If it is not unlocked, the message will continue to be thrown into the first delay queue and wait; if it is unlocked, continue S3 operation.

[0054] In a third aspect, the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the Kafka-based integrated order message sorting method as described in any one of the first aspects.

[0055] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the integrated order message sorting method based on Kafka described in any one of the above-mentioned first aspects is implemented.

[0056] Beneficial effects:

[0057] The present disclosure provides a method for sorting integrated order messages based on Kafka, a system for sorting integrated order messages based on Kafka, an electronic device and a storage medium; the Kafka priority queue model is used to send Kafka messages according to priority, and the business lock model is used to achieve the orderliness of messages within or across topics, thereby achieving a balance between priority, business order, business growth and system performance. Prior information on the priorities of multiple online business clusters is not required, which is convenient for front-end business growth and cluster expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart of a method for sorting integrated order messages based on Kafka provided in the first embodiment of the present disclosure;

[0059] Figure 2 A schematic diagram of an integrated order message sorting strategy provided by an embodiment of the present disclosure;

[0060] Figure 3 A flow chart of an integrated order message sorting method taking into account Kafka priority queue and business lock provided in the second embodiment of the present disclosure;

[0061] Figure 4 An architecture diagram of an integrated order message sorting system based on Kafka provided in Embodiment 3 of the present disclosure;

[0062] Figure 5 This is an architecture diagram of an electronic device provided in Embodiment 4 of the present disclosure. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention, rather than to limit the present invention.

[0064] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.

[0065] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.

[0066] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0067] The following is a detailed description of the technical solution of the present invention and how the technical solution of the present invention solves the problems existing in the prior art with specific embodiments. It is understandable that in the embodiments of the present application, the execution subject can perform some or all of the steps in the embodiments of the present application, and these steps or operations are only examples. The embodiments of the present application can also perform other operations or deformations of various operations. In addition, each step can be performed in a different order presented in the embodiments of the present application, and it is possible not to perform all the operations in the embodiments of the present application. And, the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0068] Figure 1 A flow chart of a method for sorting integrated order messages based on Kafka is provided in the first embodiment of the present disclosure, as shown in FIG. Figure 1 As shown, the method includes:

[0069] Step S101: receiving an integrated order and collecting topics adapted for business;

[0070] Step S102: constructing a Kafka priority queue model and a business lock model;

[0071] Step S103: Combine the Kafka priority queue with the business lock strategy, establish a message processing and transmission strategy that takes priority and sorting into consideration, solve the multi-objective optimization problem, and generate a message control mechanism according to the solution results to distribute the integrated order message.

[0072] The purpose of the embodiments of the present disclosure is to provide an integrated order (also referred to as integrated order) message sorting method that takes into account high throughput, priority queues and business sorting, and combines Kafka priority queues with business lock strategies to optimize the optimal sending of integrated order messages.

[0073] Specifically, we receive topics adapted for integration orders and collection services, and pre-build Kafka priority queue models and business lock models. For topics adapted for collection services, we build a kjPriorityConsumer that aggregates multiple priority consumers through the Kafka priority queue model. We use Kafka's native topic and partition mechanism to achieve the effect of high-priority messages being consumed first. In order to send messages in different partitions or topics in a certain order, we use the business lock model to send only specific unlocked messages when sending messages, so that messages can be sent in a specific order. For example, we execute messages corresponding to the same number in the integration order message before executing messages of other numbers to ensure the order of messages. We combine the Kafka priority queue with the business lock strategy to establish a message processing and transmission strategy that considers priority and sorting, and solve multi-objective optimization problems. Through multi-objective optimization problems, we use the Kafka priority queue and business lock strategy to achieve a balance between priority, business order, business growth and system performance while ensuring the order of business messages. We combine the two to achieve priority and order within or across topics for Kafka messages.

[0074] Furthermore,

[0075] The integrated order includes a letter control integrated order and a business integrated order;

[0076] The topics adapted by the collection service include cities, order types, telecommunication networks, and special attribute flags.

[0077] Integrated orders are the integration of order information from different channels into a centralized platform through technical means for unified management and analysis. This process involves the standardization and integration of order data. For credit control integrated orders and business integrated orders, the corresponding topics can be determined according to the order characteristics.

[0078] Furthermore, the working process of the Kafka priority queue model includes:

[0079] Create multiple topics, one topic represents one priority, and each topic is divided into multiple partitions;

[0080] Construct a kjPriorityConsumer that aggregates multiple priority consumers;

[0081] According to the business situation, set the number of messages pulled by kjPriorityConsumer each time, so that the consumer consumes the data in the corresponding partition in order from high to low priority.

[0082] For the number of messages pulled by kjPriorityConsumer each time, different quotas and adaptive adjustments can be set for consumers of different priorities. For example, the default setting is to pull 500 messages at a time, and 500 messages are pulled from high priority to low priority. Assuming there are three priorities (1, 2, 3), the quota can be set to (300, 150, 50). Then, this threshold can be adaptively adjusted based on the message delivery situation.

[0083] By collecting topics adapted to the business, a kjPriorityConsumer is constructed to aggregate consumers of multiple priorities. The native topic and partition mechanism of Kafka is used, which is simple and easy to implement, and ultimately achieves the effect of giving priority to the consumption of high-priority messages.

[0084] Furthermore, the business lock model working process includes:

[0085] S1: Set key values ​​for redis: number key value SERIAL_NUM, number lock key value MSISDN_LOCK;

[0086] S2: parse the received message and determine whether there is a lock in redis based on the number key value SERIAL_NUM;

[0087] S3: If the number is not locked, lock the number and then send a message; and,

[0088] S4: After the entire work order is executed, redis is unlocked;

[0089] S5: If there is a lock, send the message to the first delay queue; and,

[0090] S6: Parse the message received by the first delay queue and check the redis result. If it is not unlocked, the message will continue to be thrown into the first delay queue and wait; if it is unlocked, continue S3 operation.

[0091] To determine whether there is a lock in redis, you can use the key value (key: MSISDN_LOCK) to determine. If redis is locked, it means that the message corresponding to the current number is locked and cannot be sent temporarily. You need to wait for redis to be unlocked, that is, all the corresponding messages of the currently unlocked number have been sent, then unlock the next number, and determine the message that can be sent by checking the redis (key: MSISDN_LOCK) result. In this way, all messages for the same number can be sent in sequence.

[0092] Furthermore,

[0093] The key value set by redis also includes the number office key value MSISDN_SWITCH_LOCK;

[0094] The business lock model working process also includes:

[0095] In S6, check the redis result. If it is unlocked, query the order to be executed according to the number and send it to the acquisition queue;

[0096] Parse the message received by the second delay queue, detect the redis lock, and send it to the waiting queue if it is unlocked, otherwise throw it back to the queue and wait;

[0097] Parse the data of the message received by the queue to be sent, check whether there is a direction lock in the redis node according to the number key value and the number direction key value, and send it to the second delay queue if there is one;

[0098] If not, lock the number key value and the number office key value, and execute the work order to interact with the exchange side;

[0099] Parse the interactive return results, update the work order status, and unlock the number key value and number office key value in redis.

[0100] In the integrated order system of telecom operators, multiple business lock controls can also be used to perform lock-in-lock control. For example, if a number has multiple orders, an order can be split into multiple instruction work orders. Set the number key value SERIAL_NUM, the number lock key value MSISDN_LOCK, and the number switch key value MSISDN_SWITCH_LOCK: The specific functions of the business lock model are:

[0101] (1) Parse the data of the message received by topic-GET_TOPIC, check whether there is a lock in the redis node (key: MSISDN_LOCK) according to SERIAL_NUM, and send it to topic-GET_DELAY_TOPIC (i.e. the first delay queue) if there is a lock;

[0102] 1.1 Unlock asynchronous call service, split the order into work orders according to the splitting rules, set the order to be processed, query the work order by number and order number, set work order A to be processed by the time sequence field and the field of whether there is a dependent node of the split work order, and then lock the processed number (MSISDN_LOCK), and finally send the executable node to topic-SEND_TOPIC through Kafka;

[0103] 1.2 Analyze the data of the success message received by STEP_TOPIC, update the completion status of work order A, and mark the status of the next work order;

[0104] 1.3 Set the executable work order modification status to processing, and then send it to topic-SEND_TOPIC;

[0105] 1.4 If the entire order is processed, it is sent to topic-FIN_TOPIC;

[0106] 1.5 Parse the message received from topic-FIN_TOPIC, modify the status and time of the order, and unlock the redis node (key: wpa:MSISDN_LOCK)

[0107] (2) When there is a lock, parse the message received by GET_DELAY_TOPIC and check the result of the redis node (key: MSISDN_LOCK). If it is not unlocked, continue to throw it into the queue topic-GET_DELAY_TOPIC;

[0108] 2.1 The order to be executed according to the number query has been unlocked and sent to topic-GET_TOPIC;

[0109] 2.2 Parse the message received from topic-STEP_DELAY_TOPIC (i.e. the second delay queue), detect the redis lock, and send it to topic-SEND_TOPIC if it is unlocked, otherwise throw it back to the queue and sleep;

[0110] (3) Parse the data of the message received by topic-SEND_TOPIC, and check whether there is a local lock in the redis node (key: MSISDN_SWITCH_LOCK) based on SERIAL_NUM+switchId. If there is, send it to topic-STEP_DELAY_TOPIC;

[0111] 3.1 If not, lock SERIAL_NUM+switchId (key: MSISDN_SWITCH_LOCK) and execute the work order to interact with the switch side;

[0112] 3.2 Parse the interaction return results, update the work order status, and unlock two redis keys (key: MSISDN_LOCK, key: MSISDN_SWITCH_LOCK).

[0113] When splitting a work order, there may be multiple network elements. For example, when opening a mobile network account, HSS network element, AS network element, ENUM network element, VPN network element and other network elements are sent sequentially or in parallel. At this time, it is necessary to add the number office key SISDN_SWITCH_LOCK (Switch for short) under the number lock condition to divide the message and send it by network element (only after this network element message is sent, other network element messages can be sent). By adding the number office key, on the basis of sending messages in sequence by number, it is also possible to send messages in the order of network elements for the same number.

[0114] Furthermore, the establishment of a message processing and transmission strategy taking into account priorities and sorting, and solving a multi-objective optimization problem, includes:

[0115] For the received integrated order message, check whether there is a lock in redis based on the number key value SERIAL_NUM;

[0116] If there is a lock, the message is sent to the first delay queue and the redis result is checked;

[0117] If the redis result is not unlocked, it will continue to be thrown into the first delay queue and wait; if it is unlocked or not locked, the matched business adaptation topic will be obtained, the number will be locked and sent to the corresponding priority topic; the consumer consumes the data in the corresponding partition in order from high to low priority according to the number of messages pulled each time by the kjPriorityConsumer set according to the business situation, and unlocks the number after completing the business logic processing.

[0118] Integrate order message sequencing strategies such as Figure 2 As shown in the schematic diagram, for each integrated order message received, check whether there is a lock (key: MSISDN_LOCK) in redis; if there is a lock, send it to the delay queue, and check whether it is unlocked according to the set rules (such as receiving a message or periodically). If it is not unlocked, continue to throw it into the first delay queue and wait; if it is unlocked, get the matched business adaptation topic for Kafka priority queue matching; if there is no lock, match the priority according to the Kafka priority queue model, get the matched business adaptation topic, lock the number and send it to the corresponding priority topic, kjPriorityConsumer consumes it in order from high to low priority, and unlocks the number after the business logic processing is completed.

[0119] The disclosed embodiment realizes the orderliness of messages within or across topics through the business lock model, and sends the messages that need to be sorted to the DELAY queue by combining redis to set the sort key value and redis to set the lock key value; and realizes the sending of Kafka messages according to priority through the Kafka priority queue model, thereby achieving a balance between priority, business order, business growth and system performance. There is no need for prior information on the priorities of multiple simultaneously online business clusters, which facilitates the growth of front-end business and cluster expansion. There are obvious advantages in the field of integrated orders of telecom operators with real-time processing, low latency and high throughput characteristics.

[0120] The second embodiment of the present disclosure also provides an integrated order message sorting method that takes into account high throughput, priority queues and business sorting, and combines Kafka priority queues with business lock strategies to optimize the integrated order message optimal sending purpose, such as Figure 3 As shown, the specific steps are as follows:

[0121] In step S201, an integrated order and a collection service adapted topic are received.

[0122] The integrated orders include letter control integrated orders and business integrated orders;

[0123] The topics adapted by the collection service include cities, order types, telecommunication networks, and special attribute flags.

[0124] In step S202, a Kafka priority queue model and a business lock model are established.

[0125] The Kafka priority queue model construction method is as follows:

[0126] 1) Create multiple topics, one topic represents one priority, and each topic is divided into multiple partitions;

[0127] 2) Construct a kjPriorityConsumer to aggregate consumers with multiple priorities;

[0128] 3) Set the number of messages pulled by kjPriorityConsumer each time according to business conditions;

[0129] 4) Consumers consume data in the corresponding partition in descending order of priority;

[0130] Preferably, the present disclosure can also set different quotas and adaptive adjustments for the number of messages pulled each time by kjPriorityConsumer for consumers of different priorities, such as a default setting of 500 messages pulled at a time, and 500 messages pulled from high priority to low priority. Assuming there are three priorities (1, 2, 3), the quota can be set to (300, 150, 50), and then the threshold can be adaptively adjusted according to the message delivery situation.

[0131] The business lock model construction method is as follows:

[0132] 1) Redis sets the key value: number key value SERIAL_NUM, number lock key value MSISDN_LOCK;

[0133] 2) After receiving the message, the data is parsed and the number SERIAL_NUM is used to check whether there is a lock in redis (key: MSISDN_LOCK);

[0134] 3) If the number is unlocked, lock it and send a message;

[0135] 4) After the entire work order is executed, unlock redis (key: MSISDN_LOCK);

[0136] 5) If there is a lock, the message is sent to the DELAY queue;

[0137] 6) Check the result of redis (key: MSISDN_LOCK). If it is not unlocked, continue to put it into the DELAY queue and sleep; if it is unlocked, continue with 3).

[0138] Preferably, multiple service lock controls can be used in the integrated order system of the telecom operator to perform lock-in-lock control. For example, if one number has multiple orders, one order can be split into multiple instruction work orders. Set the number key value SERIAL_NUM, the number lock key value MSISDN_LOCK, and the number switch key value MSISDN_SWITCH_LOCK:

[0139] (1) Parse the data of the message received by topic-GET_TOPIC, and check whether there is a lock in the redis node (key: MSISDN_LOCK) according to SERIAL_NUM. If there is a lock, send it to topic-GET_DELAY_TOPIC;

[0140] 1.1 Unlock asynchronous call service, split the order into work orders according to the splitting rules, set the order to be processed, query the work order by number and order number, set work order A to be processed by the time sequence field and the field of whether there is a dependent node of the split work order, and then lock the processed number (MSISDN_LOCK), and finally send the executable node to topic-SEND_TOPIC through Kafka;

[0141] 1.2 Analyze the data of the success message received by STEP_TOPIC, update the completion status of work order A, and mark the status of the next work order;

[0142] 1.3 Set the executable work order modification status to processing, and then send it to topic-SEND_TOPIC;

[0143] 1.4 If the entire order is processed, it is sent to topic-FIN_TOPIC;

[0144] 1.5 Parse the message received from topic-FIN_TOPIC, modify the status and time of the order, and unlock the redis node (key: wpa:MSISDN_LOCK);

[0145] (2) When there is a lock, parse the message received by GET_DELAY_TOPIC and check the result of the redis node (key: MSISDN_LOCK). If it is not unlocked, continue to throw it into the queue topic-GET_DELAY_TOPIC;

[0146] 2.1 The order to be executed according to the number query has been unlocked and sent to topic-GET_TOPIC;

[0147] 2.2 Parse the message received from topic-STEP_DELAY_TOPIC, detect the redis lock, and send it to topic-SEND_TOPIC if it is unlocked, otherwise throw it back to the queue and sleep;

[0148] (3) Parse the data of the message received by topic-SEND_TOPIC, and check whether there is a local lock in the redis node (key: MSISDN_SWITCH_LOCK) based on SERIAL_NUM+switchId. If there is, send it to topic-STEP_DELAY_TOPIC;

[0149] 3.1 If not, lock SERIAL_NUM+switchId (key: MSISDN_SWITCH_LOCK) and execute the work order to interact with the switch side;

[0150] 3.2 Parse the interaction return results, update the work order status, and unlock two redis keys (key: MSISDN_LOCK, key: MSISDN_SWITCH_LOCK).

[0151] In step S203, the business lock strategy and the Kafka priority queue model are combined to establish an integrated order sorting solution to the multi-objective optimization problem, and the control sorting and priority are generated according to the solution results, so that high-priority messages are consumed first while taking into account the business sorting.

[0152] The multi-objective optimization problem is solved as Figure 2 Specifically shown are:

[0153] After receiving the integrated order message, check whether there is a lock in redis (key: MSISDN_LOCK) according to the number SERIAL_NUM. If there is a lock, send the message to the DELAY queue and check the redis (key: MSISDN_LOCK) result; if it is not unlocked, continue to throw it to the DELAY queue and sleep; if it is unlocked or not locked, get the matched business adaptation topic, lock the number and send it to the corresponding priority topic; the consumer consumes the data in the corresponding partition in descending order according to the priority according to the number of messages pulled each time by the kjPriorityConsumer set according to the business situation, and unlocks the number after completing the business logic processing.

[0154] The third embodiment of the present disclosure also provides an integrated order message sorting system based on Kafka, such as Figure 4 As shown, the system comprises:

[0155] A receiving module 11, which is configured to receive integrated orders and collect topics adapted for business;

[0156] A construction module 12 is configured to construct a Kafka priority queue model and a business lock model;

[0157] The sorting module 13 is configured to combine the Kafka priority queue with the business lock strategy, establish a message processing and transmission strategy that takes priority and sorting into consideration, solve the multi-objective optimization problem, and generate a message control mechanism according to the solution results to distribute the integrated order message.

[0158] Furthermore,

[0159] The integrated order includes a letter control integrated order and a business integrated order;

[0160] The topics adapted by the collection service include cities, order types, telecommunication networks, and special attribute flags.

[0161] Furthermore,

[0162] The working process of the Kafka priority queue model includes:

[0163] Create multiple topics, one topic represents one priority, and each topic is divided into multiple partitions;

[0164] Construct a kjPriorityConsumer that aggregates multiple priority consumers;

[0165] Set the number of messages pulled by kjPriorityConsumer each time according to the business situation, so that the consumer consumes the data in the corresponding partition in order from high to low priority;

[0166] The working process of the business lock model includes:

[0167] S1: Set key values ​​for redis: number key value SERIAL_NUM, number lock key value MSISDN_LOCK;

[0168] S2: parse the received message and determine whether there is a lock in redis based on the number key value SERIAL_NUM;

[0169] S3: If the number is not locked, lock the number and then send a message; and,

[0170] S4: After the entire work order is executed, redis is unlocked;

[0171] S5: If there is a lock, send the message to the first delay queue; and,

[0172] S6: Parse the message received by the first delay queue and check the redis result. If it is not unlocked, the message will continue to be thrown into the first delay queue and wait; if it is unlocked, continue S3 operation.

[0173] Furthermore,

[0174] The key value set by redis also includes the number office key value MSISDN_SWITCH_LOCK;

[0175] The business lock model working process also includes:

[0176] In S6, check the redis result. If it is unlocked, query the order to be executed according to the number and send it to the acquisition queue;

[0177] Parse the message received by the second delay queue, detect the redis lock, and send it to the waiting queue if it is unlocked, otherwise throw it back to the queue and wait;

[0178] Parse the data of the message received by the queue to be sent, check whether there is a direction lock in the redis node according to the number key value and the number direction key value, and send it to the second delay queue if there is one;

[0179] If not, lock the number key value and the number office key value, and execute the work order to interact with the exchange side;

[0180] Parse the interactive return results, update the work order status, and unlock the number key value and number office key value in redis.

[0181] Furthermore, the sorting module 13 is specifically configured as follows:

[0182] For the received integrated order message, check whether there is a lock in redis based on the number key value SERIAL_NUM;

[0183] If there is a lock, the message is sent to the first delay queue and the redis result is checked;

[0184] If the redis result is not unlocked, it will continue to be thrown into the first delay queue and wait; if it is unlocked or not locked, the matched business adaptation topic will be obtained, the number will be locked and sent to the corresponding priority topic; the consumer consumes the data in the corresponding partition in order from high to low priority according to the number of messages pulled each time by the kjPriorityConsumer set according to the business situation, and unlocks the number after completing the business logic processing.

[0185] The Kafka-based integrated order message sorting system of the disclosed embodiment is used to implement the Kafka-based integrated order message sorting method in method embodiment 1 and embodiment 2, so the description is relatively simple. For details, please refer to the relevant description in the previous method embodiment, which will not be repeated here.

[0186] In addition, if Figure 5 As shown, the fourth embodiment of the present disclosure further provides an electronic device, including a memory 100 and a processor 200, wherein the memory 100 stores a computer program, and when the processor 200 runs the computer program stored in the memory 100, the processor 200 executes the above-mentioned various possible methods.

[0187] The memory 100 is connected to the processor 200. The memory 100 may be a flash memory, a read-only memory or other memory, and the processor 200 may be a central processing unit or a single-chip microcomputer.

[0188] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the above-mentioned various possible methods.

[0189] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), Digital Video Disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0190] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A method for sorting integrated order messages based on Kafka, characterized in that: The method comprises: Receive integrated orders and collect topics adapted for business; Build the Kafka priority queue model and business lock model; Combine the Kafka priority queue with the business lock strategy to establish a message processing and transmission strategy that takes priority and sorting into account, solve multi-objective optimization problems, and generate a message control mechanism according to the solution results to distribute integrated order messages.

2. The method according to claim 1, characterized in that The integrated order includes a letter control integrated order and a business integrated order; The topics adapted by the collection service include cities, order types, telecommunication networks, and special attribute flags.

3. The method according to claim 1, characterized in that The working process of the Kafka priority queue model includes: Create multiple topics, one topic represents one priority, and each topic is divided into multiple partitions; Construct a kjPriorityConsumer that aggregates multiple priority consumers; According to the business situation, set the number of messages pulled by kjPriorityConsumer each time, so that the consumer consumes the data in the corresponding partition in order from high to low priority.

4. The method according to claim 3, characterized in that The working process of the business lock model includes: S1: Set key values ​​for redis: number key value SERIAL_NUM, number lock key value MSISDN_LOCK; S2: parse the received message and determine whether there is a lock in redis based on the number key value SERIAL_NUM; S3: If the number is not locked, lock the number and then send a message; and, S4: After the entire work order is executed, redis is unlocked; S5: If there is a lock, send the message to the first delay queue; and, S6: Parse the message received by the first delay queue and check the redis result. If it is not unlocked, the message will continue to be thrown into the first delay queue and wait; if it is unlocked, continue S3 operation.

5. The method according to claim 4, characterized in that The key value set by redis also includes the number office key value MSISDN_SWITCH_LOCK; The business lock model working process also includes: In S6, check the redis result. If it is unlocked, query the order to be executed according to the number and send it to the acquisition queue; Parse the message received by the second delay queue, detect the redis lock, and send it to the waiting queue if it is unlocked, otherwise throw it back to the queue and wait; Parse the data of the message received by the queue to be sent, check whether there is a direction lock in the redis node according to the number key value and the number direction key value, and send it to the second delay queue if there is one; If not, lock the number key value and the number office key value, and execute the work order to interact with the exchange side; Parse the interactive return results, update the work order status, and unlock the number key value and number office key value in redis.

6. The method according to claim 4, characterized in that The method of establishing a message processing and transmission strategy taking into account priorities and sorting, and solving a multi-objective optimization problem, includes: For the received integrated order message, check whether there is a lock in redis based on the number key value SERIAL_NUM; If there is a lock, the message is sent to the first delay queue and the redis result is checked; If the redis result is not unlocked, it will continue to be thrown into the first delay queue and wait; if it is unlocked or not locked, the matched business adaptation topic will be obtained, the number will be locked and sent to the corresponding priority topic; the consumer consumes the data in the corresponding partition in order from high to low priority according to the number of messages pulled each time by the kjPriorityConsumer set according to the business situation, and unlocks the number after completing the business logic processing.

7. An integrated order message sorting system based on Kafka, characterized in that: The system comprises: A receiving module, which is configured to receive integrated orders and collect topics adapted for business; Construction module, which is set to build Kafka priority queue model and business lock model; The sorting module is configured to combine the Kafka priority queue with the business lock strategy, establish a message processing and transmission strategy that takes priority and sorting into account, solve multi-objective optimization problems, and generate a message control mechanism according to the solution results to distribute integrated order messages.

8. The system according to claim 7, characterized in that The working process of the Kafka priority queue model includes: Create multiple topics, one topic represents one priority, and each topic is divided into multiple partitions; Construct a kjPriorityConsumer that aggregates multiple priority consumers; Set the number of messages pulled by kjPriorityConsumer each time according to the business situation, so that the consumer consumes the data in the corresponding partition in order from high to low priority; The working process of the business lock model includes: S1: Set key values ​​for redis: number key value SERIAL_NUM, number lock key value MSISDN_LOCK; S2: parse the received message and determine whether there is a lock in redis based on the number key value SERIAL_NUM; S3: If the number is not locked, lock the number and then send a message; and, S4: After the entire work order is executed, redis is unlocked; S5: If there is a lock, send the message to the first delay queue; and, S6: Parse the message received by the first delay queue and check the redis result. If it is not unlocked, the message will continue to be thrown into the first delay queue and wait; if it is unlocked, continue S3 operation.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the Kafka-based integrated order message sorting method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the integrated order message sorting method based on Kafka as described in any one of claims 1 to 6 is implemented.

Images