A method and device for processing online small batch messages
By using message queues and consumer scheduling mechanisms, the problem of system resources being squeezed in online small-batch message processing is solved, thereby releasing user waiting time, improving system usability, and reducing human resource costs.
Patent Information
- Application Number
- CN202110603498.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-31
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-05-31
AI Technical Summary
Existing technologies pose a risk of system resource exhaustion in online small-batch message processing, requiring users to monitor system performance for extended periods, resulting in high labor costs and poor system stability and usability.
A message queue mechanism is adopted. By generating business processing instructions one by one and setting priorities, the consumer scheduling mechanism is used to extract messages from the message queue, monitor consumption actions and production commit operations, and push monitoring information to users.
It effectively avoids the risk of system resources being occupied, reduces user waiting time, improves system usability, reduces human resource costs, and enhances system stability and maintenance capabilities.
Smart Images

Figure CN113326150B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer data processing technology, specifically to big data technology, and more particularly to an online small batch message processing method and device. Background Art
[0002] This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.
[0003] With the development of online and intelligent operations, in order to reduce the scenario where branches conduct offline personal loan issuance in a centralized manner at the beginning of each month, for approved loan notes, the host will automatically issue loans in small online batches with the help of message services during a specific time period.
[0004] The existing system needs to manually trigger multiple online transactions. The main implementation method is to query all records through the list, select N records at a time (the size of N is determined by the online time consumption of a single record and the system timeout setting) and submit them. The system cyclically calls the online interface for processing the batch of records, and displays the processing results one by one after completion. After the user completes the verification, he selects N records again to submit. This is repeated until all records to be processed are submitted and the results are received.
[0005] The existing technology mainly has the following disadvantages:
[0006] 1. From the first processing to the last result return, the user needs to keep an eye on the execution of the system, and each result needs to be analyzed and processed in a timely manner after it is returned, which occupies a high manpower cost.
[0007] 2. The number of users cannot be controlled. If a large number of users use this type of transaction in a short period of time, the system's concurrency pressure will be relatively high (such as connection pool, CPU, memory), and even affect the risk of stable operation of the system.
[0008] 3. If the current limiting mechanism is added to a certain extent, when a large number of users are operating together, the operations of some users may be frequently rejected by the system, which greatly reduces the system availability. In order to cope with this situation, the hardware equipment configuration is increased, and the resource usage is low during the idle time of the system, which is a huge waste.
[0009] It is particularly important to propose a reliable online small batch processing mechanism to solve the above problems.
[0010] Therefore, how to provide a new solution that can solve the above technical problems is a technical problem that needs to be solved urgently in this field. Summary of the invention
[0011] The embodiment of the present invention provides an online small batch message processing method, which can effectively avoid the risk of system resources being occupied, release user waiting time, and improve system usability. The method includes:
[0012] Establish a message queue and set up a message queue writing mechanism;
[0013] Generate business processing instructions for each transaction and set priorities based on the screening conditions for the transaction to be initiated, and generate messages by performing production submission operations;
[0014] Write the message into the message queue according to the message queue writing mechanism;
[0015] According to the consumer scheduling mechanism, consumers are used to extract messages from the message queue and perform consumption actions on the messages;
[0016] Monitor consumers' consumption actions, message queues, and production submission operations, and push monitoring information to users according to preset notification methods.
[0017] The embodiment of the present invention further provides an online small batch message processing device, comprising:
[0018] The message queue establishment module is used to establish the message queue and set the message queue writing mechanism;
[0019] The message generation module is used to generate business processing instructions and set priorities for each transaction based on the screening conditions of the transaction to be initiated, and to generate messages for production submission operations;
[0020] A message writing module is used to write messages into the message queue according to the message queue writing mechanism;
[0021] The consumer scheduling module is used to extract messages from the message queue using consumers according to the consumer scheduling mechanism and perform consumption actions on the messages;
[0022] The monitoring module is used to monitor consumers' consumption actions, message queues, and production submission operations, and push monitoring information to users according to the preset notification method.
[0023] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned online small batch message processing method when executing the computer program.
[0024] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the above-mentioned online small batch message processing method.
[0025] An online small batch message processing method and device provided by an embodiment of the present invention includes: first, establishing a message queue and setting a message queue writing mechanism; then, according to the screening conditions of the transaction to be initiated, generating business processing instructions for each transaction and setting the priority, performing production submission operations to generate messages; then, writing the message into the message queue according to the message queue writing mechanism; next, according to the consumer scheduling mechanism, using the consumer to extract the message from the message queue and perform consumption actions on the message; finally, monitoring the consumer's consumption actions, message queues and production submission operations, and pushing monitoring information to the user according to the preset notification method. In view of the shortcomings of the existing system technology implementation scheme, the present invention provides an online small batch message processing mechanism based on a message queue, which can effectively avoid the risk of system resources being squeezed, release user waiting time, and improve system usability. The present invention combines the characteristics of user usage scenarios, and through the design of message queue producer and consumer mechanisms, effectively solves the risk of system resource squeezing, while releasing a large amount of human resource costs of the branch. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0027] Figure 1 The present invention is a schematic diagram of an online small batch message processing method according to an embodiment of the present invention.
[0028] Figure 2 The present invention is a schematic diagram of a message generation process of an online small batch message processing method according to an embodiment of the present invention.
[0029] Figure 3 The present invention is a schematic diagram of a process of pushing monitoring information of an online small batch message processing method according to an embodiment of the present invention.
[0030] Figure 4 A schematic diagram of a computer device for running an online small batch message processing method implemented by the present invention.
[0031] Figure 5 The present invention is a schematic diagram of an online small batch message processing device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0032] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.
[0033] The present invention relates to big data technology. Figure 1 FIG. 1 is a schematic diagram of an online small batch message processing method according to an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides an online small batch message processing method, which can effectively avoid the risk of system resources being occupied, release user waiting time, and improve system usability. The method includes:
[0034] Step 101: Establish a message queue and set up a message queue writing mechanism;
[0035] Step 102: Generate business processing instructions for each transaction and set priorities based on the screening conditions of the transaction to be initiated, and perform production submission operations to generate messages;
[0036] Step 103: Write the message into the message queue according to the message queue writing mechanism;
[0037] Step 104: According to the consumer scheduling mechanism, the consumer is used to extract the message from the message queue and perform consumption actions on the message;
[0038] Step 105: Monitor the consumer's consumption actions, message queues, and production submission operations, and push monitoring information to the user according to a preset notification method.
[0039] An online small batch message processing method provided by an embodiment of the present invention includes: first, establishing a message queue and setting a message queue writing mechanism; then, according to the screening conditions of the transaction to be initiated, generating business processing instructions for each transaction and setting the priority, performing production submission operations to generate messages; then, writing the message into the message queue according to the message queue writing mechanism; next, according to the consumer scheduling mechanism, using the consumer to extract the message from the message queue and perform consumption actions on the message; finally, monitoring the consumer's consumption actions, message queues and production submission operations, and pushing monitoring information to the user according to the preset notification method. In view of the shortcomings of the existing system technology implementation scheme, the present invention provides an online small batch message processing mechanism based on a message queue, which can effectively avoid the risk of system resources being squeezed, release user waiting time, and improve system usability. The present invention combines the characteristics of user usage scenarios, and through the design of message queue producer and consumer mechanisms, effectively solves the risk of system resource squeezing, while releasing a large amount of human resource costs of the branch.
[0040] In the embodiment of the present invention, the message queue MQ (Message Queue) is a storage container for caching upstream and downstream interactive information (text, pictures, audio, emails, etc.), a first-in-first-out data structure, after the upstream application pushes the message to the downstream application cache, it waits for external push or active pull by the downstream application. MQ middleware is widely used in distributed systems, mainly to solve problems such as application coupling, asynchronous messaging, and traffic peak shaving.
[0041] When implementing an online small batch message processing method provided by an embodiment of the present invention, in one embodiment, the following may be included:
[0042] Establish a message queue and set up a message queue writing mechanism;
[0043] Generate business processing instructions for each transaction and set priorities based on the screening conditions for the transaction to be initiated, and generate messages by performing production submission operations;
[0044] Write the message into the message queue according to the message queue writing mechanism;
[0045] According to the consumer scheduling mechanism, consumers are used to extract messages from the message queue and perform consumption actions on the messages;
[0046] Monitor consumers' consumption actions, message queues, and production submission operations, and push monitoring information to users according to preset notification methods.
[0047] Aiming at the shortcomings of existing system technology implementation solutions, the present invention provides an online small batch mechanism based on message queue, which can effectively avoid the risk of system resources being occupied, release user waiting time, and improve system usability.
[0048] When implementing an online small batch message processing method provided by an embodiment of the present invention, in one embodiment, a message queue is established and a message queue writing mechanism is set, including:
[0049] Establish multiple message queues and set a queue number for each message queue to store messages containing business processing instructions;
[0050] According to the number of message queues, set the global variable; if the global variable satisfies the initial value of 0, divide the global variable by the number of message queues and assign it to the next global variable;
[0051] Before writing a message to the message queue each time, first obtain the global variable;
[0052] According to the value of the global variable, determine the queue number of the message queue to which the current message is written, and write the current message into the message queue;
[0053] After the current message is written into the message queue, the global variable is reassigned and the writing of the next message is continued; wherein, the value of the global variable is increased by one and then divided by the number of message queues and assigned to the next global variable.
[0054] The embodiment of the present invention provides a writing mechanism based on a message queue, which mainly includes:
[0055] Establish N (N>1, and N is an integer) message queues, set a queue number j for each message queue, and the value of j is 0, 1, 2, ..., N-1, which is used to store messages containing business processing instructions;
[0056] According to the number of message queues, set the global variable q; the global variable q satisfies the initial value of 0, and the global variable q is divided by the number of message queues N and assigned to the next global variable q; that is, the value of q is equal to the remainder of q divided by N, that is, q = q% N, and the subsequent remainder operators are all represented by %;
[0057] Before writing a message to the message queue each time, first obtain the global variable q;
[0058] According to the value of the global variable q, the queue number of the message queue to which the current message is written is determined, and the current message is written into the message queue;
[0059] After the current message is written into the message queue, the global variable is reassigned to (q+1)%N, and the writing of the next message is continued; wherein, the value of the global variable is increased by one and then divided by the number of message queues and assigned to the next global variable, that is, q=(q+1)%N.
[0060] In an example of an embodiment of the present invention, the message queue writing mechanism design process includes:
[0061] Set N (N>1, and N is an integer) message queues to store business instructions. Number each queue j, and the value of j is 0, 1, 2...N-1;
[0062] Set a global variable q, and satisfy the initial value of q is 0, and the value of q is equal to the remainder of q divided by N (that is, q = q% N, and the remainder operator in this article is represented by %);
[0063] Before writing a message to the message queue, first obtain the q value to determine the queue number to which the current message is written, and then reassign q to (q+1)%N.
[0064] Finally, write the current message to queue q.
[0065] The present invention can effectively control the maximum number of concurrent messages in the system through the number of message queues, realize peak shaving and valley filling, and avoid system risks caused by system resource crowding.
[0066] The expressions of the message queue writing mechanism design process mentioned above are for illustration purposes only. Those skilled in the art will appreciate that, during implementation, the above formulas may be deformed in a certain form and other parameters or data may be added, or other specific formulas may be provided as needed. These variations shall all fall within the scope of protection of the present invention.
[0067] Figure 2 FIG. 1 is a schematic diagram of a message generation process of an online small batch message processing method according to an embodiment of the present invention. Figure 2 As shown, when implementing an online small batch message processing method provided by an embodiment of the present invention, in one embodiment, according to the screening conditions of the transaction to be initiated, a business processing instruction is generated and a priority is set, and a production submission operation is performed to generate a message, including:
[0068] Step 201: Setting screening conditions on the user submission page;
[0069] Step 202: According to the screening conditions of the transaction to be initiated submitted by the user on the submission page, all business records that meet the conditions are extracted on the application server side, and business processing instructions are generated for each transaction according to the set format;
[0070] Step 203: setting a priority for each generated business processing instruction according to the business transaction type dimension and the urgency of the business transaction;
[0071] Step 204: Submit the business processing instructions of the set priority for each transaction to generate a message, and return a successful submission result after the submission is completed.
[0072] In the embodiment, the main process of generating a message by a production submission operation includes:
[0073] First, set the screening conditions on the user submission page. Considering that the number of users cannot be controlled in the existing technology, if a large number of users use such transactions in a short period of time, the system's concurrent pressure will be high, which will affect the risk of stable operation of the system. In setting the user submission interface, only the screening conditions for the intended transaction need to be entered.
[0074] Then, based on the screening conditions of the transaction to be initiated submitted by the user on the submission page, all business records that meet the conditions are extracted on the application server side, and business processing instructions are generated for each transaction in the set format;
[0075] Then, according to the business transaction type dimension and the urgency of the business transaction, set the priority of each generated business processing instruction. Considering the different urgency of different business transactions, set the message priority based on the business transaction type dimension. Set the priority for each type of business transaction. The priorities of different businesses can be the same.
[0076] Finally, the business processing instructions with set priorities are submitted to generate a message, and a successful submission result is returned after the submission is completed. At this point, the user's submission operations in the system are all completed at once.
[0077] The present invention achieves loose coupling between the system and user operations through the design of the message queue production process, effectively reduces the number of personnel involved in the branch operation by adjusting the user operation mode, and effectively releases the user's idle waiting time by adjusting the system notification method. The system usability is greatly improved. Through the priority design mechanism, it can accommodate the hierarchical processing of multiple types of business at the same time, greatly increasing the reusability of functions and effectively reducing the development cost of similar functions.
[0078] When implementing an online small batch message processing method provided by an embodiment of the present invention, in one embodiment, according to a consumer scheduling mechanism, a consumer is used to extract messages from a message queue and perform consumption actions on the messages, including:
[0079] Set up multiple consumers and set a consumer number for each consumer;
[0080] Set constants; the setting of constants must satisfy the requirement that the constants and the number of message queues do not have any common divisors other than 1;
[0081] Set a time period; the time period is simulated and acquired based on the differences in business processing of different systems;
[0082] According to the consumer number, constant, time period, queue number of the message queue, priority of the message and the time when the message arrives at the message queue, combined with the consumer scheduling mechanism, the consumer is used to extract the message from the message queue and perform consumption actions on the message.
[0083] In the embodiment, the process of scheduling the consumer to execute the consumption action mainly includes:
[0084] Set M (M>=1, M is an integer) consumers, set the consumer number i for each consumer, and the value of i is 0, 1, 2, ..., M-1;
[0085] Set a constant p; the setting of the constant must satisfy the requirement that the constant p and the number of message queues N do not have any common divisor other than 1;
[0086] Set the timing time period t; the timing time period t is obtained by simulation based on the differences in business processing of different systems; the t value can generally be obtained by simulation based on the differences in business processing of different systems, and is mainly used to solve the queue synchronization problem of different consumers due to long-term operation of the system, and a re-hashing mechanism after automatic restart needs to be set;
[0087] According to the consumer number, constant, time period, queue number of the message queue, priority of the message and the time when the message arrives at the message queue, combined with the consumer scheduling mechanism, the consumer is used to extract the message from the message queue and perform consumption actions on the message.
[0088] In an example of an embodiment of the present invention, a consumer scheduling process based on a message queue includes:
[0089] Set M (M>=1, M is an integer) consumers, number each consumer as i, and the value of i is 0, 1, 2...M-1;
[0090] Set a constant p, which must satisfy the requirement that p and N do not have any common divisor other than 1. Generally, p can be 1.
[0091] Set the timing time period t. The t value can generally be obtained through simulation according to the differences in business processing of different systems. It is mainly used to solve the queue synchronization problem of different consumers due to long-term operation of the system. It is necessary to set a re-hashing mechanism after automatic restart;
[0092] According to the consumer number, constant, time period, queue number of the message queue, priority of the message and the time when the message arrives at the message queue, combined with the consumer scheduling mechanism, the consumer is used to extract the message from the message queue, perform consumption actions on the message, and perform consumer scheduling.
[0093] When implementing an online small batch message processing method provided by an embodiment of the present invention, in one embodiment, according to the consumer number, constant, timing time period, queue number of the message queue, priority of the message and time point when the message arrives at the message queue, in combination with the consumer scheduling mechanism, the consumer is used to extract the message from the message queue, and the consumption action is performed on the message, including:
[0094] Select one consumer from multiple consumers to start, and obtain one message queue from multiple message queues as the initial consumer queue of the consumer through the random module;
[0095] Get the priority of all messages in the initial consumption queue, take out the message with the highest priority and the earliest arrival time in the message queue, and use the consumer to perform consumption actions on the taken out message;
[0096] After the consumer completes the consumption of the retrieved message, the next consumption queue is changed, and the process of obtaining the priority of the next consumption queue and executing the consumption action is repeated until the startup time of the consumer reaches the timing time period;
[0097] When the startup time of the consumer reaches the timed period, the consumption action is completed, the consumer is restarted, and the process of obtaining the message queue through the random module is re-executed.
[0098] In the embodiment, a plurality of scheduling parameters are combined to schedule consumers to perform consumption actions, mainly including:
[0099] Select a consumer i from M consumers to start, and obtain a message queue j from N message queues through the random module as the initial consumption queue of consumer i;
[0100] Get the priority of all messages in the initial consumption queue j, take out the message m with the highest priority and the earliest arrival time in the message queue, and perform consumption actions on the taken out message m through the consumer i;
[0101] After the consumer i completes the consumption of the retrieved message m, the next consumption queue is changed to (j+p)%M, and the process of obtaining the priority of the next consumption queue (j+p)%M to executing the consumption action is repeated until the startup time length of the consumer i reaches the timing time period t;
[0102] When the startup time length of the consumer i reaches the timing time period t, the consumption action is completed, the consumer i is restarted, and the process of obtaining a new message queue through the random module is re-executed.
[0103] In an example of an embodiment of the present invention, the consumer scheduling process based on the message queue mainly includes:
[0104] 1) When consumer i starts, a queue j among N message queues is obtained through the random module as the initial consumption queue of consumer i;
[0105] 2) Get the priorities of all messages in queue j and take out the message m with the highest priority and the earliest arrival;
[0106] 3) After consumer i completes the consumption of message m, it will change the next consumption queue to (j+p)%M and repeat the previous step 2) until the start time of consumer i reaches t;
[0107] 4) When the start time of consumer i reaches t, the consumption action is completed, consumer i is restarted, and the above step 1) of consumer scheduling is executed.
[0108] The present invention realizes an automatic drift strategy of consumers between queues through a unique consumer scheduling design, and effectively prevents the possibility of stopping the consumption behavior of the entire queue caused by consumer deadlock.
[0109] The above-mentioned expressions involved in the process of scheduling consumers to execute consumption actions are for illustration only. Those skilled in the art will appreciate that, during implementation, the above formulas may be deformed in a certain form and other parameters or data may be added, or other specific formulas may be provided as needed. All of these variations should fall within the scope of protection of the present invention.
[0110] Figure 3FIG. 1 is a schematic diagram of a process of pushing monitoring information of an online small batch message processing method according to an embodiment of the present invention. Figure 3 As shown, when implementing an online small batch message processing method provided by an embodiment of the present invention, in one embodiment, monitoring the consumer's consumption actions, message queues, and production submission operations, and pushing monitoring information to the user according to a preset notification method, including:
[0111] Step 301: when performing a production submission operation, obtaining a notification method of information submitted by a user;
[0112] Step 302: monitor the operation indicators including the execution of consumer consumption actions, the consumption speed of the message queue, the production submission operation and the system memory usage, and generate monitoring data;
[0113] Step 303: Perform real-time analysis on the monitoring data, and generate monitoring information according to the warning threshold of each operating indicator when the operating indicator exceeds the warning threshold, and push it to the user in the information notification method submitted by the user.
[0114] When specifically implementing an online small batch message processing method provided by an embodiment of the present invention, in one embodiment, the method further includes:
[0115] Generate a transaction execution report based on the submitting user and business type dimensions to count production submissions, pending messages, and processed messages within a set time period;
[0116] The transaction execution report is pushed to the user according to the information notification method submitted by the user.
[0117] In the embodiment, the consumer's consumption actions, message queues and production submission operations are monitored, mainly including: statistics on submissions, pending and processed within a certain period of time according to the submitting user and business type dimensions, and push information to users according to the set notification method. At the same time, through the monitoring module, indicators such as the consumption speed of the message queue, consumer execution status, and system memory usage are monitored; through the early warning module, after real-time analysis of the monitoring data, according to the setting of the early warning threshold of each operating indicator, the early warning result is sent to notify the operation and maintenance personnel.
[0118] The present invention effectively shortens the discovery time of various business-level failures and system-level failures during operation by combining the early warning monitoring module and the notification module, thereby improving the system operation and maintenance capabilities.
[0119] The present invention combines the characteristics of user usage scenarios and provides an online small batch mechanism based on message queues. By designing the message queue producer and consumer mechanism, it effectively solves the risk of system resource crowding out and releases a large amount of human resource costs of the branch.
[0120] The following briefly describes an online small batch message processing method provided by an embodiment of the present invention in conjunction with a specific scenario:
[0121] Aiming at the shortcomings of existing system technology implementation solutions, the present invention provides an online small batch mechanism based on message queue, which can effectively avoid the risk of system resources being occupied, release user waiting time, and improve system usability.
[0122] The main mechanism of the present invention mainly includes three components:
[0123] 1. A production mechanism based on message queue:
[0124] 1) Message queue design:
[0125] a. Set N (N>1, and N is an integer) message queues to store business instructions, and number each queue j, where the value of j is 0, 1, 2...N-1;
[0126] b. Set a global variable q, and satisfy the initial value of q is 0, and the value of q is equal to the remainder of q divided by N (i.e. q = q% N, the remainder operator in this article is represented by %);
[0127] c. Before writing a message to the message queue each time, first obtain the q value to determine the queue number to which the current message is written, and then reassign q to (q+1)%N;
[0128] d. Finally, write the current message to queue q.
[0129] 2) Set message priority based on the business transaction type dimension. Considering the different urgency levels of different business transactions, set a priority for each type of business transaction. The priorities of different businesses can be the same.
[0130] 3) Adjust the user submission method, set the user submission interface to only enter the screening conditions for the transaction to be initiated, and after the user submits the screening conditions, the application server extracts all business records that meet the conditions, generates a transaction-by-transaction processing instruction in the specified format and inserts it into the message queue, and returns a successful submission result after completion. At this point, the user's submission operation in the system is completed at one time.
[0131] 2. A consumer scheduling mechanism based on message queue:
[0132] 1) Set M (M>=1, M is an integer) consumers, number each consumer as i, and the value of i is 0, 1, 2...M-1;
[0133] 2) Set a constant p, which must satisfy the requirement that p and N do not have any common divisor other than 1. Generally, p can be 1.
[0134] 3) Set a timing time period t. The t value can generally be obtained through simulation according to the differences in business processing of different systems. It is mainly used to solve the queue synchronization problem of different consumers due to long-term operation of the system. It is necessary to set a re-hashing mechanism after automatic restart.
[0135] 4) Consumer Scheduling:
[0136] a. When consumer i starts, a queue j among N message queues is obtained through the random module as the initial consumption queue of consumer i;
[0137] b. Get the priorities of all messages in queue j and take out the message m with the highest priority and arriving earliest;
[0138] c. After consumer i completes the consumption of message m, it will change the next consumption queue to (j+p)%M and repeat the previous step b until the start time of consumer i reaches t;
[0139] d. When the start time of consumer i reaches t, the consumption action is completed, consumer i is restarted, and the first step a of consumer scheduling is executed.
[0140] 3. A notification mechanism for the execution of message queues:
[0141] 1) Notification method setting: When a user submits a request, a functional unit is provided for the user to enter the notification method, such as mobile phone number, email address, etc.
[0142] 2) Early warning monitoring and notification mechanism: Statistics are collected on submissions, pending processing, and processed within a certain period of time according to the dimensions of submitting users and business types, and information is pushed to users according to the set notification method. At the same time, the monitoring module monitors indicators such as the consumption speed of the message queue, consumer execution status, and system memory usage; the early warning module analyzes the monitoring data in real time, and sends the early warning results to notify the operation and maintenance personnel according to the setting of the early warning thresholds of each operating indicator.
[0143] The present invention combines the characteristics of user usage scenarios and provides an online small batch mechanism based on message queues. By designing the message queue producer and consumer mechanism, it effectively solves the risk of system resource crowding and releases a large amount of human resource costs of branches. Its advantages are as follows:
[0144] 1. The present invention can effectively control the maximum number of concurrent messages through the number of message queues, realize peak shaving and valley filling, and avoid system risks caused by system resource crowding.
[0145] 2. The present invention achieves loose coupling between the system and user operations through the design of the message queue production process, effectively reduces the number of personnel involved in the branch operation by adjusting the user operation mode, and effectively releases the user's idle waiting time by adjusting the system notification method. This greatly improves the usability of the system.
[0146] 3. The present invention realizes the automatic drift strategy of consumers between queues through the unique consumer scheduling design, effectively preventing the possibility of stopping the consumption behavior of the entire queue caused by consumer deadlock.
[0147] 4. The present invention effectively shortens the discovery time of various business-level failures and system-level failures during operation by combining the early warning monitoring module and the notification module, thereby improving the system operation and maintenance capabilities.
[0148] 5. The present invention can accommodate the hierarchical processing of multiple types of services through a priority mechanism, greatly increasing the reusability of functions and effectively reducing the development cost of similar functions.
[0149] Figure 4 A schematic diagram of a computer device for running an online small batch message processing method implemented by the present invention, such as Figure 4 As shown, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned online small batch message processing method when executing the computer program.
[0150] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing and implementing the above-mentioned online small batch message processing method.
[0151] The present invention also provides an online small batch message processing device, as described in the following embodiments. Since the principle of solving the problem by the device is similar to that of an online small batch message processing method, the implementation of the device can refer to the implementation of an online small batch message processing method, and the repeated parts will not be repeated.
[0152] Figure 5 FIG. 1 is a schematic diagram of an online small batch message processing device according to an embodiment of the present invention. Figure 5 As shown, the embodiment of the present invention further provides an online small batch message processing device, which may include:
[0153] The message queue establishment module 501 is used to establish a message queue and set a message queue writing mechanism;
[0154] The message generation module 502 is used to generate business processing instructions and set priorities for each transaction according to the screening conditions of the transaction to be initiated, and to generate messages for production submission operations;
[0155] A message writing module 503, used to write the message into the message queue according to the message queue writing mechanism;
[0156] The consumer scheduling module 504 is used to extract messages from the message queue using consumers according to the consumer scheduling mechanism and perform consumption actions on the messages;
[0157] The monitoring module 505 is used to monitor the consumer's consumption actions, message queues and production submission operations, and push monitoring information to the user according to a preset notification method.
[0158] When implementing an online small batch message processing device provided by an embodiment of the present invention, in one embodiment, the message generation module is specifically used to:
[0159] Set filter conditions on the user submission page;
[0160] According to the screening conditions of the transaction to be initiated submitted by the user on the submission page, all business records that meet the conditions are extracted on the application server side, and business processing instructions are generated for each transaction in the set format;
[0161] Prioritize the generated business processing instructions based on the business transaction type and the urgency of the business transaction.
[0162] The business processing instructions of the set priority are produced and submitted to generate a message, and the submission success result is returned after the submission is completed.
[0163] When implementing an online small batch message processing device provided by an embodiment of the present invention, in one embodiment, the aforementioned monitoring module is specifically used to:
[0164] When performing production submission operations, obtain the information notification method submitted by the user;
[0165] Monitor the operation indicators including the execution of consumer consumption actions, the consumption speed of message queues, production submission operations, and system memory usage, and generate monitoring data;
[0166] Perform real-time analysis on monitoring data and generate monitoring information based on the warning threshold of each operating indicator when the operating indicator exceeds the warning threshold, and push it to the user in the form of information notification submitted by the user.
[0167] When implementing an online small batch message processing device provided by an embodiment of the present invention, in one embodiment, the aforementioned monitoring module is also used to:
[0168] Generate a transaction execution report based on the submitting user and business type dimensions to count production submissions, pending messages, and processed messages within a set time period;
[0169] The transaction execution report is pushed to the user according to the information notification method submitted by the user.
[0170] In summary, an online small batch message processing method and device provided by an embodiment of the present invention includes: first, establishing a message queue and setting a message queue writing mechanism; then, according to the screening conditions of the transaction to be initiated, generating business processing instructions for each transaction and setting the priority, performing production submission operations to generate messages; then, writing the message into the message queue according to the message queue writing mechanism; next, according to the consumer scheduling mechanism, using the consumer to extract the message from the message queue and perform consumption actions on the message; finally, monitoring the consumer's consumption actions, message queues and production submission operations, and pushing monitoring information to the user according to the preset notification method. In view of the shortcomings of the existing system technology implementation scheme, the present invention provides an online small batch message processing mechanism based on a message queue, which can effectively avoid the risk of system resources being squeezed, release user waiting time, and improve system usability. The present invention combines the characteristics of user usage scenarios, and through the design of message queue producer and consumer mechanisms, effectively solves the risk of system resource squeezing, while releasing a large amount of human resource costs of the branch.
[0171] The present invention combines the characteristics of user usage scenarios and provides an online small batch mechanism based on message queues. By designing the message queue producer and consumer mechanism, it effectively solves the risk of system resource crowding and releases a large amount of human resource costs of branches. Its advantages are as follows:
[0172] 1. The present invention can effectively control the maximum number of concurrent messages through the number of message queues, realize peak shaving and valley filling, and avoid system risks caused by system resource crowding.
[0173] 2. The present invention achieves loose coupling between the system and user operations through the design of the message queue production process, effectively reduces the number of personnel involved in the branch operation by adjusting the user operation mode, and effectively releases the user's idle waiting time by adjusting the system notification method. This greatly improves the usability of the system.
[0174] 3. The present invention realizes the automatic drift strategy of consumers between queues through the unique consumer scheduling design, effectively preventing the possibility of stopping the consumption behavior of the entire queue caused by consumer deadlock.
[0175] 4. The present invention effectively shortens the discovery time of various business-level failures and system-level failures during operation by combining the early warning monitoring module and the notification module, thereby improving the system operation and maintenance capabilities.
[0176] 5. The present invention can accommodate the hierarchical processing of multiple types of services through a priority mechanism, greatly increasing the reusability of functions and effectively reducing the development cost of similar functions.
[0177] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0178] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0181] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. An online small batch message processing method, characterized in that: include: Establish a message queue and set up a message queue writing mechanism; Generate business processing instructions for each transaction and set priorities based on the screening conditions for the transaction to be initiated, and generate messages by performing production submission operations; Write the message into the message queue according to the message queue writing mechanism; According to the consumer scheduling mechanism, consumers are used to extract messages from the message queue and perform consumption actions on the messages; Monitor consumers' consumption actions, message queues, and production submission operations, and push monitoring information to users according to the preset notification method; Among them, establishing a message queue and setting a message queue writing mechanism include: Establish multiple message queues and set a queue number for each message queue to store messages containing business processing instructions; According to the number of message queues, set the global variable; if the global variable satisfies the initial value of 0, divide the global variable by the number of message queues and assign it to the next global variable; Before writing a message to the message queue each time, first obtain the global variable; According to the value of the global variable, determine the queue number of the message queue to which the current message is written, and write the current message into the message queue; After the current message is written into the message queue, the global variable is reassigned and the writing of the next message is continued; wherein, the value of the global variable is increased by one and then divided by the number of message queues and assigned to the next global variable; Among them, according to the consumer scheduling mechanism, the consumer is used to extract messages from the message queue and perform consumption actions on the messages, including: Set up multiple consumers and set a consumer number for each consumer; Set constants; the setting of constants must satisfy the requirement that the constants and the number of message queues do not have any common divisors other than 1; Set a time period; the time period is simulated and acquired based on the differences in business processing of different systems; According to the consumer number, constant, time period, queue number of the message queue, priority of the message and the time when the message arrives at the message queue, combined with the consumer scheduling mechanism, the consumer is used to extract the message from the message queue and perform consumption actions on the message.
2. The method according to claim 1, characterized in that Generate business processing instructions for each transaction and set priorities based on the screening conditions of the transaction to be initiated, and generate messages for production submission operations, including: Set filter conditions on the user submission page; According to the screening conditions of the transaction to be initiated submitted by the user on the submission page, all business records that meet the conditions are extracted on the application server side, and business processing instructions for each transaction are generated according to the set format; Prioritize the generated business processing instructions based on the business transaction type and the urgency of the business transaction. The business processing instructions of the set priority are produced and submitted to generate a message, and the submission success result is returned after the submission is completed.
3. The method according to claim 1, characterized in that Monitor consumer consumption actions, message queues, and production submission operations, and push monitoring information to users according to preset notification methods, including: When performing production submission operations, obtain the information notification method submitted by the user; Monitor the operation indicators including the execution of consumer consumption actions, the consumption speed of message queues, production submission operations, and system memory usage, and generate monitoring data; Perform real-time analysis on monitoring data and generate monitoring information based on the warning threshold of each operating indicator when the operating indicator exceeds the warning threshold, and push it to the user in the form of information notification submitted by the user.
4. The method according to claim 3, characterized in that Also includes: Generate a transaction execution report based on the submitting user and business type dimensions to count production submissions, pending messages, and processed messages within a set time period; The transaction execution report is pushed to the user according to the information notification method submitted by the user.
5. An online small batch message processing device, characterized in that: include: The message queue establishment module is used to establish the message queue and set the message queue writing mechanism; The message generation module is used to generate business processing instructions and set priorities for each transaction based on the screening conditions of the transaction to be initiated, and to generate messages for production submission operations; A message writing module is used to write messages into the message queue according to the message queue writing mechanism; The consumer scheduling module is used to extract messages from the message queue using consumers according to the consumer scheduling mechanism and perform consumption actions on the messages; The monitoring module is used to monitor the consumer's consumption actions, message queues, and production submission operations, and push monitoring information to users according to the preset notification method; Among them, the message queue establishment module is specifically used for: Establish multiple message queues and set a queue number for each message queue to store messages containing business processing instructions; According to the number of message queues, set the global variable; if the global variable satisfies the initial value of 0, divide the global variable by the number of message queues and assign it to the next global variable; Before writing a message to the message queue each time, first obtain the global variable; According to the value of the global variable, determine the queue number of the message queue to which the current message is written, and write the current message into the message queue; After the current message is written into the message queue, the global variable is reassigned and the writing of the next message is continued; wherein, the value of the global variable is increased by one and then divided by the number of message queues and assigned to the next global variable; Among them, the consumer scheduling module is specifically used for: Set up multiple consumers and set a consumer number for each consumer; Set constants; the setting of constants must satisfy the requirement that the constants and the number of message queues do not have any common divisors other than 1; Set a time period; the time period is simulated and acquired based on the differences in business processing of different systems; According to the consumer number, constant, time period, queue number of the message queue, priority of the message and the time when the message arrives at the message queue, combined with the consumer scheduling mechanism, the consumer is used to extract the message from the message queue and perform consumption actions on the message.
6. The device according to claim 5, characterized in that The message generation module is specifically used for: Set filter conditions on the user submission page; According to the screening conditions of the transaction to be initiated submitted by the user on the submission page, all business records that meet the conditions are extracted on the application server side, and business processing instructions for each transaction are generated according to the set format; Prioritize the generated business processing instructions based on the business transaction type and the urgency of the business transaction. The business processing instructions of the set priority are produced and submitted to generate a message, and the submission success result is returned after the submission is completed.
7. The device according to claim 5, characterized in that Monitoring module, specifically used for: When performing production submission operations, obtain the information notification method submitted by the user; Monitor the operation indicators including the execution of consumer consumption actions, the consumption speed of message queues, production submission operations, and system memory usage, and generate monitoring data; Perform real-time analysis on monitoring data and generate monitoring information based on the warning threshold of each operating indicator when the operating indicator exceeds the warning threshold, and push it to the user in the form of information notification submitted by the user.
8. The device according to claim 7, characterized in that The monitoring module is also used to: Generate a transaction execution report based on the submitting user and business type dimensions to count production submissions, pending messages, and processed messages within a set time period; The transaction execution report is pushed to the user according to the information notification method submitted by the user.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
