Data processing method and device, equipment and computer readable storage medium
By dynamically adjusting the number of producer and consumer threads, the problem of message backlog in the message queue was solved, thereby improving message processing efficiency and maximizing resource utilization.
Patent Information
- Application Number
- CN202411732423.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2026-05-29
AI Technical Summary
In a business system, if the producer produces message objects too quickly while the consumer consumes them too slowly, it will cause excessive message backlog in the message queue, affecting normal business processes.
By obtaining the message queues and their quantities corresponding to the business messages, it is determined whether the preset thread conversion conditions are met. Based on the number of messages and the number of producer threads, the number of producer and consumer threads is dynamically adjusted to reduce producer threads and increase consumer threads, thereby dynamically adjusting the ratio of message production to consumption speed and reducing message backlog in the message queue.
It effectively reduces the occurrence rate of message backlog in the message queue, ensures maximum utilization of resources, and improves message processing efficiency.
Smart Images
Figure CN122120217A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Technology
[0002] Currently, message queue middleware is mainly used for message object production and consumption. The system acts as both a message producer and a message consumer. The producer sends the processed data to the message queue, and the consumer retrieves the data from the message queue for consumption.
[0003] However, if the producer produces message objects too quickly while the consumer consumes them too slowly in the business system, it will cause excessive message backlog in the message queue, making the message queue unable to operate normally and affecting normal business processes.
[0004] Therefore, how to reduce the occurrence of message backlog in message queues is an urgent problem that needs to be solved. Summary of the Invention
[0005] The main objective of this application is to provide a data processing method, apparatus, device, and computer-readable storage medium, which aims to solve the technical problem of how to reduce the occurrence rate of message backlog in message queues.
[0006] To achieve the above objectives, this application provides a data processing method, which includes the following steps:
[0007] Get the first message queue corresponding to the business message, and get the first message count of the business message and the second message count of the messages that already exist in the first message queue;
[0008] Based on the first message count and the second message count, determine whether the first message queue meets the first preset thread conversion condition;
[0009] If the first message queue meets the preset thread replacement conditions, then the number of threads to be converted is determined based on the number of the first messages, the number of the second messages, and the number of producer threads corresponding to the first message queue.
[0010] Based on the first number of threads to be converted, perform a thread conversion operation on the producer thread corresponding to the first message queue.
[0011] Further, the step of determining whether the first message queue meets the first preset thread conversion condition based on the first message count and the second message count includes:
[0012] The message number threshold corresponding to the first message queue is determined based on the message number threshold ratio and the maximum message number.
[0013] The first number of messages to be sent is determined based on the first number of messages and the second number of messages.
[0014] Based on the message quantity threshold and the first number of messages to be sent, it is determined whether the first message queue meets the first preset thread conversion condition. If the first number of messages to be sent is greater than the message quantity threshold, it is determined that the first message queue meets the first preset thread conversion condition.
[0015] Further, the step of determining the number of threads to be converted based on the number of the first messages, the number of the second messages, and the number of producer threads corresponding to the first message queue includes:
[0016] The first number of messages to be sent is determined based on the first number of messages and the second number of messages.
[0017] Obtain the message count threshold and the maximum message count corresponding to the first message queue;
[0018] The number of threads to be converted is determined based on the first number of messages to be sent, the message number threshold, the maximum message number, and the number of producer threads.
[0019] Further, the step of performing thread conversion operation on the producer thread corresponding to the first message queue based on the first number of threads to be converted includes:
[0020] Sequentially retrieve the producer thread object from the producer thread queue corresponding to the first message queue;
[0021] Based on the JVM address corresponding to the producer thread object, modify the Hash type data corresponding to the producer thread object in Redis to the exit status;
[0022] Create a new consumer thread object and set the first type identifier of the consumer thread object;
[0023] Based on the first type identifier, the object information of the newly created consumer thread object is stored in the consumer thread queue corresponding to the first message queue in Redis;
[0024] Update the first thread conversion count. If the updated first thread conversion count is less than the first number of threads to be converted, return to the step of retrieving producer thread objects from the producer thread queue corresponding to the first message queue in turn.
[0025] Furthermore, after the step of performing thread conversion operation on the producer thread corresponding to the first message queue based on the first number of threads to be converted, the method further includes:
[0026] The business messages are stored in Redis, and the number of third messages that already exist in the first message queue is obtained in real time.
[0027] The second number of messages to be sent is determined based on the number of the first messages and the number of the third messages.
[0028] If the number of the first messages to be sent is less than or equal to the message number threshold corresponding to the first message queue, then a thread switching operation is performed on the consumer thread corresponding to the first message queue.
[0029] Further, the step of performing a thread switching operation on the consumer thread corresponding to the first message queue includes:
[0030] Get the producer production rate and consumer consumption rate corresponding to the first message queue;
[0031] The second number of threads to be converted is determined based on the first number of threads to be converted, the producer's production rate, and the consumer's consumption rate.
[0032] Based on the second number of threads to be converted, a thread conversion operation is performed on the consumer thread corresponding to the first message queue.
[0033] Further, the step of performing thread conversion operation on the consumer thread corresponding to the first message queue based on the second number of threads to be converted:
[0034] Retrieve consumer thread objects sequentially from the consumer thread queue corresponding to the first message queue;
[0035] Based on the JVM address corresponding to the consumer thread object, modify the Hash type data corresponding to the consumer thread object in Redis to the exit status;
[0036] Create a new producer thread object and set the second type identifier of the producer thread object;
[0037] Based on the second type identifier, the object information of the newly created producer thread object is stored in the producer thread queue corresponding to the first message queue in Redis;
[0038] Update the number of transformations for the second thread. If the updated number of transformations for the second thread is less than the number of threads to be transformed, return to the previous step of retrieving consumer thread objects from the consumer thread queue corresponding to the first message queue.
[0039] Furthermore, to achieve the above objectives, this application also provides a data processing apparatus, the data processing apparatus comprising:
[0040] The acquisition module is used to acquire the first message queue corresponding to the business message, and to acquire the first message count of the business message and the second message count of the messages already existing in the first message queue.
[0041] The first determining module is used to determine whether the first message queue meets the first preset thread conversion condition based on the first message quantity and the second message quantity;
[0042] The second determining module is used to determine the number of threads to be converted based on the number of the first messages, the number of the second messages, and the number of producer threads corresponding to the first message queue if the first message queue meets the preset thread replacement conditions.
[0043] The thread conversion module is used to perform thread conversion operations on the producer thread corresponding to the first message queue based on the first number of threads to be converted.
[0044] In addition, to achieve the above objectives, this application also provides a data processing device, which includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program, when executed by the processor, implements the steps of the aforementioned data processing method.
[0045] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a data processing program, which, when executed by a processor, implements the steps of the aforementioned data processing method.
[0046] This application obtains a first message queue corresponding to a business message, and acquires the first message count and the second message count already existing in the first message queue. Then, based on the first and second message counts, it determines whether the first message queue meets a first preset thread conversion condition. If the first message queue meets the preset thread conversion condition, it determines a first number of threads to be converted based on the first, second, and corresponding producer threads. Then, based on the first number of threads to be converted, it performs a thread conversion operation on the producer threads corresponding to the first message queue. By determining whether the message queue meets the first preset thread conversion condition based on the number of business messages and existing messages, the application performs a thread conversion operation on the producer threads corresponding to the message queue to reduce the number of producer threads while increasing the number of consumer threads, dynamically adjusting the ratio of message object production speed to consumption speed to reduce the number of messages in the message queue and decrease the message backlog rate. Simultaneously, thread conversion is performed without changing resources, ensuring maximum resource utilization. Attached Figure Description
[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 This is a flowchart illustrating an embodiment of the data processing method of this application.
[0050] Figure 2 This is a schematic diagram of the functional framework of an embodiment of the data processing method of this application;
[0051] Figure 3 This is a schematic diagram of the module structure of the data processing device according to an embodiment of this application;
[0052] Figure 4 This is a schematic diagram of the module structure of the data processing device according to an embodiment of this application.
[0053] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0054] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0055] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0056] The main solution of this application is as follows: obtain the first message queue corresponding to the business message, and obtain the first message count of the business message and the second message count of the messages already existing in the first message queue; based on the first message count and the second message count, determine whether the first message queue meets the first preset thread conversion condition; if the first message queue meets the preset thread replacement condition, determine the first number of threads to be converted based on the first message count, the second message count, and the number of producer threads corresponding to the first message queue; based on the first number of threads to be converted, perform a thread conversion operation on the producer threads corresponding to the first message queue.
[0057] Currently, message queue middleware is mainly used for message object production and consumption. The system acts as both a message producer and a message consumer. The producer sends the processed data to the message queue, and the consumer retrieves the data from the message queue for consumption.
[0058] However, if producers in a business system generate message objects too quickly while consumers consume them too slowly, it can lead to excessive message backlog in the message queue, causing it to malfunction and disrupting normal business processes. Therefore, reducing the rate of message backlog in the message queue is a pressing issue that needs to be addressed.
[0059] This application determines whether a message queue meets the first preset thread conversion condition by measuring the number of business messages and the number of existing messages. It then performs a thread conversion operation on the producer thread corresponding to the message queue to reduce the number of producer threads while increasing the number of consumer threads. This dynamically adjusts the ratio of message object production speed to consumption speed to reduce the message backlog rate in the message queue.
[0060] It should be noted that the execution subject in this embodiment can be a data processing device, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a data processing device capable of performing the above functions. This embodiment does not specifically limit it in this way. The following uses a data processing device as the execution subject as an example to describe this embodiment and the following embodiments.
[0061] Based on this, this application proposes a data processing method according to a first embodiment, please refer to... Figure 1The data processing method includes steps S101 to S103:
[0062] Step S101: Obtain the first message queue corresponding to the business message, and obtain the first message count of the business message and the second message count of the messages already existing in the first message queue;
[0063] Step S102: Based on the first message count and the second message count, determine whether the first message queue meets the first preset thread conversion condition;
[0064] Step S103: If the first message queue meets the preset thread replacement condition, then the number of threads to be converted is determined based on the number of the first messages, the number of the second messages, and the number of producer threads corresponding to the first message queue.
[0065] Step S104: Based on the first number of threads to be converted, perform a thread conversion operation on the producer thread corresponding to the first message queue.
[0066] The general asynchronous notification mechanism corresponding to the data processing method in this application is referenced by the business system in the form of a program package, such as... Figure 2 As shown, Figure 2 This refers to the overall functional architecture corresponding to the program package.
[0067] There are multiple message queues, including message queue 1, message queue 2, message queue 3, message queue 4, etc. Each message object can only be placed in one message queue. Message queue 1 and message queue 2 in the blue box are just examples to illustrate that there are multiple message objects in the message queue. Message queue 3 and message queue 4 have the same structure as message queue 1.
[0068] The program entry point is to call the method of this package. The package needs to be imported into the business module. After importing, you only need to pass the corresponding parameters when calling the method, and the package will complete the asynchronous notification operation and the asynchronous notification compensation operation.
[0069] A message is data transmitted between programs.
[0070] A message queue is a container for storing messages, used for message passing and synchronization. Message queues allow applications or components to put messages into a queue, which are then retrieved and processed asynchronously by other applications or components.
[0071] Redis is a powerful and flexible in-memory data storage system that can be used to solve various data storage and caching problems.
[0072] Message queue monitoring refers to the process where, every time the business system calls the application package, Redis (which records the number of messages in a message queue based on the message body and the name of the corresponding message queue during the call, and operates on the corresponding record value in Redis each time the message queue is accessed, so this monitoring is based on this record) listens to all message queues and calculates whether the current message queue exceeds the set message count threshold. If it does, the application package (automatically triggered by the application package, which automatically retrieves the record value of the corresponding message queue stored in Redis, containing the number of messages in the current message queue) will automatically trigger the creation of thread objects and the switching of thread roles to increase the number of consumers and complete message consumption, thereby preventing message backlog in the message queue and affecting normal business processes.
[0073] Message tagging refers to marking a message in Redis when a program puts it into a message queue, to prevent a message from being consumed by multiple consumer threads.
[0074] To import the package into a business system (a business system that requires an asynchronous notification mechanism), you first need to configure the information in the business system's configuration file. Taking a yml (or properties file) configuration file as an example, the configuration information includes configuration file parameters and method parameters.
[0075] The configuration file parameters include queueThreshold and queueMaxNumber.
[0076] queueThreshold is a threshold percentage of the number of message queues (message count threshold ratio). The unit of the message count threshold ratio is %, and the message count threshold ratio is an integer with a value range of 1 to 100.
[0077] queueMaxNumber is the maximum number of message queues (maximum number of messages), used to monitor whether the threshold has been reached.
[0078] The method parameters include exchangeName, routingKey, queueName, data, sendCompensationType, sendCompensationTime, sendCompensationNumber, conventionInfo, and new SendCallback(){}.
[0079] exchangeName is the name of the exchange in the message queue, a string type, and is a required field;
[0080] routingKey is the routing key, a string type, and is a required field.
[0081] `queueName` is the name of the message queue, a string type, and is a required field.
[0082] `data` is the data object for sending the message; it is a string type and is required.
[0083] `sendCompensationType` is the asynchronous notification compensation mechanism type, an integer type, where 0 = geometric type, 1 = arithmetic type, and 2 = custom. When real-time asynchronous notification fails to be sent, the asynchronous notification compensation mechanism will be triggered. This parameter controls the notification interval of the compensation mechanism, and the default value is 0. It is not a required field.
[0084] sendCompensationTime is the time (in seconds) for the asynchronous notification compensation mechanism. This parameter is an integer array type. When sendCompensationType is 0 or 1, there is only one array variable for this parameter. If sendCompensationType is 2, it needs to be used in conjunction with the sendCompensationNumber parameter. The number of array member variables must correspond to sendCompensationNumber. This is a required field.
[0085] sendCompensationNumber is the maximum number of times the asynchronous notification compensation mechanism can be sent. It is an integer with a value range of 0 to 100. If every asynchronous notification compensation fails, then the maximum number of compensation attempts will be this parameter value. The default value is 5. It is not a required field.
[0086] conventionInfo is a string type of convention information. When asynchronously notifying a third-party system, if the third-party system returns this information, the asynchronous notification is considered successful; otherwise, the asynchronous notification fails. The default value is "SUCCESS" and it is not a required field.
[0087] `new SendCallback(){}` is a callback function that is executed after the asynchronous notification completes its operation (the notification status can be either success or failure). This function is used to handle the subsequent processes of the record.
[0088] The package's specific implementation of thread operations involves first designing the code to be executed in a multi-threaded mode. When implementing the thread code logic, it selects to bind a FutureTask object. Here, the specific thread execution code can be placed in new Callable(), initializing the number of producer thread objects and consumer thread objects to 1:1 (assuming S is 100, then the number of producer threads is 50 and the number of consumer threads is 50). Then, when creating thread objects, the thread object information is stored in the corresponding Redis List object. Based on the Redis List, the number of different types of thread objects can be obtained. Furthermore, the package's thread management code logic borrows hook methods encountered in other languages (i.e., code logic that needs to be processed before or after the actual execution of a task, which can be linked with Redis in the hook method).
[0089] Based on the hook method borrowed from other methods, when a thread object is created, the key that controls the state of the thread object is stored in Redis using a hash type (the hash type is a key-value structure, i.e., key-data, storing the JVM address of the thread object as the key and the state key as the value in Redis). After the thread object completes its task or pauses its execution due to resource terminal issues, the state key that controls the thread object is retrieved from Redis, and then the lifecycle of the thread object is terminated based on the key.
[0090] Each time a new thread object is created, in addition to storing the data in the Redis Hash, the thread object is also saved in the Redis List data type. That is, the thread object information is stored in the Redis List. The corresponding List in Redis is different for producer threads and consumer threads, and it records the thread object information (including the object address information of the JVM where the thread object is located) to facilitate the counting of thread objects.
[0091] After receiving the parameters from the business system, the system checks whether the required fields exist. If the check passes, the producer thread generates a business message, determines the first message queue corresponding to the business message, and retrieves the listening object information corresponding to the first message queue from Redis. The system then retrieves the number of first messages and the number of second messages that already exist in the first message queue. In other words, the system retrieves the number of second messages that already exist in the first message queue by using the listening object information.
[0092] After obtaining the first message count and the second message count, based on the first message count and the second message count, it is determined whether the first message queue meets the first preset thread conversion condition, so as to determine whether adding business messages to the first message queue will cause message backlog in the first message queue. In a feasible implementation, step S101 may include steps S1021 to S1023:
[0093] Step S1021: Determine the message number threshold corresponding to the first message queue based on the message number threshold ratio and the maximum message number.
[0094] Step S1022: Determine the first number of messages to be sent based on the first number of messages and the second number of messages;
[0095] Step S1023: Based on the message quantity threshold and the first number of messages to be sent, determine whether the first message queue meets the first preset thread conversion condition. If the first number of messages to be sent is greater than the message quantity threshold, then the first message queue is determined to meet the first preset thread conversion condition.
[0096] In this embodiment, after obtaining the first message count and the second message count, the first message count to be sent is determined based on the first message count x and the second message count z, where the first message count to be sent = x + z.
[0097] Simultaneously, the message number threshold ratio and the maximum message number corresponding to the first message queue are obtained. Based on the message number threshold ratio queueThreshold and the maximum message number queueMaxNumber, the message number threshold corresponding to the first message queue is determined. Specifically, the message number threshold = queueThreshold × queueMaxNumber ÷ 100.
[0098] Next, based on the message quantity threshold and the first number of messages to be sent, it is determined whether the first message queue meets the first preset thread conversion condition, that is, whether the first number of messages to be sent is greater than the message quantity threshold. If the first number of messages to be sent is greater than the message quantity threshold, it is determined that the first message queue meets the first preset thread conversion condition.
[0099] In this embodiment, if the first message queue meets the preset thread replacement condition, then based on the number of first messages, the number of second messages, and the number of producer threads corresponding to the first message queue, the number of first threads to be converted is determined, wherein the number of first threads to be converted is the number of producer threads converted into consumer threads. Specifically, in a feasible implementation, step S103 may include steps S1031 to S1033:
[0100] Step S1031: Determine the first number of messages to be sent based on the first number of messages and the second number of messages;
[0101] Step S1032: Obtain the message quantity threshold and the maximum message quantity corresponding to the first message queue;
[0102] Step S1033: Determine the number of threads to be converted based on the first number of messages to be sent, the message number threshold, the maximum message number, and the number of producer threads.
[0103] In this embodiment, if the first message queue meets the preset thread replacement conditions, the number of first messages to be sent, the message number threshold, and the maximum message number are obtained. Based on the number of first messages to be sent, the message number threshold, the maximum message number, and the number of producer threads, the number of first threads to be converted is determined. The formula for the number of first threads to be converted, N, is as follows:
[0104] N = (x + z - (queueThreshold × queueMaxNumber ÷ 100)) ÷ (queueMaxNumber - queueThreshold × queueMaxNumber ÷ 100) × number of producer threads.
[0105] Where x+z is the first number of messages to be sent, queueThreshold×queueMaxNumber÷100 is the message number threshold, and queueMaxNumber is the maximum number of messages.
[0106] For example, if the first number of messages to be sent is x + z = 90, the message threshold is queueThreshold × queueMaxNumber ÷ 100 = 75, queueMaxNumber = 100, and the number of producer threads is 50, then: N = (90 - 75) ÷ (100 - 75) × 50 = 15 ÷ 25 × 50 = 0.6 × 50 = 30. That is to say, 30 producer threads need to be converted into consumer threads.
[0107] After obtaining the first number of threads to be converted, a thread conversion operation is performed on the producer thread corresponding to the first message queue based on the first number of threads to be converted, so as to convert the producer thread of the first number of threads to be converted into a consumer thread, that is, to end the life cycle of the conversion of the producer thread of the first number of threads to be converted, and to create a new consumer thread for the conversion of the producer thread of the first number of threads to be converted. Specifically, in a feasible implementation, step S104 may include steps S1041 to S1045:
[0108] Step S1041: Sequentially retrieve producer thread objects from the producer thread queue corresponding to the first message queue;
[0109] Step S1042: Based on the JVM address corresponding to the producer thread object, modify the Hash type data corresponding to the producer thread object in Redis to the exit status.
[0110] Step S1043: Create a new consumer thread object and set the first type identifier of the consumer thread object;
[0111] Step S1044: Based on the first type identifier, store the object information of the newly created consumer thread object into the consumer thread queue corresponding to the first message queue in Redis.
[0112] Step S1045: Update the number of first thread conversions. If the updated number of first thread conversions is less than the number of first threads to be converted, return to the step of sequentially retrieving producer thread objects from the producer thread queue corresponding to the first message queue.
[0113] In this embodiment, when the producer thread performs a thread conversion operation, the producer thread object is retrieved sequentially from the producer thread queue corresponding to the first message queue. Specifically, a thread object information, i.e., the producer thread object, is popped from the left side of the producer thread List queue using the LPOP command from Redis. Based on the JVM address corresponding to the producer thread object, the Hash type data corresponding to the producer thread object is modified to an exit state in Redis. That is, based on the JVM address of the producer thread object, the Hash type data of the producer thread object in Redis is modified to an exit state, so that after the producer thread object completes the current task, it exits the current run() method according to the status value in Redis, and the life cycle of the thread corresponding to the producer thread object ends.
[0114] Next, a new consumer thread object is created and its first type identifier is set, i.e., the type value of the consumer thread object is set to C. In the run() method of this consumer thread object, the method to be executed is made concrete. Then, based on the first type identifier, the object information of the newly created consumer thread object is stored in the consumer thread queue corresponding to the first message queue in Redis. That is, the key of the consumer thread object is stored using the Redis Hash type, and the object information is also stored in the Redis consumer List.
[0115] At the same time, update the first thread conversion count, that is, record the number E of threads that have been converted in Redis. The initial data of E is 0. After each loop to process the thread object role conversion process, the first thread conversion count E will be automatically incremented by 1 by updating it. If the updated first thread conversion count is less than the first number of threads to be converted, then return to the step of retrieving the producer thread object from the producer thread queue corresponding to the first message queue in turn, until E=N.
[0116] In this embodiment, when the condition for a thread role change is triggered, the dequeue and enqueue operations are performed based on the thread objects in the Redis List data type. The dequeueed producer thread object's lifecycle is terminated according to the Redis hash state command. Then, this thread resource is created as a consumer thread object. If N is calculated as 1, and the current number of producer threads is 10, then after triggering this thread role change condition, the number of producer threads decreases by 1 to 9, and the number of consumer threads increases by 1. Similarly, there are 9 thread object information entries in the Redis producer thread List and 11 thread object information entries in the consumer thread List. This ensures maximum resource utilization by changing the direction of thread resources from producer to consumer threads without changing system resources.
[0117] It should be noted that the execution weight of the consumer thread object can also be increased. The thread execution rate will change according to the execution weight of the thread object, achieving the desired transformation of the thread object's role. Thus, in the program package, the thread object can be completed during the task execution process, and different types of thread objects can be precisely executed for different tasks, avoiding the problem of not being able to control the thread to execute the corresponding task.
[0118] When a business message is added to the first message queue, a message flag corresponding to the business message is set in Redis. When a consumer thread consumes a message from the first message queue, it determines whether the message is in a consumption state (i.e., whether it is being consumed by another consumer thread) based on the message flag in Redis. If the message is in a consumption state, it is rejected; if the message is not in a consumption state, the consumer linearly generates consumption data based on the message and sends the consumption data to the corresponding third-party system, i.e., via an HTTP / HTTPS request.
[0119] If the network times out or the agreed-upon response string is not received, asynchronous notification compensation is triggered. If this is the first time asynchronous notification compensation has been triggered for the message to be consumed, and an ACK confirmation message has been received, the ACK confirmation message will be sent to the first message queue. The first message queue will then delete the message to be consumed from the queue, and Redis will also delete the message tag for the message to be consumed.
[0120] If the message to be consumed is the first time asynchronous notification compensation is triggered and no ACK confirmation message is received, or if the message to be consumed is the second time asynchronous notification compensation is triggered and no ACK confirmation message is received, then the message to be consumed is re-encapsulated, and the attribute data of the message to be consumed is set to obtain a new message to be consumed. The new message to be consumed is added to the delayed message queue, and the corresponding delay time for the message to be consumed is set. When the delay time expires, the consumption operation is performed on the message to be consumed, thereby realizing the asynchronous notification compensation mechanism.
[0121] The data processing method proposed in this embodiment obtains a first message queue corresponding to a business message, and acquires the first message count and the second message count already existing in the first message queue. Then, based on the first and second message counts, it determines whether the first message queue meets a first preset thread conversion condition. If the first message queue meets the preset thread conversion condition, it determines a first number of threads to be converted based on the first, second, and producer threads corresponding to the first message queue. Then, based on the first number of threads to be converted, it performs a thread conversion operation on the producer threads corresponding to the first message queue. When the message queue meets the first preset thread conversion condition based on the number of business messages and the number of existing messages, the thread conversion operation on the producer threads corresponding to the message queue reduces the number of producer threads while increasing the number of consumer threads, dynamically adjusting the ratio of message object production speed to consumption speed to reduce the number of messages in the message queue and reduce the message backlog rate. Simultaneously, thread conversion is performed without changing resources, ensuring maximum resource utilization.
[0122] In one feasible implementation, after step S104, the data processing method may further include steps S201 to S203:
[0123] Step S201: Store the business message in Redis and obtain the number of third messages that already exist in the first message queue in real time;
[0124] Step S202: Determine the second number of messages to be sent based on the first number of messages and the third number of messages;
[0125] Step S203: If the number of the first messages to be sent is less than or equal to the message number threshold corresponding to the first message queue, then perform a thread switching operation on the consumer thread corresponding to the first message queue.
[0126] In this embodiment, after the producer thread performs a thread conversion operation, ends the lifecycle of the first number of producer threads to be converted, and creates a new number of consumer threads to be converted, the number of producer threads decreases. Therefore, the business messages need to be temporarily stored in Redis Stream type data. That is, the business messages are stored in Redis. The ratio of the producer's production speed to the consumer's consumption speed will become smaller and smaller, and the number of message objects in the first message queue queueName will gradually decrease. The number of third messages that already exist in the first message queue is obtained in real time. Based on the number of first messages and the number of third messages, the number of second messages to be sent is determined. The number of second messages to be sent = the number of first messages + the number of third messages. The program will only restore the normal ratio of production speed to consumption speed when the number of messages in the queueName queue becomes the set threshold (queueThreshold*queueMaxNumber÷100).
[0127] If the number of messages to be sent is less than or equal to the message count threshold corresponding to the first message queue, i.e., the ratio of the restored normal production speed to the consumption speed is equal, a thread switching operation is performed on the consumer thread corresponding to the first message queue to improve message processing efficiency. Specifically, in a feasible implementation, step S203 may include steps S2031 to S2033:
[0128] Step S2031: Obtain the producer production rate and consumer consumption rate corresponding to the first message queue.
[0129] Step S2032: Determine the second number of threads to be converted based on the first number of threads to be converted, the producer's production rate, and the consumer's consumption rate;
[0130] Step S2033: Based on the second number of threads to be converted, perform a thread conversion operation on the consumer thread corresponding to the first message queue.
[0131] In this embodiment, when the consumer thread performs a thread switching operation, it obtains the producer production rate and consumer consumption rate corresponding to the current first message queue. The producer production rate is the number of messages that the producer thread corresponding to the first message queue can generate per second, and the consumer consumption rate is the number of messages that the consumer thread corresponding to the first message queue can consume per second.
[0132] After obtaining the producer's production rate and the consumer's consumption rate, the number of threads to be converted in the second phase is determined based on the first number of threads to be converted, the producer's production rate, and the consumer's consumption rate. Specifically,
[0133] For example, if the producer's production rate is 10, the consumer's consumption rate is 40, and the number of threads waiting to be converted, E, is 30, then:
[0134] Second number of threads to be converted
[0135] After obtaining the second number of threads to be converted, a thread conversion operation is performed on the consumer thread corresponding to the first message queue based on the second number of threads to be converted. Specifically, in a feasible implementation, step S2033 may include steps a1 to a5:
[0136] Step a1: Sequentially retrieve consumer thread objects from the consumer thread queue corresponding to the first message queue;
[0137] Step a2: Based on the JVM address corresponding to the consumer thread object, modify the Hash type data corresponding to the consumer thread object in Redis to the exit status;
[0138] Step a3: Create a new producer thread object and set the second type identifier of the producer thread object;
[0139] Step a4: Based on the second type identifier, store the object information of the newly created producer thread object into the producer thread queue corresponding to the first message queue in Redis;
[0140] Step a5: Update the number of second thread conversions. If the updated number of second thread conversions is less than the number of second threads to be converted, return to the step of retrieving consumer thread objects from the consumer thread queue corresponding to the first message queue in sequence.
[0141] In this embodiment, when the consumer thread performs a thread switching operation, the consumer thread object is retrieved sequentially from the consumer thread queue corresponding to the first message queue. Specifically, a thread object is popped from the left side of the consumer thread List queue in Redis using the LPOP command. Based on the JVM address of the consumer thread object, the Hash type data corresponding to the consumer thread object in Redis is modified to an exit status. That is, based on the JVM address of the consumer thread object, the Hash type data of the consumer thread object in Redis is modified to an exit status, so that after the consumer thread object completes the current task, it exits the current run() method according to the status value in Redis, and the lifecycle of the thread corresponding to the consumer thread object ends.
[0142] Next, a new producer thread object is created and its second type identifier is set, i.e., the type value of the producer thread object is set to P. In the run() method of this producer thread object, the method to be executed is made concrete. Then, based on the second type identifier, the object information of the newly created producer thread object is stored in the producer thread queue corresponding to the first message queue in Redis. That is, the key of the producer thread object is stored using the Redis Hash type, and the information of the producer thread object is stored in the producer List in Redis.
[0143] Simultaneously, update the second thread conversion count, that is, record the number K of threads that have been converted in Redis. The initial data of K is 0. After each loop to process the thread object role conversion process, the second thread conversion count K will be automatically incremented by 1. If the updated second thread conversion count is less than the number of threads to be converted, then return to execute the step of retrieving consumer thread objects from the consumer thread queue corresponding to the first message queue in turn, until K = M.
[0144] The data processing method proposed in this embodiment stores the business messages in Redis and obtains the number of third messages already existing in the first message queue in real time; then, it determines the number of second messages to be sent based on the number of first messages and the number of third messages; then, if the number of first messages to be sent is less than or equal to the message number threshold corresponding to the first message queue, it performs a thread conversion operation on the consumer thread corresponding to the first message queue. By dynamically adjusting the number of producer thread objects and consumer thread objects, the message processing efficiency is improved.
[0145] This application also provides a data processing apparatus, please refer to... Figure 3 The data processing device includes:
[0146] The acquisition module 10 is used to acquire the first message queue corresponding to the business message, and to acquire the first message count of the business message and the second message count of the messages already existing in the first message queue.
[0147] The first determining module 20 is used to determine whether the first message queue meets the first preset thread conversion condition based on the first message quantity and the second message quantity;
[0148] The second determining module 30 is used to determine the number of threads to be converted based on the number of the first messages, the number of the second messages, and the number of producer threads corresponding to the first message queue if the first message queue meets the preset thread replacement conditions.
[0149] The thread conversion module 40 is used to perform thread conversion operations on the producer thread corresponding to the first message queue based on the first number of threads to be converted.
[0150] The data processing apparatus provided in this application, employing the data processing method described in the above embodiments, can solve the technical problem of how to reduce the occurrence rate of message backlog in message queues. Compared with the prior art, the beneficial effects of the data processing apparatus provided in this application are the same as those of the data processing method described in the above embodiments, and other technical features in the data processing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0151] This application provides a data processing apparatus, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method in Embodiment 1 above.
[0152] The following is for reference. Figure 4 This document illustrates a structural diagram of a data processing device suitable for implementing embodiments of this application. The data processing device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The data processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0153] like Figure 4 As shown, the data processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data processing device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the data processing device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show data processing devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.
[0154] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0155] The data processing device provided in this application, employing the data processing method described in the above embodiments, can solve the technical problem of how to reduce the occurrence rate of message backlog in message queues. Compared with the prior art, the beneficial effects of the data processing device provided in this application are the same as those of the data processing method provided in the above embodiments, and other technical features of the data processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0156] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0158] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data processing method described in the above embodiments.
[0159] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0160] The aforementioned computer-readable storage medium may be included in a data processing device or may exist independently without being assembled into a data processing device.
[0161] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by a data processing device, the data processing device causes the data processing device to: acquire a first message queue corresponding to a business message, and acquire a first message count of the business message and a second message count of messages already existing in the first message queue; determine whether the first message queue meets a first preset thread conversion condition based on the first message count and the second message count; if the first message queue meets the preset thread replacement condition, determine a first number of threads to be converted based on the first message count, the second message count, and the number of producer threads corresponding to the first message queue; and perform a thread conversion operation on the producer threads corresponding to the first message queue based on the first number of threads to be converted.
[0162] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0164] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0165] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data processing method, thereby solving the technical problem of how to reduce the occurrence rate of message backlog in message queues. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method provided in the above embodiments, and will not be repeated here.
[0166] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method described above.
[0167] The computer program product provided in this application can solve the technical problem of how to reduce the occurrence rate of message backlog in message queues. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as the beneficial effects of the data processing method provided in the above embodiments, and will not be repeated here.
[0168] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A data processing method, characterized in that, The data processing method includes the following steps: Get the first message queue corresponding to the business message, and get the first message count of the business message and the second message count of the messages that already exist in the first message queue; Based on the first message count and the second message count, determine whether the first message queue meets the first preset thread conversion condition; If the first message queue meets the preset thread replacement conditions, then the number of threads to be converted is determined based on the number of the first messages, the number of the second messages, and the number of producer threads corresponding to the first message queue. Based on the first number of threads to be converted, perform a thread conversion operation on the producer thread corresponding to the first message queue.
2. The data processing method as described in claim 1, characterized in that, The step of determining whether the first message queue meets the first preset thread conversion condition based on the first message count and the second message count includes: The message number threshold corresponding to the first message queue is determined based on the message number threshold ratio and the maximum message number. The first number of messages to be sent is determined based on the first number of messages and the second number of messages. Based on the message quantity threshold and the first number of messages to be sent, it is determined whether the first message queue meets the first preset thread conversion condition. If the first number of messages to be sent is greater than the message quantity threshold, it is determined that the first message queue meets the first preset thread conversion condition.
3. The data processing method as described in claim 1, characterized in that, The step of determining the number of threads to be converted based on the number of the first messages, the number of the second messages, and the number of producer threads corresponding to the first message queue includes: The first number of messages to be sent is determined based on the first number of messages and the second number of messages. Obtain the message count threshold and the maximum message count corresponding to the first message queue; The number of threads to be converted is determined based on the first number of messages to be sent, the message number threshold, the maximum message number, and the number of producer threads.
4. The data processing method as described in claim 1, characterized in that, The step of performing thread conversion operation on the producer thread corresponding to the first message queue based on the first number of threads to be converted includes: Sequentially retrieve the producer thread object from the producer thread queue corresponding to the first message queue; Based on the JVM address corresponding to the producer thread object, modify the Hash type data corresponding to the producer thread object in Redis to the exit status; Create a new consumer thread object and set the first type identifier of the consumer thread object; Based on the first type identifier, the object information of the newly created consumer thread object is stored in the consumer thread queue corresponding to the first message queue in Redis; Update the number of first thread conversions. If the updated number of first thread conversions is less than the number of first threads to be converted, return to the step of retrieving producer thread objects from the producer thread queue corresponding to the first message queue in turn.
5. The data processing method according to any one of claims 1 to 4, characterized in that, After the step of performing thread conversion operation on the producer thread corresponding to the first message queue based on the first number of threads to be converted, the method further includes: The business messages are stored in Redis, and the number of third messages that already exist in the first message queue is obtained in real time. The second number of messages to be sent is determined based on the number of the first messages and the number of the third messages. If the number of the first messages to be sent is less than or equal to the message number threshold corresponding to the first message queue, then a thread switching operation is performed on the consumer thread corresponding to the first message queue.
6. The data processing method as described in claim 5, characterized in that, The step of performing a thread switching operation on the consumer thread corresponding to the first message queue includes: Get the producer production rate and consumer consumption rate corresponding to the first message queue; The second number of threads to be converted is determined based on the first number of threads to be converted, the producer's production rate, and the consumer's consumption rate. Based on the second number of threads to be converted, a thread conversion operation is performed on the consumer thread corresponding to the first message queue.
7. The data processing method as described in claim 6, characterized in that, The step of performing thread conversion operation on the consumer thread corresponding to the first message queue based on the second number of threads to be converted: Retrieve consumer thread objects sequentially from the consumer thread queue corresponding to the first message queue; Based on the JVM address corresponding to the consumer thread object, modify the Hash type data corresponding to the consumer thread object in Redis to the exit status; Create a new producer thread object and set the second type identifier of the producer thread object; Based on the second type identifier, the object information of the newly created producer thread object is stored in the producer thread queue corresponding to the first message queue in Redis; Update the number of transformations for the second thread. If the updated number of transformations for the second thread is less than the number of threads to be transformed, return to the previous step of retrieving consumer thread objects from the consumer thread queue corresponding to the first message queue.
8. A data processing apparatus, characterized in that, The data processing device includes: The acquisition module is used to acquire the first message queue corresponding to the business message, and to acquire the first message count of the business message and the second message count of the messages already existing in the first message queue. The first determining module is used to determine whether the first message queue meets the first preset thread conversion condition based on the first message quantity and the second message quantity; The second determining module is used to determine the number of threads to be converted based on the number of the first messages, the number of the second messages, and the number of producer threads corresponding to the first message queue if the first message queue meets the preset thread replacement conditions. The thread conversion module is used to perform thread conversion operations on the producer thread corresponding to the first message queue based on the first number of threads to be converted.
9. A data processing device, characterized in that, The data processing device includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 8.