Message data processing method and device, storage medium and electronic equipment

Abstract

The present disclosure relates to a message data processing method and device, a storage medium and an electronic device. The method comprises: in response to receiving message data sent by a message queue, determining a data source identifier of the message data; determining a first consumer thread from a first preset thread pool of a primary preset service according to the data source identifier and a preset flow sequence of the message data, the first preset thread pool comprising a plurality of consumer threads; processing the message data through the first consumer thread to generate a first service processing result; determining a second consumer thread from a second preset thread pool of a secondary preset service according to the preset flow sequence and the data source identifier, the second preset thread pool comprising a plurality of consumer threads; and processing the first service processing result through the second consumer thread to generate a second service processing result. Thus, a mapping relationship between the consumer thread and the data source identifier is established to ensure the order of the message processing result and avoid the timing disorder of the processing result.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and more specifically, to a method, apparatus, storage medium, and electronic device for processing message data. Background Technology

[0002] Message Queues (MQ, also known as message middleware) are the infrastructure for building distributed internet applications. The loosely coupled architecture implemented through MQ improves system availability and scalability, making it an optimal design solution for modern applications. The message system is responsible for transferring data from one application to another, allowing applications to focus on the data itself without worrying too much about how it's delivered. Message systems provide a means of decoupling data transmission from application systems and offer high real-time guarantees, making application systems easier to scale and more stable, hence their widespread use in large-scale systems. However, applications cannot control the order in which each processing step finishes during data processing; that is, there is no guarantee that the first thread started will complete processing before the later threads, and the computation process and completion time are uncertain. For some business processes, the time order of messages is sensitive, meaning that the first message to arrive must be processed first. In this case, there is a conflict between conventional high-concurrency processing and the time-order requirements of messages. Summary of the Invention

[0003] The purpose of this disclosure is to provide a method, apparatus, storage medium, and electronic device for processing message data, so as to solve the technical problem that the inability to guarantee the order of processing during message processing leads to disordered processing results.

[0004] To achieve the above objectives, a first aspect of this disclosure provides a method for processing message data, comprising:

[0005] In response to receiving message data sent from the message queue, determine the data source identifier of the message data;

[0006] Based on the data source identifier and the preset flow order of the message data, a first consumer thread is determined from the first preset thread pool of the primary preset service, and the first preset thread pool includes multiple consumer threads;

[0007] The message data is processed by the first consumer thread to generate a first service processing result;

[0008] Based on the preset flow order and the data source identifier, a second consumer thread is determined from the second preset thread pool of the secondary preset service. The second preset thread pool includes multiple consumer threads.

[0009] The second consumer thread processes the first service processing result to generate the second service processing result.

[0010] Optionally, determining the first consumer thread from the first preset thread pool of the primary preset service includes:

[0011] Obtain the first preset data source identifier table of the primary preset service, wherein the first preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers;

[0012] Based on the first preset data source identifier table, determine the first consumer thread corresponding to the data source identifier.

[0013] Optionally, determining the second consumer thread from the second preset thread pool of the secondary preset service includes:

[0014] Obtain the second preset data source identifier table of the secondary preset service, wherein the second preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers;

[0015] Based on the second preset data source identifier table, determine the second consumer thread corresponding to the data source identifier.

[0016] Optionally, determining the first consumer thread from the first preset thread pool of the primary preset service based on the data source identifier and the preset flow order of the message data includes:

[0017] Obtain the partition identifier of the message data in the message queue;

[0018] Based on the preset flow order, the primary preset service for initial consumption processing of the message data is determined;

[0019] Based on the partition identifier, determine the first consumer member corresponding to the partition identifier from among the multiple consumer members of the primary preset service;

[0020] Based on the data source identifier, the first consumer thread is determined from the first preset thread pool of the first consumer group member.

[0021] Optionally, the method further includes:

[0022] Based on the terminal's data processing capabilities, the primary preset service is divided into consumer groups to generate the multiple consumer groups;

[0023] The multiple consumer group members are divided into threads according to a preset partitioning rule to generate multiple preset thread pools.

[0024] Optionally, the method further includes:

[0025] Based on the preset flow order and the second service processing result, a target service processing result is generated;

[0026] The result of the target service processing is sent to the client.

[0027] According to a second aspect of the present disclosure, a message data processing apparatus is provided, the apparatus comprising:

[0028] The first determining module is used to determine the data source identifier of the message data in response to receiving message data sent by the message queue.

[0029] The second determining module is used to determine a first consumer thread from a first preset thread pool of the primary preset service based on the data source identifier and the preset flow order of the message data. The first preset thread pool includes multiple consumer threads.

[0030] The first generation module is used to process the message data through the first consumption thread to generate a first service processing result;

[0031] The third determining module is used to determine a second consumer thread from the second preset thread pool of the secondary preset service according to the preset flow order and the data source identifier. The second preset thread pool includes multiple consumer threads.

[0032] The second generation module is used to process the first service processing result through the second consumer thread to generate a second service processing result.

[0033] Optionally, the second determining module is configured to:

[0034] Obtain the first preset data source identifier table of the primary preset service, wherein the first preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers;

[0035] Based on the first preset data source identifier table, determine the first consumer thread corresponding to the data source identifier.

[0036] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in any of the first aspects of the present disclosure.

[0037] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0038] A memory on which computer programs are stored;

[0039] A processor for executing the computer program in the memory to implement the steps of the method according to any one of the first aspects of this disclosure.

[0040] The above technical solution, in response to receiving message data from a message queue, determines the data source identifier. Based on the data source identifier and the preset flow order of the message data, a first consumer thread is selected from the first preset thread pool of the primary preset service. The first preset thread pool includes multiple consumer threads. The first consumer thread processes the message data to generate a first service processing result. Based on the preset flow order and the data source identifier, a second consumer thread is selected from the second preset thread pool of the secondary preset service. The second preset thread pool includes multiple consumer threads. The second consumer thread processes the first service processing result to generate a second service processing result. By establishing a mapping relationship between consumer threads and data source identifiers, messages from the same data source are processed in the same thread. The sequential nature of thread message processing is utilized to ensure the order of message processing results and avoid disordered processing.

[0041] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a schematic diagram illustrating a publish-subscribe model for a messaging system according to an exemplary embodiment.

[0044] Figure 2 This is a schematic diagram illustrating a message system broadcast model according to an exemplary embodiment.

[0045] Figure 3 This is a schematic diagram illustrating a multi-threaded processing mode for a consumer client according to an exemplary embodiment.

[0046] Figure 4 This is a schematic diagram illustrating a consumer-side multi-instance model according to an exemplary embodiment.

[0047] Figure 5 This is a schematic diagram illustrating a multi-threaded, multi-instance message model according to an exemplary embodiment.

[0048] Figure 6 This is a flowchart illustrating a message data processing method according to an exemplary embodiment.

[0049] Figure 7This is a flowchart illustrating a method for determining a consumer thread according to an exemplary embodiment.

[0050] Figure 8 This is a schematic diagram illustrating a group consumption pattern according to an exemplary embodiment.

[0051] Figure 9 This is a schematic diagram illustrating a multi-instance, multi-threaded overall consumption pattern according to an exemplary embodiment.

[0052] Figure 10 This is a block diagram illustrating a message data processing apparatus according to an exemplary embodiment.

[0053] Figure 11 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0054] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0055] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0056] Before outlining the technical solution of this application, it is necessary to describe the messaging system in this application. The messaging system in this application generally has two transmission modes: point-to-point mode and publish-subscribe mode. Figure 1 This is a schematic diagram illustrating a publish-subscribe model for a messaging system according to an exemplary embodiment, such as... Figure 1 As shown, after a message producer sends message data to the messaging system, the system reviews the message data to determine the subscriber identifier and then sends the message data to the corresponding consumer based on that identifier. It's important to note that the same message data can include multiple subscribed consumers. The messaging system checks the subscription data of each message data item and sends the message data to each of those consumers. For example... Figure 2 This is a schematic diagram illustrating a broadcast model of a messaging system according to an exemplary embodiment, such as... Figure 2 As shown, this model uses a publish-subscribe approach to send message data to multiple data processing consumers. This broadcast model improves message processing efficiency and provides a high real-time guarantee for message processing. On the business system side, to improve message processing throughput, a thread pool is used to process subscribed messages concurrently. Figure 3This is a schematic diagram illustrating a multi-threaded processing mode for a consumer client according to an exemplary embodiment, such as... Figure 3 As shown, this processing model is suitable for servers requiring a large volume of message processing. The server includes a thread pool with multiple threads for parallel processing of received message data. After receiving message data from the messaging system, the message-consuming client splits the message data into multiple sub-messages and sends these sub-messages to different message-processing threads for parallel processing, thereby improving the efficiency of message processing on the server. For microservice architecture systems, the server can also consume and process message data from the messaging system in parallel by running multiple service instances. Figure 4 This is a schematic diagram illustrating a consumer-side multi-instance model according to an exemplary embodiment, such as... Figure 4 As shown, in a client equipped with a microservice architecture system, multiple business instances are allocated. When the client receives message data sent by the consumption system, it sends different types of messages to different business instances. Different computers are used to implement multi-process concurrent processing, thereby improving the efficiency of message processing in the client.

[0057] When processing message data sent by a messaging system using the model in the example above, the system thread adjuster in the application server controls the process. Thread execution is affected by I / O interfaces, system terminals, etc., and the processing time for each step cannot be guaranteed to be equal. In other words, it's not guaranteed that the first thread started will complete processing before the later thread. Therefore, the computation process and completion time of the corresponding data processing are uncertain. For multiple processes, since they run in different physical computing units and have no temporal correlation, their computation time and completion time are even more uncertain. Furthermore, some business processes are sensitive to the time order of messages, requiring that the first arriving message data be processed first, following the order of message reception. In this case, there is a conflict between conventional high-concurrency processing and the time-order requirement of messages, leading to disordered message processing results on the server. Therefore, a logical processing method is needed to ensure both high concurrency processing capabilities and the business requirement of message order processing.

[0058] For example, using the monitoring of a large number of edge devices by an IoT platform as an example, this illustrates how to perform message timing processing to ensure that the latest device status is not misaligned. The most basic function of IoT is remote monitoring of edge devices, which involves collecting the operating status of numerous edge devices, centrally processing and storing the data, and pushing it to clients for display. Typically, device operating status is sent and propagated in the form of messages. Given the large volume of message data, to achieve real-time display, a message system is needed to handle concurrent message data, thereby improving message processing efficiency. Since IoT platforms can connect to a large number of edge devices and have a high data reporting frequency, a medium-sized IoT platform can handle up to 5000 messages per second. To respond to and process large amounts of message data in a timely manner, IoT platforms generally adopt a microservice architecture, with each service deployed in multiple instances, and each instance internally running multiple threads to improve message processing capabilities. For example... Figure 5 This is a schematic diagram illustrating a multi-threaded, multi-instance message model according to an exemplary embodiment, such as... Figure 5 As shown in the diagram, the overall message processing model of an IoT platform is illustrated. The IoT consists of multiple services (businesses), each service comprising multiple instances. Each instance uses a thread pool, and each thread pool contains multiple message processing threads. Data flows between different services through server-defined processing flows. For example,... Figure 5 As shown, message data is processed by service A and then flows to service B, which in turn forwards it to service C. Finally, the device status is pushed to the client for real-time display. Because the client uses a multi-process and multi-threaded data processing method, without ensuring timely delivery, message order can be disrupted, leading to incorrect display of the latest status on the client. For example, if the edge device is a sensor monitoring the open / closed status of a door, the sensor transmits data to the client for processing, and then displays the door's open / closed status on the client's display. In reality, the door is open first and then closed, but due to the message timing error, the status transmitted to the client is initially closed and then open, ultimately remaining in the open state, causing an error in the displayed door status.

[0059] In view of this, this embodiment provides a method, apparatus, storage medium, and electronic device for processing message data. Figure 6 This is a flowchart illustrating a message data processing method according to an exemplary embodiment, such as... Figure 6 As shown, the method includes the following steps.

[0060] Step S101: In response to receiving message data sent by the message queue, determine the data source identifier of the message data.

[0061] For example, this embodiment is applied to a server, which is equipped with a multi-threaded, multi-instance message processing model. This model performs multi-instance, multi-threaded data processing on the data messages received by the message system to improve data processing efficiency. The message data received by the message system includes a data source identifier, which identifies the source of the message data. For example, the data source identifier can be the device number of the acquisition device corresponding to the message data. This device number is unique; different acquisition devices will have different device numbers. The message system can generate a data source identifier based on the source of the message data and store this identifier as attribute information within the message data. The message system identifies the consumers of the message data, checks the attribute information, and determines the data source identifier.

[0062] Step S102: Based on the data source identifier and the preset flow order of message data, determine the first consumer thread from the first preset thread pool of the primary preset service. The first preset thread pool includes multiple consumer threads.

[0063] For example, the server in this embodiment includes multiple preset services. Message data sent by the messaging system needs to be processed by each preset service before being sent to the client's display device. According to the message data processing flow, a preset flow order is pre-set in the server. Message data is processed sequentially in each preset service according to this preset flow order. For example, after message data arrives from the messaging system, the primary preset service in the preset flow order processes the message data and generates a corresponding data processing result. This data processing result is then passed to the secondary preset service, which processes the data processing result to generate a secondary data processing result. This secondary data processing result is then processed by the next-level preset service according to the preset flow order, until it reaches the final preset service in the preset flow order. Finally, the last preset service processes the previous level's data processing result before sending it to the client. Each preset service in the preset flow order includes a preset thread pool, which contains multiple data processing threads. After determining the primary preset service in the server by pre-setting the flow order, the first consumer thread for processing the message data is determined from the first preset thread pool of the primary preset service based on the data source identifier. The first preset thread pool of the primary preset service includes multiple consumer threads. Each consumer thread processes data according to the order in which it receives the data. Within the same consumer thread, the message data received earlier is processed first, and subsequently received message data must wait until the preceding message data has been processed before it can be processed by the consumer thread.

[0064] Optionally, in one embodiment, step S102 above includes:

[0065] Obtain the first preset data source identifier table of the primary preset service. The first preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers.

[0066] Based on the first preset data source identifier table, determine the first consumer thread corresponding to the data source identifier.

[0067] It is worth mentioning that each preset service in this embodiment is configured with a preset data source identifier table. This preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers in the preset thread pool. One data source identifier corresponds to one consumer thread, and one consumer thread may include multiple data source identifiers. Based on this preset data source identifier table, the consumer thread corresponding to the data source identifier in the message data is determined, thereby determining the target consumer thread in the preset thread pool for processing the message data. For example, in this embodiment, for each consumer thread in the preset thread pool in each preset service, an attribute list is bound to store the data source identifier of the message data. The message data is sent to the corresponding consumer thread by comparing the data source identifier of the message data.

[0068] Step S103: The message data is processed by the first consumer thread to generate the first service processing result.

[0069] For example, the first consumer thread processes the received message data to generate a first service processing result. It's worth noting that the processing method of the consumer thread in this embodiment can be set according to the actual data processing needs of the server; this embodiment does not limit this. Based on this data processing method, the message data is processed to generate a first service processing result corresponding to the primary preset service. Through this method, message data sent from the same device is sent to the same consumer thread of the primary preset service for processing. Utilizing the sequential processing of the consumer thread ensures the timeliness of the message data processing results, thereby avoiding the problem of disordered processing results. This ensures both processing efficiency and the timeliness of the processing results.

[0070] Step S104: Based on the preset flow order and data source identifier, determine the second consumer thread from the second preset thread pool of the secondary preset service. The second preset thread pool includes multiple consumer threads.

[0071] For example, in this embodiment, after the primary preset service generates the corresponding first service processing result in the above manner, it sends the first service processing result to the secondary preset service for processing according to the preset data flow data. The secondary preset service determines the second consumer thread from the corresponding second preset thread pool according to the data source identifier. The method of determining the second consumer thread according to the data source identifier can refer to the method of determining the first consumer thread in step S103 above, and will not be repeated here. The second preset thread pool includes multiple consumer threads, and different consumer threads correspond to different edge devices. Message data generated by the same edge device will be transmitted to the same consumer thread for data processing.

[0072] Alternatively, in another embodiment, step S103 above includes:

[0073] Obtain the second preset data source identifier table of the secondary preset service. The second preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers.

[0074] Based on the second preset data source identifier table, determine the second consumer thread corresponding to the data source identifier.

[0075] For example, in this embodiment, the secondary preset service includes a second preset data source identifier table. The method of determining the second consumer thread through the second preset data source identifier table in this embodiment is the same as in the above embodiments. You can refer to the above embodiments, and it will not be repeated here.

[0076] Step S105: The second consumer thread processes the first service processing result to generate the second service processing result.

[0077] For example, the first service processing result generated by the primary preset service in the above steps is processed by the preset data processing method of the second consumer thread to generate the second service processing result corresponding to the secondary preset service.

[0078] Optionally, after step S105 above, the method further includes:

[0079] Based on the preset flow order and the result of the second service processing, the result of the target service processing is generated.

[0080] Send the target service processing result to the client.

[0081] For example, in this embodiment, referring to the message data processing flow in the primary preset service and secondary preset service, the processing results of each preset service are transferred according to the preset transfer order until they are transferred to the final preset service in the preset transfer order. After the final preset service processes the data processing results of the previous preset service using the above data processing method, the target service processing result is generated and transmitted to the client. The target data processing result corresponding to the final preset service is then displayed on the client. For example, the client can also alternately display the data processing results of each preset service in the preset flow order. For instance, after the primary preset service processes the message data and generates the first service processing result, the server sends the first service processing result to the client, where it is displayed. Then, according to the preset flow order, the first service processing result is transferred to the secondary preset service for data processing, generating the second service processing result. The server then sends the second service processing result to the client, where it is displayed to replace the first service processing result. This data flow according to the preset flow order achieves the alternating display of data processing results in the client.

[0082] The above technical solution, in response to receiving message data from a message queue, determines the data source identifier. Based on the data source identifier and the preset flow order of the message data, a first consumer thread is selected from the first preset thread pool of the primary preset service. The first preset thread pool includes multiple consumer threads. The first consumer thread processes the message data to generate a first service processing result. Based on the preset flow order and the data source identifier, a second consumer thread is selected from the second preset thread pool of the secondary preset service. The second preset thread pool includes multiple consumer threads. The second consumer thread processes the first service processing result to generate a second service processing result. By establishing a mapping relationship between consumer threads and data source identifiers, messages from the same data source are processed in the same thread. The sequential nature of thread message processing is utilized to ensure the order of message processing results and avoid disordered processing.

[0083] Figure 7 This is a flowchart illustrating a method for determining a consumer thread according to an exemplary embodiment, such as... Figure 7 As shown, in one embodiment, step S102 includes:

[0084] Step S201: Obtain the partition identifier of the message data in the message queue.

[0085] Step S202: Determine the primary preset service for initial consumption processing of message data according to the preset flow order.

[0086] Step S203: Based on the partition identifier, determine the first consumer group member corresponding to the partition identifier from among the multiple consumer group members of the primary preset service.

[0087] Step S204: Determine the first consumer thread from the first preset thread pool of the first consumer group member based on the data source identifier.

[0088] It is worth mentioning that the server in this embodiment is equipped with a multi-instance, multi-group consumption model, for example, Figure 8 This is a schematic diagram illustrating a group consumption pattern according to an exemplary embodiment, such as... Figure 8 As shown, in this group consumption mode, the message system is loaded with a group consumption strategy. Different consumer groups can consume message data in parallel; that is, members of group A and group B can receive the same message data without affecting each other. Different consumption threads within the same group share consumption. That is, if member 1 in consumer group A consumes message A, other members in the same group will not receive message A. This also corresponds to the mapping relationship between group members and message system partitions. This feature can be used to solve the time-series correlation between different processes, that is, to allow different processes to consume messages that are not time-series correlated. Specifically, different instances of the same service are set to the same consumer group identifier, thus forming shared consumption within the group and establishing a fixed mapping with message system partitions. Different instances consume messages from different partitions and send time-series correlated messages to the same partition. In this embodiment, the messaging system manages edge devices by partition. Message data generated by edge devices within the same partition is allocated to the same message partition for relay and storage. After reading the message data, the messaging system sends the partition identifier and the message data to the client. After determining the primary preset service in the server according to the preset flow order, the received message data is sent to the primary preset service for data processing. Based on the partition identifier of the message data, the first consumer group member to process the message data is determined from multiple consumer group members of the primary preset service. Based on the data identifier in the message data, the first consumer thread to process the message data is determined from the first preset thread pool of the first consumer group member.

[0089] Alternatively, in another implementation, the method further includes:

[0090] Based on the terminal's data processing capabilities, the primary preset services are divided into consumer groups to generate multiple consumer groups.

[0091] Multiple consumer group members are divided into threads according to preset partitioning rules to generate multiple preset thread pools.

[0092] For example, in this embodiment, to increase the server's message processing capacity, the preset service can be divided into consumer groups based on the data computing capabilities of the deployed servers, and these consumer groups can be further divided into threads, thereby generating multiple consumer groups. Within each consumer group, multiple preset thread pools are generated, thus appropriately increasing the number of partitions in the message system, the number of service instances, and the total number of threads in the thread pools. It is worth noting that under the above processing model, increasing or decreasing the relevant configuration numbers does not affect the timing, meaning that timing processing for a single device can always be guaranteed. Generally, the number of service instances should match the number of message system partitions to ensure that each partition can be consumed in parallel by its corresponding instance, achieving maximum efficiency.

[0093] The above technical solution establishes multiple threads and instances in the server for message processing to improve data processing efficiency. By establishing a mapping relationship between consumer threads and data source identifiers, messages from the same data source are processed in the same thread. The sequential nature of thread message processing is used to ensure the order of message processing results and avoid disorder in the processing results.

[0094] Figure 9 This is a schematic diagram illustrating a multi-instance, multi-threaded overall consumption pattern according to an exemplary embodiment, such as... Figure 9 As shown, the IoT message processing platform is equipped with an overall message timing processing model. Different instances of the same service share the same group ID and operate in a shared consumption mode within the group, mapping messages to different partitions. The thread pool within the same instance is customized, binding an attribute list to each thread to store the message's data source ID. In this case, timing guarantees are required for each device's status messages; therefore, the device ID can be used as the message's data source ID, establishing a fixed mapping relationship between the device ID and the thread. When a message flows to the next service, the fixed partitioning characteristic of messages from the same device should be guaranteed. Specifically, the device ID is specified as the partitioning parameter when sending the message, thereby achieving time-sequential message processing. For example, based on the above multi-instance, multi-threaded overall consumption model, this embodiment also provides a message data processing method including the following steps.

[0095] 1. Messages from the same data source are assigned to the same partition of the messaging system;

[0096] 2. Messages from the same partition are mapped to fixed consumer groups in different services;

[0097] 3. Messages from the same data source are mapped to the same thread for processing.

[0098] Through the above approach, the message system application processing mode supports both high-concurrency multi-threaded processing and strict message timing processing. This mode guarantees the timing of message processing for the same data source, which is beneficial for applications in industrial control and other high-security control fields. By fully utilizing the conventional characteristics of the message system, such as data processing rules, partitioning, and group consumption strategies, and through customized application of thread pools, timing processing of messages from the same data source is achieved. Specifically, the message system client is divided into a producer and a consumer, corresponding to message sending and receiving respectively. The producer is the manufacturer and possesses a unique device ID, while the consumer is the user, possessing two attributes: a group ID and their own ID. The manufacturer is responsible for producing and sending message data. When sending a message, the producer specifies its own ID or other unique attributes as the partitioning basis so that the message system maps the message data to a fixed partition. Message queues within the same partition in the consumption system are guaranteed to be sequential. Consumers identify their consumption group by specifying a group ID and also specify their own unique ID. When multiple consumer group members subscribe to messages simultaneously, a mapping relationship is established between the member ID and the message system partition. Once the mapping is determined, it will not change randomly. In this way, each consumer within the group receives and processes message data from different partitions without interference, eliminating the message timing dependency between different consumers. Referring to the previous processing method, to ensure that messages from the same data source enter the same thread, since the previous processing has already stored messages from the same data source in the same partition, now we only need to put messages from the same partition into the same thread for processing. Specifically, a custom thread pool management class is needed, binding a list to each thread. The list stores message data source IDs. When a new message is received, the thread pool first checks if the data source ID of this message exists in each thread's bound list. If found, it is placed into that thread for processing; otherwise, a thread is randomly selected from the thread pool, and the message data source ID is stored in its list. This ensures that messages from each data source enter the same thread for processing, and the processing within the same thread is serial, guaranteeing timing.

[0099] Figure 10 This is a block diagram illustrating a message data processing apparatus according to an exemplary embodiment, such as... Figure 10 As shown, the device 100 includes: a first determining module 110, a second determining module 120, a first generating module 130, a third determining module 140, and a second generating module 150.

[0100] The first determining module 110 is used to determine the data source identifier of the message data in response to receiving message data sent by the message queue.

[0101] The second determining module 120 is used to determine the first consumer thread from the first preset thread pool of the primary preset service according to the data source identifier and the preset flow order of message data. The first preset thread pool includes multiple consumer threads.

[0102] The first generation module 130 is used to process message data through the first consumer thread to generate the first service processing result.

[0103] The third determining module 140 is used to determine the second consumer thread from the second preset thread pool of the secondary preset service according to the preset flow order and the data source identifier. The second preset thread pool includes multiple consumer threads.

[0104] The second generation module 150 is used to process the first service processing result through the second consumption thread to generate the second service processing result.

[0105] Optionally, the second determining module 120 is used for:

[0106] Obtain the first preset data source identifier table of the primary preset service. The first preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers.

[0107] Based on the first preset data source identifier table, determine the first consumer thread corresponding to the data source identifier.

[0108] Optionally, the third determining module 140 is used for:

[0109] Obtain the second preset data source identifier table of the secondary preset service. The second preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers.

[0110] Based on the second preset data source identifier table, determine the second consumer thread corresponding to the data source identifier.

[0111] Optionally, the second determining module 120 is used for:

[0112] Retrieve the partition identifier of the message data in the message queue.

[0113] Based on the preset flow order, determine the primary preset service for initial consumption processing of message data.

[0114] Based on the partition identifier, determine the first consumer member corresponding to the partition identifier from among the multiple consumer members of the primary preset service.

[0115] Based on the data source identifier, the first consumer thread is determined from the first preset thread pool of the first consumer group member.

[0116] Optionally, the device 100 includes a third generation module, which is used for:

[0117] Based on the terminal's data processing capabilities, the primary preset services are divided into consumer groups to generate multiple consumer groups.

[0118] Multiple consumer group members are divided into threads according to preset partitioning rules to generate multiple preset thread pools.

[0119] Optionally, the device 100 further includes an execution module, which is used for:

[0120] Based on the preset flow order and the result of the second service processing, the result of the target service processing is generated.

[0121] Send the target service processing result to the client.

[0122] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0123] Figure 11 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, electronic device 1100 may be provided as a server. (Refer to...) Figure 11 The electronic device 1100 includes a processor 1122, which may be one or more, and a memory 1132 for storing computer programs executable by the processor 1122. The computer program stored in the memory 1132 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 1122 may be configured to execute the computer program to perform the aforementioned message data processing method.

[0124] Additionally, the electronic device 1100 may also include a power supply component 1126 and a communication component 1150. The power supply component 1126 can be configured to perform power management of the electronic device 1100, and the communication component 1150 can be configured to enable communication of the electronic device 1100, such as wired or wireless communication. Furthermore, the electronic device 1100 may also include an input / output (I / O) interface 1158. The electronic device 1100 can operate on an operating system stored in the memory 1132.

[0125] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the message data processing method described above. For example, the non-transitory computer-readable storage medium may be the memory 1132 including the program instructions described above, which may be executed by the processor 1122 of the electronic device 1100 to complete the message data processing method described above.

[0126] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described message data processing method when executed by the programmable device.

[0127] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0128] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction.

[0129] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method for processing message data, characterized in that, include: In response to receiving message data sent from the message queue, determine the data source identifier of the message data; Based on the data source identifier and the preset flow order of the message data, a first consumer thread is determined from the first preset thread pool of the primary preset service, and the first preset thread pool includes multiple consumer threads; The message data is processed by the first consumer thread to generate a first service processing result; Based on the preset flow order and the data source identifier, a second consumer thread is determined from the second preset thread pool of the secondary preset service. The second preset thread pool includes multiple consumer threads. The second consumer thread processes the first service processing result to generate the second service processing result. The step of determining the first consumer thread from the first preset thread pool of the primary preset service based on the data source identifier and the preset flow order of the message data includes: Obtain the partition identifier of the message data in the message queue; Based on the preset flow order, the primary preset service for initial consumption processing of the message data is determined; Based on the partition identifier, determine the first consumer member corresponding to the partition identifier from among the multiple consumer members of the primary preset service; Based on the data source identifier, the first consumer thread is determined from the first preset thread pool of the first consumer group member.

2. The processing method according to claim 1, characterized in that, The step of determining the first consumer thread from the first preset thread pool of the primary preset service includes: Obtain the first preset data source identifier table of the primary preset service, wherein the first preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers; Based on the first preset data source identifier table, determine the first consumer thread corresponding to the data source identifier.

3. The processing method according to claim 2, characterized in that, The step of determining the second consumer thread from the second preset thread pool of the secondary preset service includes: Obtain the second preset data source identifier table of the secondary preset service, wherein the second preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers; Based on the second preset data source identifier table, determine the second consumer thread corresponding to the data source identifier.

4. The processing method according to claim 1, characterized in that, The method further includes: Based on the terminal's data processing capabilities, the primary preset service is divided into consumer groups to generate the multiple consumer groups; The multiple consumer group members are divided into threads according to a preset partitioning rule to generate multiple preset thread pools.

5. The processing method according to claim 1, characterized in that, The method further includes: Based on the preset flow order and the second service processing result, a target service processing result is generated; The result of the target service processing is sent to the client.

6. A message data processing apparatus, characterized in that, include: The first determining module is used to determine the data source identifier of the message data in response to receiving message data sent by the message queue. The second determining module is used to determine a first consumer thread from a first preset thread pool of the primary preset service based on the data source identifier and the preset flow order of the message data. The first preset thread pool includes multiple consumer threads. The first generation module is used to process the message data through the first consumption thread to generate a first service processing result; The third determining module is used to determine a second consumer thread from the second preset thread pool of the secondary preset service according to the preset flow order and the data source identifier. The second preset thread pool includes multiple consumer threads. The second generation module is used to process the first service processing result through the second consumption thread to generate a second service processing result; The second determining module is used for: Obtain the partition identifier of the message data in the message queue; Based on the preset flow order, the primary preset service for initial consumption processing of the message data is determined; Based on the partition identifier, determine the first consumer member corresponding to the partition identifier from among the multiple consumer members of the primary preset service; Based on the data source identifier, the first consumer thread is determined from the first preset thread pool of the first consumer group member.

7. The apparatus according to claim 6, characterized in that, The second determining module is used for: Obtain the first preset data source identifier table of the primary preset service, wherein the first preset data source identifier table includes a one-to-one correspondence between multiple consumer threads and multiple data source identifiers; Based on the first preset data source identifier table, determine the first consumer thread corresponding to the data source identifier.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-5.

9. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-5.

Images