Hotspot data processing method and system, electronic equipment and storage medium
By identifying and sorting hot messages in a memory queue, the problem of database connection exhaustion in high-concurrency hot message data processing is solved, enabling ordered serial processing of hot messages and improving processing efficiency and success rate.
Patent Information
- Application Number
- CN202511729512.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-24
- Publication Date
- 2026-02-13
AI Technical Summary
Existing technologies are prone to exhausting database connections when handling high-concurrency hot data, affecting the processing of other objects, resulting in a high failure rate for concurrent processing and an inability to effectively handle messages from hot objects.
Hot messages are identified and stored in a memory queue. High-concurrency hot messages are sorted into serial processing through the memory queue. Forwarding messages for target objects are generated and their states are updated. The memory queue is used to achieve ordered processing of hot messages.
By deploying memory queues in a distributed manner, the limitations of local databases and in-memory databases are avoided, ensuring the orderly processing of hot messages, reducing the failure rate, and improving message processing efficiency.
Smart Images
Figure CN121523929A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method and system for processing hot data, an electronic device, and a storage medium. Background Technology
[0002] Distributed architecture, through concurrent processing of multiple messages, has become the mainstream technology choice in the industry. The real-time data bus, as the data interaction hub in a distributed environment, bears the core responsibility of real-time data transmission and collaboration between heterogeneous systems. However, when a large number of messages involve the same hot object within a short period, high concurrency can occur. This can lead to only a portion of the messages being locked, while the majority maintain database connections and wait for locks until the wait times out, resulting in errors and preventing transactions. It may even exhaust database connections, preventing subsequent messages from obtaining database connections, thus affecting the processing of other objects.
[0003] Currently, this is mainly addressed through local databases or distributed locks. The local database approach utilizes the database's built-in queuing mechanism to implement object message queuing logic. Essentially, it leverages the database's own locking mechanism to convert parallel processing into serial processing. The distributed lock approach, on the other hand, uses in-memory database list or stream data structures to implement distributed locks. This escalates lock-contention errors from the database layer to the application layer, thus preventing lock waiting and excessive database connection consumption issues when handling high-concurrency, hot data.
[0004] However, the first method relies on database locking mechanisms, which leads to prolonged lock holding time, increased database resource consumption, and message blocking at the database level. In extreme cases, database connections may be exhausted, making it impossible to effectively process messages for not only hot objects but also messages for other objects. As for the second method, the in-memory database, as a centralized component, easily becomes a bottleneck in high-concurrency scenarios. Therefore, although it can avoid affecting the processing of non-hot objects, when multiple concurrent messages for the same hot object are processed, widespread transaction errors will still occur, meaning a large number of message processing failures will still occur. Summary of the Invention
[0005] In view of the shortcomings of the prior art, this application provides a method and system for processing hot data, an electronic device, and a storage medium to solve the problem that the prior art cannot effectively process each message and that a large number of message processing failures will occur.
[0006] To achieve the above objectives, this application provides the following technical solution:
[0007] The first aspect of this application provides a method for processing hotspot data, including:
[0008] Multiple current messages are consumed from the real-time data bus, and each hot message in each current message is identified and added to the corresponding hot message queue in the memory queue; wherein, the memory queue includes an object set and a message queue set; the object set records each target object and its status information; the message queue set includes the hot message queues of each target object;
[0009] The system scans the memory queue in real time to identify target objects whose status messages indicate an unlocked state. Based on the number of hot messages in the hot message queue of the target object, it generates a current forwarding message for the target object and updates the status message of the target object. The current forwarding message of the target object includes at least the identifier of the target object and the number of forwarded hot messages.
[0010] Based on the current forwarding message of the target object, read the hot messages of the target object sequentially from the hot message queue of the target object, and forward them to the non-consumer cluster for processing.
[0011] Optionally, in the above-described method for processing hot data, the step of consuming multiple current messages from the real-time data bus and identifying each hot message in each current message and adding it to the corresponding hot message queue in the memory queue includes:
[0012] Consume multiple current messages from the real-time data bus;
[0013] Hotspot identification is performed on each of the current messages in turn;
[0014] If the current message is identified as a hot message, determine whether the capacity of the current memory queue has exceeded the upper limit;
[0015] If the current capacity of the memory queue has not reached its limit, the primary key of the hot message is added to the hot message queue of the target object corresponding to the hot message in the memory queue, and the entire hot message is stored in the database.
[0016] Optionally, in the above-described method for processing hot data, adding the primary key of the hot message to the hot message queue of the target object corresponding to the hot message in the memory queue includes:
[0017] Determine whether a message item for the target object corresponding to the hot message exists in the memory queue;
[0018] If a message item for the target object corresponding to the hot message exists in the memory queue, then the primary key of the hot message is added to the hot message queue of the target object corresponding to it; wherein, the key-value pair of the target object is the unique identifier of the target object and its status message;
[0019] If the message item corresponding to the target object for the hot message does not exist in the memory queue, then the message item corresponding to the target object for the hot message is added to the object set, and the primary key of the hot message is added to the hot message queue of the target object corresponding to it.
[0020] Optionally, in the above-described method for processing hot data, the step of real-time scanning to identify target objects whose status messages in the memory queue indicate an unlocked state, generating a current forwarding message for the target object based on the number of hot messages in the target object's hot message queue, and updating the target object's status message includes:
[0021] Perform a cyclic scan of the set of objects in the memory queue;
[0022] When it is detected that the unlocking time of the target object has expired, the smaller value between the current maximum processing quantity of a single forwarding thread and the number of hot messages in the hot message queue of the target object is selected as the current forwarding quantity;
[0023] Calculate the end time for processing the current number of forwarded hot messages, and use the end time to update the unlock time of the target object;
[0024] The target object's identifier, the current forwarding count, and the end time are combined to form the target object's current forwarding message.
[0025] Optionally, in the above-described method for processing hot data, before selecting the smaller value between the current maximum processing quantity of a single forwarding thread and the number of hot messages in the hot message queue of the target object as the current forwarding quantity, the method further includes:
[0026] Determine whether the number of hot messages in the hot message queue of the target object is zero;
[0027] If the number of hot messages in the hot message queue of the target object is not zero, then the smaller value between the current maximum processing number of a single forwarding thread and the number of hot messages in the hot message queue of the target object is used as the current forwarding number.
[0028] If the number of hot messages in the hot message queue of the target object is zero, then the messages of the target object in the memory queue are deleted.
[0029] Optionally, in the above-described method for processing hot data, the step of sequentially reading the hot messages of the target object from the hot message queue of the target object based on the current forwarding messages of the target object, and forwarding them to the non-consumer cluster for processing, includes:
[0030] Based on the identifier of the target object in the current forwarding message of the target object, the primary keys of the hot messages are sequentially obtained from the hot message queue of the target object in the memory queue according to the sorting order. The hot messages are then retrieved from the database based on the primary keys of the hot messages and forwarded to the non-consumer cluster for processing until the number of forwarded hot messages reaches the current forwarding count in the current forwarding message of the target object.
[0031] Optionally, in the above-described method for processing hot data, after sequentially reading the hot messages of the target object from the hot message queue of the target object according to the current forwarding messages of the target object, and forwarding them to the non-consumer cluster for processing, the method further includes:
[0032] When the forwarding ends, if the current time is less than the unlock time recorded in the status message of the target object, the unlock time recorded in the status message of the target object will be updated to the current time.
[0033] A second aspect of this application provides a system for processing hotspot data, comprising:
[0034] A message consumption unit is used to consume multiple current messages from a real-time data bus and identify each hot message in each current message and add it to the corresponding hot message queue in a memory queue; wherein, the memory queue includes an object set and a message queue set; the object set records each target object and its status information; the message queue set includes the hot message queues of each target object;
[0035] The message scanning unit is used to scan the target object in the memory queue in real time to identify the target object whose status message indicates an unlocked state, generate the current forwarding message of the target object based on the number of hot messages in the hot message queue of the target object, and update the status message of the target object; wherein, the current forwarding message of the target object includes at least the identifier of the target object and the number of forwarded hot messages;
[0036] The message forwarding unit is used to read the hot messages of the target object sequentially from the hot message queue of the target object according to the current forwarding messages of the target object, and forward them to the non-consumer cluster for processing.
[0037] Optionally, in the aforementioned hot data processing system, the message consumption unit includes:
[0038] A consumption unit is used to consume multiple current messages from the real-time data bus.
[0039] The identification unit is used to sequentially identify hotspots for each of the current messages;
[0040] The capacity determination unit is used to determine whether the capacity of the current memory queue has exceeded the upper limit when the current message is identified as the hot message.
[0041] The message adding unit is used to add the primary key of the hot message to the hot message queue of the target object corresponding to the hot message in the memory queue when the capacity of the current memory queue has not reached the upper limit, and to store the entire hot message in the database.
[0042] Optionally, in the above-mentioned hot data processing system, the message adding unit includes:
[0043] The message judgment unit is used to determine whether there is a message item for the target object corresponding to the hot message in the memory queue;
[0044] The first adding unit is configured to add the primary key of the hot message to the hot message queue of the target object corresponding to the hot message when a message item of the target object corresponding to the hot message exists in the memory queue; wherein, the key-value pair of the target object is the unique identifier of the target object and its status message;
[0045] The second adding unit is used to add a message item of the target object corresponding to the hot message to the object set when there is no message item of the target object corresponding to the hot message in the memory queue, and add the primary key of the hot message to the hot message queue of the target object corresponding to it.
[0046] Optionally, in the aforementioned hotspot data processing system, the message scanning unit includes:
[0047] A cyclic scanning unit is used to cyclically scan the set of objects in the memory queue;
[0048] The quantity determination unit is used to select the smaller value between the current maximum processing quantity of a single forwarding thread and the number of hot messages in the hot message queue of the target object as the current forwarding quantity when it is detected that the unlocking time of the target object has been exceeded.
[0049] A time calculation unit is used to calculate the end time of processing the current number of forwarded hot messages, and to update the unlock time of the target object using the end time;
[0050] The component unit is used to assemble the target object's identifier, the current forwarding count, and the end time into the target object's current forwarding message.
[0051] Optionally, the above-mentioned hotspot data processing system also includes:
[0052] A quantity determination unit is used to determine whether the number of hot messages in the hot message queue of the target object is zero;
[0053] If the number of hot messages in the hot message queue of the target object is not zero, the quantity determination unit executes the smaller value between the current maximum processing quantity of a single forwarding thread and the number of hot messages in the hot message queue of the target object as the current forwarding quantity.
[0054] The deletion unit is used to delete the messages of the target object from the memory queue when the number of hot messages in the hot message queue of the target object is zero.
[0055] Optionally, in the aforementioned hotspot data processing system, the message forwarding unit includes:
[0056] The sequential forwarding unit is used to obtain the primary key of the hot message from the hot message queue of the target object in the memory queue according to the identifier of the target object in the current forwarding message of the target object, and retrieve the hot message from the database according to the primary key of the hot message and forward it to the non-consumer cluster for processing, until the number of forwarded hot messages reaches the current forwarding number in the current forwarding message of the target object.
[0057] Optionally, the above-mentioned hotspot data processing system also includes:
[0058] The time update unit is used to update the unlock time recorded in the status message of the target object to the current time when the current forwarding ends, if the current time is less than the unlock time recorded in the status message of the target object.
[0059] A third aspect of this application provides an electronic device, comprising:
[0060] Memory and processor;
[0061] The memory is used to store programs;
[0062] The processor is used to execute the program, which, when executed, is specifically used to implement the hotspot data processing method as described in any of the above.
[0063] A fourth aspect of this application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the hotspot data processing method as described in any of the preceding claims.
[0064] This application provides a method for processing hot data. A memory queue is designed in local memory, allowing for distributed deployment within a distributed system. This enables linear expansion as the application grows, ensuring requirements are met. It is not limited by local or in-memory databases. During data transmission, multiple current messages are consumed from the real-time data bus, and hot messages within each current message are identified and added to their corresponding hot message queues in the memory queue. The memory queue includes an object set and a message queue set. The object set records each target object and its status information. The message queue set includes the hot message queues for each target object, thus sorting the hot messages for each hot object sequentially for easier subsequent serial processing. Non-hot messages do not require additional processing, allowing for continued concurrent messaging and avoiding impact on message processing efficiency. Then, the method scans the memory queue in real-time to identify target objects whose status messages indicate an unlocked state. Based on the number of hot messages in the target object's hot message queue, a current forwarding message for the target object is generated, and the target object's status message is updated. The current forwarding message for the target object includes at least the target object's identifier and the number of forwarded hot messages. Finally, based on the current forwarding message of the target object, the hot messages of the target object are read sequentially from the hot message queue of the target object and forwarded to the non-consumer cluster for processing. In this way, the concurrent processing of high-concurrency hot messages is converted into serial processing through the memory queue, so that they can be processed in an orderly manner and are not limited by the local database or the memory database, ensuring that each hot message can be processed in an orderly and effective manner. Attached Figure Description
[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0066] Figure 1 A flowchart illustrating a method for processing hotspot data provided in an embodiment of this application;
[0067] Figure 2A flowchart illustrating a method for consuming real-time data bus messages provided in an embodiment of this application;
[0068] Figure 3 A flowchart illustrating a method for adding the primary key of a hot message to a memory queue, as provided in an embodiment of this application;
[0069] Figure 4 A schematic diagram illustrating an example of a memory queue provided in an embodiment of this application;
[0070] Figure 5 A flowchart illustrating a method for scanning hotline messages provided in an embodiment of this application;
[0071] Figure 6 This application provides an embodiment of a hotspot data processing system.
[0072] Figure 7 This is a schematic diagram of the architecture of an electronic device provided in an embodiment of this application. Detailed Implementation
[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0074] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0075] This application provides a method for processing hotspot data, such as... Figure 1 As shown, it includes the following steps:
[0076] S101. Consume multiple current messages from the real-time data bus and identify each hot message in each current message and add it to the corresponding hot message queue in the memory queue.
[0077] In this embodiment, the memory queue is a temporary data storage structure based on local memory. It temporarily stores data in local memory during program execution, enabling efficient communication and data transfer between different modules, threads, or processes. Compared to disk storage, memory queues offer extremely fast read and write speeds, making them more suitable for task scheduling and other needs in high-concurrency scenarios. Correspondingly, because it operates in local memory, it is deployed in a distributed manner within a distributed system, thus allowing for linear scaling as the application expands to meet greater demands.
[0078] In this embodiment, the memory queue includes an object set and a message queue set. The object set records each target object and its state information. Optionally, it may only record the identifier of the target object with pending hot messages and the state information of that target object, thus avoiding excessive memory consumption.
[0079] The target object is the object that the real-time data bus message needs to operate on, such as an account or other data. The target object's status information is a message reflecting whether the target object can be operated on at present, which is equivalent to a message reflecting the target object's unlock status, such as the unlock time. Therefore, by using the object set, we can know each target object with pending messages and its status information.
[0080] The message queue collection includes hot message queues for each target object, which are queues of all pending hot messages for each target object in the object collection. Hot messages are those messages for target objects currently experiencing high concurrency.
[0081] Therefore, in order to effectively handle high-concurrency hot messages, that is, to effectively handle high-concurrency messages of the same target object, multiple current messages are consumed from the real-time data bus by the consumer thread, and each hot message in each current message is identified and then added to the corresponding hot message queue in the memory queue. In this way, the hot messages that originally needed to be processed concurrently are sorted in order through the memory queue, which facilitates subsequent serial processing.
[0082] Optionally, to ensure that the consuming thread can distinguish between hot and non-hot messages, an identifier field can be added when constructing messages for hot objects, depending on the application scenario. For example, in business scenarios related to the distribution and redemption of red envelopes, the corresponding messages are tagged to distinguish between hot and non-hot messages. Of course, other methods can also be used for identification, such as the number of messages for a target object within a currently set time period.
[0083] Optionally, in another embodiment of this application, one specific implementation of step S101 is as follows: Figure 2 As shown, it includes:
[0084] S201, Consume multiple current messages from the real-time data bus.
[0085] S202. Perform hotspot identification on each current message in turn to determine whether the current message is a hot message.
[0086] If a message is not a hot message, no additional processing is performed; it is processed according to the original logic. This allows for concurrent processing of non-hot messages, avoiding impact on message processing efficiency. If a message is identified as a hot message, it can be added to a memory object for sequential processing.
[0087] S203. Determine whether the current memory queue capacity has exceeded the upper limit.
[0088] Since the memory queue is located in memory, and memory capacity is limited, it is necessary to determine whether the current capacity of the memory queue has exceeded the upper limit. If the current capacity of the memory queue has not reached the upper limit, then step S204 is executed.
[0089] S204. Add the primary key of the hot message to the hot message queue of the target object corresponding to the hot message in the memory queue, and store the entire hot message in the database.
[0090] To reduce memory usage, in this embodiment, only the primary key of the hot messages is added to the hot message queue; that is, the hot message queue is a queue composed of the primary keys of each hot message. The entire hot message is stored in the database for subsequent forwarding.
[0091] Optionally, in another embodiment of this application, one specific implementation of step S204 is as follows: Figure 3 As shown, it includes:
[0092] S301. Determine whether there is a message item for the target object corresponding to a hot message in the memory queue.
[0093] The message item for the target object refers to the messages of the target object in the object collection, namely, the target object and its status messages. Optionally, the messages of the target object are stored in a key-value pair format, with the target object's identifier as the key and its status message as the value. For example, ... Figure 4 As shown, when the target object is an account and the status message is the unlock time, the unique identifier of the account and the unlock time are stored in the object collection as key-value pairs.
[0094] In this embodiment, to avoid consuming too much memory, only the target objects of the currently stored hot messages to be processed are recorded. Therefore, it is necessary to first determine whether there is a message item for the target object corresponding to the hot message in the memory queue. If there is a message item for the target object corresponding to the hot message in the memory queue, then step S302 is executed. If there is no message item for the target object corresponding to the hot message in the memory queue, then step 303 is executed.
[0095] S302. Add the primary key of the hot message to the hot message queue of its corresponding target object.
[0096] Among them, the key-value pairs of the target object are the unique identifier of the target object and its status message.
[0097] like Figure 4 As shown, the hot message queue is composed of the location identifier of the target object as the key and the queue composed of the primary key of each hot message as the value.
[0098] S303. Add the message item of the target object corresponding to the hot message to the object collection, and add the primary key of the hot message to the hot message queue of the corresponding target object.
[0099] S102. Scan the target objects in the memory queue in real time to identify those with unlocked status messages. Based on the number of hot messages in the hot message queue of the target object, generate the current forwarding message for the target object and update the status message of the target object.
[0100] The target object's current forwarded message includes at least the target object's identifier and the number of forwarded hot messages.
[0101] Specifically, after the consumer thread maintains the primary keys of hot messages in an in-memory queue, the scanning thread is responsible for packaging and distributing the messages for processing. The scanning thread will cyclically scan the set of accounts to be processed. If there are hot messages to be processed and their corresponding hot messages are in an unlocked state, they can be allocated. In order for the forwarding thread to forward messages of the target object and to determine the number of messages to forward based on the current situation, the current number of messages to be forwarded for the target object is provided to the forwarding thread for forwarding. Since messages are already being forwarded and no other messages can be forwarded, the status message of the target object needs to be updated to indicate that it is no longer in an unlocked state.
[0102] Optionally, in another embodiment of this application, the status message is the unlock time, and correspondingly, a specific implementation of step S102 is as follows: Figure 5 As shown, it includes:
[0103] S501, Perform a cyclic scan of the object collection in the memory queue.
[0104] S502. When it is detected that the unlocking time of the target object has been exceeded, select the smaller value between the current maximum processing quantity of a single forwarding thread and the number of hot messages in the hot message queue of the target object as the current forwarding quantity.
[0105] Specifically, the `unlockTime` of the scanned target object is compared with the current time `now` to determine the lock state of the target object. See Table 1 below for details:
[0106] Table 1
[0107]
[0108] When the target object is in an unlocked state, calculate the number of messages, n, that a single forwarding thread needs to process. Take the smaller of the length of the queue corresponding to the current account (i.e., the number of hot messages in the target object's hot message queue) and the maximum number of messages that a single forwarding thread can process. The calculation formula is n = Math.min(Queue[id].length(), N), where Queue[id] represents the queue with the key id stored in msgMap, and N is the current maximum number of messages that a single forwarding thread can process.
[0109] Optionally, in another embodiment of this application, to avoid data occupancy by target objects that do not have hot topics, the following is further performed before executing step S502:
[0110] Determine if the number of hot messages in the hot message queue of the target object is zero.
[0111] If the number of hot messages in the hot message queue of the target object is not zero, then step S502 is executed. If the number of hot messages in the hot message queue of the target object is zero, then the messages of the target object in the memory queue are deleted. Specifically, the messages of the target object and the hot message queue of the object set need to be deleted.
[0112] S503. Calculate the end time of processing the current number of forwarded hot messages, and use the end time to update the unlock time of the target object.
[0113] In order to unlock the target object in a timely manner after forwarding the current batch of trending messages, so as to forward subsequent messages, the end time of processing the current number of trending messages is calculated, and the unlock time of the target object is updated using the end time.
[0114] The end time for the current number of forwarded hot messages can be calculated by adding the current number of forwards and the estimated processing time for a single hot message to the current time. Specifically, it can be expressed as: unlockTime = now + n * interval. Here, now represents the current time; n is the number of messages a single forwarding thread needs to process, i.e., the current number of forwards; and interval is the estimated processing time for a single message, which needs to be estimated based on the actual application context.
[0115] S504. Combine the target object's identifier, current forwarding count, and end time to form the target object's current forwarding message.
[0116] S103. Based on the current forwarding messages of the target object, read the hot messages of the target object sequentially from the hot message queue of the target object, and forward them to the non-consumer cluster for processing.
[0117] Specifically, forwarding can be performed using forwarding threads in a forwarding thread pool. Each forwarding thread is responsible for forwarding n messages from one account. Its main parameters are: account ID, number of messages to process n, and unlock time (unlockTime). The forwarding thread will loop n times, sequentially retrieving the target object's hot messages from the head of the hot message queue and forwarding them, then waiting for a response. If the response returns within the estimated single transaction time interval, the waiting process ends.
[0118] Optionally, in another embodiment of this application, one specific implementation of step S103 includes:
[0119] Based on the target object's identifier in the target object's current forwarding message, the primary keys of the hot messages are retrieved sequentially from the target object's hot message queue in the memory queue, according to their order. Then, the hot messages are retrieved from the database based on their primary keys and forwarded to the non-consumer cluster for processing, until the number of forwarded hot messages reaches the current forwarding count in the target object's current forwarding message.
[0120] Optionally, in another embodiment of this application, in order to update the unlock time in a timely manner so that the next round of forwarding to the target object can begin as soon as possible, after executing step S103, the following is further executed:
[0121] When the forwarding ends, if the current time is less than the unlock time recorded in the target object's status message, then the unlock time recorded in the target object's status message will be updated to the current time.
[0122] Therefore, by updating the unlock time recorded in the target object's status message to the current time, its status is updated to the unlocked state in a timely manner, allowing the next round of forwarding to begin and shortening the idle time.
[0123] This application provides a method for processing hot data. A memory queue is designed in local memory, allowing for distributed deployment within a distributed system. This enables linear expansion as the application grows, ensuring requirements are met. It is not limited by local or in-memory databases. During data transmission, multiple current messages are consumed from the real-time data bus, and hot messages within each current message are identified and added to the corresponding hot message queue in the memory queue. The memory queue includes an object set and a message queue set. The object set records each target object and its status information. The message queue set includes the hot message queues for each target object, thus sequentially sorting the hot messages for each hot object for subsequent serial processing. Non-hot messages do not require additional processing, allowing for continued concurrent message processing and avoiding impact on message processing efficiency. Then, the method scans the memory queue in real-time to identify target objects whose status messages indicate an unlocked state. Based on the number of hot messages in the target object's hot message queue, a current forwarding message for the target object is generated, and the target object's status message is updated. The current forwarding message for the target object includes at least the target object's identifier and the number of forwarded hot messages. Finally, based on the current forwarding message of the target object, the hot messages of the target object are read sequentially from the hot message queue of the target object and forwarded to the non-consumer cluster for processing. In this way, the concurrent processing of high-concurrency hot messages is converted into serial processing through the memory queue, so that they can be processed in an orderly manner and are not limited by the local database or the memory database, ensuring that each hot message can be processed in an orderly and effective manner.
[0124] Another embodiment of this application provides a system for processing hotspot data, such as... Figure 6 As shown, it includes:
[0125] The message consumption unit 601 is used to consume multiple current messages from the real-time data bus and identify each hot message in each current message and add it to the corresponding hot message queue in the memory queue.
[0126] The memory queue includes a collection of objects and a collection of message queues. The object collection records the status information of each target object. The message queue collection includes the hot message queues for each target object.
[0127] The message scanning unit 602 is used to scan in real time to identify target objects in the memory queue whose status messages indicate an unlocked state. Based on the number of hot messages in the hot message queue of the target object, it generates a current forwarding message for the target object and updates the target object's status message. The current forwarding message for the target object includes at least the target object's identifier and the number of forwarded hot messages.
[0128] The message forwarding unit 603 is used to read the hot messages of the target object sequentially from the hot message queue of the target object according to the current forwarding messages of the target object, and forward them to the non-consumer cluster for processing.
[0129] Optionally, in another embodiment of the hot data processing system provided in this application, the message consumption unit includes:
[0130] The consumption unit is used to consume multiple current messages from the real-time data bus.
[0131] The identification unit is used to sequentially identify hotspots for each current message.
[0132] The capacity judgment unit is used to determine whether the capacity of the current memory queue has exceeded the upper limit when the current message is identified as a hot message.
[0133] The message addition unit is used to add the primary key of the hot message to the hot message queue of the target object corresponding to the hot message in the memory queue when the current memory queue capacity has not reached the upper limit, and to store the entire hot message in the database.
[0134] Optionally, in another embodiment of the hotspot data processing system provided in this application, the message adding unit includes:
[0135] The message judgment unit is used to determine whether there is a message item in the memory queue corresponding to the target object of the hot message.
[0136] The first addition unit is used to add the primary key of the hot message to the hot message queue of the corresponding target object when a message item for the target object corresponding to the hot message exists in the memory queue. Here, the key-value pair of the target object consists of the unique identifier of the target object and its status message.
[0137] The second adding unit is used to add a message item of the target object corresponding to the hot message to the object collection when there is no message item of the target object corresponding to the hot message in the memory queue, and add the primary key of the hot message to the hot message queue of the corresponding target object.
[0138] Optionally, in another embodiment of the hotspot data processing system provided in this application, the message scanning unit includes:
[0139] The circular scan unit is used to perform a circular scan of the collection of objects in the memory queue.
[0140] The quantity determination unit is used to select the smaller value between the current maximum processing quantity of a single forwarding thread and the number of hot messages in the hot message queue of the target object when the unlocking time of the target object has been exceeded, as the current forwarding quantity.
[0141] The time calculation unit is used to calculate the end time of processing the current number of forwarded hot messages, and to update the unlock time of the target object using the end time.
[0142] The component unit is used to assemble the target object's identifier, current forwarding count, and end time into the target object's current forwarding message.
[0143] Optionally, in another embodiment of the hotspot data processing system provided in this application, the system further includes:
[0144] The quantity judgment unit is used to determine whether the number of hot messages in the hot message queue of the target object is zero.
[0145] If the number of hot messages in the hot message queue of the target object is not zero, the quantity determination unit selects the smaller value between the current maximum processing quantity of a single forwarding thread and the number of hot messages in the hot message queue of the target object as the current forwarding quantity.
[0146] The deletion unit is used to delete messages of the target object from the memory queue when the number of hot messages in the hot message queue of the target object is zero.
[0147] Optionally, in another embodiment of the hotspot data processing system provided in this application, the message forwarding unit includes:
[0148] The sequential forwarding unit is used to retrieve the primary keys of hot messages from the hot message queue of the target object in the memory queue according to the identifier of the target object in the current forwarding message of the target object, and retrieve the hot messages from the database according to the primary keys of the hot messages and forward them to the non-consumer cluster for processing, until the number of forwarded hot messages reaches the current forwarding number in the current forwarding message of the target object.
[0149] Optionally, in another embodiment of the hotspot data processing system provided in this application, the system further includes:
[0150] The time update unit is used to update the unlock time recorded in the target object's status message to the current time when the current forwarding ends, if the current time is less than the unlock time recorded in the target object's status message.
[0151] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.
[0152] Another embodiment of this application provides an electronic device, such as... Figure 7 As shown, it includes:
[0153] Memory 701 and processor 702.
[0154] The memory 701 is used to store the program.
[0155] The processor 702 is used to execute the program stored in the memory 701. When the program is executed, it is specifically used to implement the hot data processing method provided in any of the above embodiments.
[0156] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the hot data processing method provided in any of the above embodiments.
[0157] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0158] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0159] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for processing hotspot data, characterized in that, include: Consume multiple current messages from the real-time data bus and identify each hot message in each current message and add it to the corresponding hot message queue in the memory queue; The memory queue includes an object set and a message queue set; the object set records each target object and its state information; the message queue set includes hot message queues for each target object. The system scans the memory queue in real time to identify target objects whose status messages indicate an unlocked state. Based on the number of hot messages in the hot message queue of the target object, it generates a current forwarding message for the target object and updates the status message of the target object. The current forwarding message of the target object includes at least the identifier of the target object and the number of forwarded hot messages. Based on the current forwarding message of the target object, read the hot messages of the target object sequentially from the hot message queue of the target object, and forward them to the non-consumer cluster for processing.
2. The method according to claim 1, characterized in that, The step of consuming multiple current messages from the real-time data bus and identifying and adding each hot message in each current message to the corresponding hot message queue in the memory queue includes: Consume multiple current messages from the real-time data bus; Hotspot identification is performed on each of the current messages in turn; If the current message is identified as a hot message, determine whether the capacity of the current memory queue has exceeded the upper limit; If the current capacity of the memory queue has not reached its limit, the primary key of the hot message is added to the hot message queue of the target object corresponding to the hot message in the memory queue, and the entire hot message is stored in the database.
3. The method according to claim 2, characterized in that, Adding the primary key of the hot message to the hot message queue of the target object corresponding to the hot message in the memory queue includes: Determine whether a message item for the target object corresponding to the hot message exists in the memory queue; If a message item for the target object corresponding to the hot message exists in the memory queue, then the primary key of the hot message is added to the hot message queue of the target object corresponding to it; wherein, the key-value pair of the target object is the unique identifier of the target object and its status message; If the message item corresponding to the target object for the hot message does not exist in the memory queue, then the message item corresponding to the target object for the hot message is added to the object set, and the primary key of the hot message is added to the hot message queue of the target object corresponding to it.
4. The method according to claim 1, characterized in that, The process of real-time scanning identifies target objects in the memory queue whose status messages indicate an unlocked state, generates a current forwarding message for the target object based on the number of hot messages in the target object's hot message queue, and updates the target object's status message, including: Perform a cyclic scan of the set of objects in the memory queue; When it is detected that the unlocking time of the target object has expired, the smaller value between the current maximum processing quantity of a single forwarding thread and the number of hot messages in the hot message queue of the target object is selected as the current forwarding quantity; Calculate the end time for processing the current number of forwarded hot messages, and use the end time to update the unlock time of the target object; The target object's identifier, the current forwarding count, and the end time are combined to form the target object's current forwarding message.
5. The method according to claim 4, characterized in that, Before selecting the smaller value between the current maximum processing capacity of a single forwarding thread and the number of hot messages in the hot message queue of the target object as the current forwarding capacity, the method further includes: Determine whether the number of hot messages in the hot message queue of the target object is zero; If the number of hot messages in the hot message queue of the target object is not zero, then the smaller value between the current maximum processing number of a single forwarding thread and the number of hot messages in the hot message queue of the target object is used as the current forwarding number. If the number of hot messages in the hot message queue of the target object is zero, then the messages of the target object in the memory queue are deleted.
6. The method according to claim 1, characterized in that, The step of sequentially reading the target object's hot messages from the target object's hot message queue based on the target object's current forwarding messages, and forwarding them to the non-consumer cluster for processing, includes: Based on the identifier of the target object in the current forwarding message of the target object, the primary keys of the hot messages are sequentially obtained from the hot message queue of the target object in the memory queue according to the sorting order. The hot messages are then retrieved from the database based on the primary keys of the hot messages and forwarded to the non-consumer cluster for processing until the number of forwarded hot messages reaches the current forwarding count in the current forwarding message of the target object.
7. The method according to claim 1, characterized in that, After sequentially reading the target object's hot messages from the target object's hot message queue based on the target object's current forwarding messages, and forwarding them to the non-consumer cluster for processing, the method further includes: When the forwarding ends, if the current time is less than the unlock time recorded in the status message of the target object, the unlock time recorded in the status message of the target object will be updated to the current time.
8. A system for processing hotspot data, characterized in that, include: A message consumption unit is used to consume multiple current messages from a real-time data bus and identify each hot message in each current message and add it to the corresponding hot message queue in a memory queue; wherein, the memory queue includes an object set and a message queue set; the object set records each target object and its status information; the message queue set includes the hot message queues of each target object; The message scanning unit is used to scan the target object in the memory queue in real time to identify the target object whose status message indicates an unlocked state, generate the current forwarding message of the target object based on the number of hot messages in the hot message queue of the target object, and update the status message of the target object; wherein, the current forwarding message of the target object includes at least the identifier of the target object and the number of forwarded hot messages; The message forwarding unit is used to read the hot messages of the target object sequentially from the hot message queue of the target object according to the current forwarding messages of the target object, and forward them to the non-consumer cluster for processing.
9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program, which, when executed, is specifically used to implement the hotspot data processing method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, is used to implement the hotspot data processing method as described in any one of claims 1 to 7.