A multi-threaded data consumption method and apparatus

Abstract

This invention discloses a multi-threaded data consumption method and apparatus. The method includes: receiving subscription messages from users on a subscription messaging system; when it is determined that a first data packet containing the subscription message does not exist in memory, recording the offset of the subscription message in the original sequence and placing the subscription message into a thread pool for processing; when the subscription message processing is normal, storing the second data packet to which the subscription message belongs, and recording the offset of the subscription message in the processing sequence; comparing the processing sequence with the original sequence; when the comparison between the processing sequence and the original sequence is successful, submitting the offset to the subscription messaging system and sending a feedback message to the producer of the subscription message. This prevents both duplicate message consumption and message loss, ensuring the stable operation of the data exchange application. Simultaneously, it improves the degree of automation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multi-threaded data consumption method and apparatus. Background Technology

[0002] In a distributed architecture, data exchange, as the core node for data or file transfer, is a crucial point in data flow and a key element of the entire data flow architecture. It demands high levels of stability, security, and transmission efficiency. Therefore, improving the stability of data exchange or data transmission is a technical problem that those skilled in the art need to solve. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a multi-threaded data consumption method, apparatus and readable medium to improve the stability of data exchange or data transmission.

[0004] One aspect of this invention provides a multi-threaded data consumption method, the method comprising:

[0005] Receive subscription messages from users on the subscription messaging system;

[0006] When it is determined that there is no first data packet containing the subscription message in memory, the offset of the subscription message is recorded in the original sequence, and the subscription message is put into the thread pool for processing;

[0007] When the subscription message is processed normally, the second data packet to which the subscription message belongs is stored, and the offset of the subscription message is recorded in the processing sequence;

[0008] The processed sequence is compared with the original sequence;

[0009] When the processed sequence is successfully compared with the original sequence, the offset is submitted to the subscription message system, and a feedback message is sent to the producer of the subscription message.

[0010] Furthermore, receiving subscription messages from users on the subscription messaging system includes:

[0011] Read the configuration file of the first extension type to obtain the consumer subscription configuration of the subscription messaging system, where the offset commit configuration is set to manual commit;

[0012] Start the consumer program for the subscription messaging system;

[0013] Query the memory cache type configured in the database;

[0014] Call the memory read interface based on the memory cache type;

[0015] When the memory cache type is the first cache type, the configuration file of the first extended type is read through the read interface to connect to the database.

[0016] Furthermore, when it is determined that the first data packet containing the subscribed message does not exist in memory, the offset of the subscribed message is recorded in the original sequence, and the subscribed message is placed into the thread pool for processing, including:

[0017] When it is determined that the first data packet containing the subscription message does not exist in memory, the configuration file of the first extended type is read to obtain the number of processing queues for the subscription message;

[0018] Parse the subscribed messages and obtain the organization code of the data packets corresponding to the subscribed messages;

[0019] Obtain the unique numerical code based on the organization's internal code;

[0020] Based on the unique numeric code and the number of processing queues, obtain the remainder M of the processing queues;

[0021] Subscribed messages are placed into the Mth processing queue for processing.

[0022] Furthermore, the processed sequence is compared with the original sequence, including:

[0023] When the consumer program of the subscription messaging system is started, the program that processes the queue is executed;

[0024] Each time the processing queue is refreshed, the processing queue is sorted in ascending order.

[0025] The processed sequence is compared with the original sequence;

[0026] When the data at the index position and subsequent positions are equal, the last successfully matched value is used as the offset and incremented by 1 before being submitted to the subscription message system.

[0027] Furthermore, the methods also include:

[0028] When it is determined that the first data packet containing the subscribed message exists in memory, the offset is submitted to the subscription message system and a feedback message is sent to the producer.

[0029] Furthermore, the methods also include:

[0030] When an exception occurs during message processing, an exception log is generated. The exception log records the offset and packet information of the subscribed message and sends a feedback message to the producer.

[0031] Furthermore, the methods also include:

[0032] When the processing time of a subscribed message exceeds the preset time, an exception log is generated. The exception log is used to record the offset and packet information of the subscribed message and to send a feedback message to the producer.

[0033] To achieve the above objectives, one aspect of the present invention provides a multi-threaded data consumption apparatus, which includes: a receiving module, a processing module, a recording module, a comparison module, and a submission module, wherein...

[0034] The receiving module is used to receive subscription messages from users on the subscription messaging system;

[0035] The processing module is used to record the offset of the subscription message into the original sequence and put the subscription message into the thread pool for processing when it is determined that there is no first data packet containing the subscription message in memory;

[0036] The recording module is used to store the second data packet to which the subscription message belongs when the subscription message is processed normally, and to record the offset of the subscription message into the processing sequence;

[0037] The comparison module is used to compare the processed sequence with the original sequence;

[0038] The submission module is used to submit the offset to the subscription message system and send a feedback message to the producer of the subscription message when the processed sequence is successfully compared with the original sequence.

[0039] To achieve the above objectives, one aspect of the present invention provides a computer device, comprising:

[0040] At least one processor; and

[0041] A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method described above.

[0042] To achieve the above objectives, one aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0043] The present invention has at least the following beneficial technical effects:

[0044] This invention, in its embodiment, receives subscription messages from users on a subscription messaging system. When it is determined that a first data packet containing the subscription message does not exist in memory, the offset of the subscription message is recorded in the original sequence, and the subscription message is placed in a thread pool for processing. When the subscription message processing is successful, the second data packet to which the subscription message belongs is stored, and the offset of the subscription message is recorded in the processing sequence. The processing sequence is compared with the original sequence. When the comparison between the processing sequence and the original sequence is successful, the offset is submitted to the subscription messaging system, and a feedback message is sent to the producer of the subscription message. This approach prevents both duplicate message consumption and message loss, ensuring the stable operation of the data exchange application. Simultaneously, it improves the level of automation. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of an embodiment of a multi-threaded data consumption method provided by the present invention;

[0047] Figure 2 This is a schematic diagram of an embodiment of a multi-threaded data consumption device provided by the present invention;

[0048] Figure 3 A schematic diagram of an embodiment of the computer device provided by the present invention;

[0049] Figure 4 A schematic diagram illustrating an embodiment of the computer-readable storage medium provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0051] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0052] Based on the above objectives, the first aspect of the present invention provides an embodiment of a multi-threaded data consumption method. Figure 1 The diagram shown is a schematic representation of an embodiment of a multi-threaded data consumption method provided by the present invention. Figure 1 As shown, a multi-threaded data consumption method according to an embodiment of the present invention is applied to a server, and the method may include:

[0053] S101, Receive subscription messages from users on the subscription messaging system.

[0054] The server's memory caching component internally supports both Redis and Caffeine storage modes, providing a unified put / get interface externally. The memory cache is configured via the database system parameter table XT_XTYXCS. Caffeine storage mode is used at acquisition points (with lower server configurations) to reduce resource consumption and system complexity, while Redis storage mode is used in directly managed databases, branch offices, and headquarters (with relatively higher server configurations) to increase storage capacity and persistence. The memory cache supports a data expiration mechanism to prevent data from overflowing memory. Redis reads the connection information of the Redis in-memory database through the redis.properties configuration file.

[0055] Specifically, S101 can be implemented as follows: 1. Read the configuration file with the extension type "properties" to obtain the consumer subscription configuration of the subscription messaging system, where the offset commit configuration is set to manual. 2. Start the consumer program of the subscription messaging system, subscribe to the subscription messages, and record the offset of the subscription messages. 3. Query the configured memory cache type in the database, where the memory cache type can be either Redis or Caffeine. 4. Call the memory read interface according to the memory cache type. 5. When the memory cache type is Redis, read the properties configuration file through the read interface to connect to the database.

[0056] The consumer program initializes and starts the message queue receiver when the application starts. The message queue receiver is a timer that periodically restarts the message receiving thread; the timer restart interval is configured in the kafka_consumer.properties configuration file.

[0057] Within the message receiving thread, N thread pool queues are created. Each thread pool has a core thread size of 1, a maximum thread count of 1, and a bounded blocking queue of length X called an ArrayBlockingQueue. The core thread size, the number of thread pool queues N, and the work queue length X are all configured in the `kafka_consumer.properties` configuration file. When the organization is a purchasing point, N = 1 and X = 3; when the organization is a directly affiliated database, N = 20 and X = 5; when the organization is a branch office, N = 20 and X = 5; and when the organization is the head office, N = 30 and X = 10. The thread pool queues are used to process subscribed messages.

[0058] S102. When it is determined that there is no first data packet containing the subscription message in memory, the offset of the subscription message is recorded in the original sequence, and the subscription message is put into the thread pool for processing.

[0059] After subscribing to a message, the topic, partition, and offset of the message are recorded. The message is then converted into a DataBlock object, and the identifier (e.g., Id) of the DataPackage is extracted from the block object. Based on the identifier, it is determined whether the first data packet containing the subscribed message exists in the memory cache.

[0060] When it is determined that there is no first data packet containing the subscription message in memory, S102 can be implemented as follows: Read the properties configuration file to obtain the number of processing queues for the subscription message. Parse the subscription message and obtain the organization code of the data packet corresponding to the subscription message. Obtain the unique numeric code based on the organization code. Obtain the remainder M of the processing queues based on the unique numeric code and the number of processing queues. Place the subscription message into the Mth processing queue for processing.

[0061] For example, if the first data packet containing the subscribed message does not exist in the memory cache, the offset of the subscribed message is first recorded in the original offset sequence of type List. The data packet's internal code is obtained, and the encapsulated method for obtaining a unique numeric code based on the internal code is called to retrieve the unique numeric code. The unique numeric code is modulo (%) the number of message processing queues, and the remainder M is taken. The subscribed message is then placed into the Mth queue for processing, with each task having a timeout of 3 minutes.

[0062] S103. When it is determined that the first data packet containing the subscription message exists in memory, the offset is submitted to the subscription message system and a feedback message is sent to the producer of the subscription message.

[0063] In other words, when it is determined that the first data packet containing the subscribed message exists in memory, the offset + 1 is submitted to the subscribed message system Kafka, a feedback message is sent to the producer of the subscribed message, and the process ends.

[0064] S104. When the subscription message is processed normally, the second data packet to which the subscription message belongs is stored, and the offset of the subscription message is recorded in the processing sequence.

[0065] After each subscribed message is processed, its processing status is checked. If the message is processed normally, the thread pool queue returns true, the offset processing sequence is recorded, and the data packet is stored in the memory cache. Simultaneously, the offset commit logic and message feedback logic are executed.

[0066] S105. When an exception occurs during message processing, an exception log is generated. The exception log is used to record the offset and packet information of the subscribed message and to send a feedback message to the producer.

[0067] Among them, message subscription processing exceptions can include processing timeouts and process exceptions.

[0068] S105 specifically states that when an exception occurs during message subscription processing, the thread pool queue returns false, and the exception handling logic is invoked. This exception handling logic can include: logging the exception to a log file, along with information such as the topic, partition, and offset of the subscribed message. Furthermore, the log folder is configured as an upload folder, and the logs are uploaded level by level to branch offices for easier subsequent investigation.

[0069] Additionally, when the processing time of a subscribed message exceeds a preset duration, an exception log is generated. This exception log records the offset and packet information of the subscribed message and sends a feedback message to the producer. For example, if the processing queue times out (e.g., exceeding 3 minutes), the offset and packet information of the subscribed message are recorded in the exception log, a feedback message is sent to the producer, and the message processing flow ends.

[0070] S106. Compare the processed sequence with the original sequence.

[0071] Specifically, S106 can be as follows: When the consumer program of the subscription message system starts, the program for processing the queue is executed. Each time the processing queue is refreshed, it is sorted in ascending order. The processed sequence is compared with the original sequence. When the data at the index position and subsequent positions are equal, the last successfully matched value is used as the offset and incremented by 1, then submitted to the subscription message system.

[0072] For example, the offset sequence is stored in the form of a List, where List[0] stores the offset value of the last submission; the offset handler is a timer that is called once per second and sorts the offset processing sequence in ascending order, comparing the offset processing sequence with the original offset sequence.

[0073] S107. When the processed sequence is successfully compared with the original sequence, submit the offset to the subscription message system and send a feedback message to the producer of the subscription message.

[0074] Following the example above, check if the data in List[1] and subsequent positions are equal. If they are equal, take the last successfully matched value as the offset and increment it by one before submitting. At the same time, delete the data before the two offset sequences and end the process.

[0075] In addition, if the index positions of the two sequences (original sequence and processing sequence) are still not equal 200 seconds after the offset handler is started, it indicates that an exception has occurred in the middle of the message processing. The current thread pool queue is terminated, the exception log and offset sequence are recorded in the log file, and a feedback message is sent to the producer to end the message processing process.

[0076] In this embodiment of the invention, a user's subscription message is received on a subscription messaging system. When it is determined that a first data packet containing the subscription message does not exist in memory, the offset of the subscription message is recorded in the original sequence, and the subscription message is placed in a thread pool for processing. When the subscription message processing is normal, the second data packet to which the subscription message belongs is stored, and the offset of the subscription message is recorded in the processing sequence. The processing sequence is compared with the original sequence. When the comparison between the processing sequence and the original sequence is successful, the offset is submitted to the subscription messaging system, and a feedback message is sent to the producer of the subscription message. This prevents both duplicate message consumption and message loss, ensuring the stable operation of the data exchange application. Simultaneously, it improves the degree of automation.

[0077] To achieve the above objectives, a second aspect of the present invention provides a multi-threaded data consumption device. Figure 2 The diagram shown is a schematic representation of an embodiment of the multi-threaded data consumption device provided by the present invention. Figure 2 As shown, the multi-threaded data consumption device 200 may include: a receiving module 201, a processing module 202, a recording module 203, a comparison module 204, and a submission module 205, wherein,

[0078] The receiving module 201 is used to receive subscription messages from users on the subscription messaging system;

[0079] The processing module 202 is used to record the offset of the subscription message into the original sequence and put the subscription message into the thread pool for processing when it is determined that there is no first data packet containing the subscription message in the memory;

[0080] The recording module 203 is used to store the second data packet to which the subscription message belongs when the subscription message is processed normally, and to record the offset of the subscription message into the processing sequence;

[0081] The comparison module 204 is used to compare the processed sequence with the original sequence;

[0082] The submission module 205 is used to submit the offset to the subscription message system and send a feedback message to the producer of the subscription message when the processed sequence is successfully compared with the original sequence.

[0083] The present invention also provides a computer device. Figure 3 The diagram shown is a schematic representation of an embodiment of the computer device provided by the present invention. Figure 3 As shown, the computer device of this embodiment includes: at least one processor 021; and a memory 022, the memory 022 storing computer instructions 023 that can be executed on the processor 021, the instructions 023 implementing the method described above when executed by the processor 021.

[0084] The present invention also provides a computer-readable storage medium. Figure 4 The diagram illustrates an embodiment of the computer-readable storage medium provided by the present invention. Figure 4 As shown, computer-readable storage medium 031 stores a computer program 032 that, when executed by a processor, performs the methods described above.

[0085] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for the document alert method can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0086] Furthermore, the method disclosed in the embodiments of the present invention can also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. When the computer program is executed by the processor, it performs the functions defined in the method disclosed in the embodiments of the present invention.

[0087] Furthermore, the above-described method steps and system units can also be implemented using a controller and a computer-readable storage medium for storing a computer program that enables the controller to perform the functions of the above-described steps or units.

[0088] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0089] In one or more exemplary designs, functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on or transmitted via a computer-readable medium. Computer-readable media include computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. Storage media may be any available medium accessible to a general-purpose or special-purpose computer. By way of example, and not limitation, computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that may be used to carry or store the required program code in the form of instructions or data structures and is accessible to a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. Furthermore, any connection may be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the aforementioned coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are all included in the definition of media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0090] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0091] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0092] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0093] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0094] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A multi-threaded data consumption method, characterized in that, The method includes: Receiving subscription messages from users on a subscription messaging system includes: reading a configuration file of a first extension type to obtain the consumer subscription configuration of the subscription messaging system, wherein the offset commit configuration is set to manual commit; When it is determined that a first data packet containing the subscription message does not exist in memory, the offset of the subscription message is recorded in the original sequence, and the subscription message is placed into a thread pool for processing, which includes: When it is determined that there is no first data packet containing the subscription message in memory, the configuration file of the first extended type is read to obtain the number of processing queues for the subscription message; Parse the subscription message and obtain the organization code of the data packet corresponding to the subscription message; Based on the organization's internal code, obtain a unique numerical code; Based on the unique digital code and the number of processing queues, the remainder M of the processing queues is obtained, wherein the unique digital code is modulo the number of processing queues, and the remainder M is obtained. The subscribed message is placed into the Mth processing queue for processing; When the subscription message is processed normally, the second data packet to which the subscription message belongs is stored, and the offset of the subscription message is recorded in the processing sequence; The processed sequence is compared with the original sequence; When the processed sequence is successfully compared with the original sequence, the offset is submitted to the subscription message system, and a feedback message is sent to the producer of the subscription message.

2. The method according to claim 1, characterized in that, Receiving user subscription messages on the subscription messaging system includes: Start the consumer program of the subscription messaging system; Query the memory cache type configured in the database; The memory read interface is invoked according to the memory cache type; When the memory cache type is the first cache type, the configuration file of the first extended type is read through the read interface to connect to the database.

3. The method according to claim 1, characterized in that, The step of comparing the processed sequence with the original sequence includes: When the consumer program of the subscription messaging system is started, the program for the processing queue is executed; Each time the processing queue is refreshed, the processing queue is sorted in ascending order. The processed sequence is compared with the original sequence; When the data at the index position and subsequent positions are equal, the last successfully matched value is used as the offset and incremented by 1 before being submitted to the subscription message system.

4. The method according to claim 1, characterized in that, Also includes: When it is determined that a first data packet containing the subscribed message exists in memory, the offset is submitted to the subscription message system, and a feedback message is sent to the producer.

5. The method according to claim 1, characterized in that, Also includes: When the subscription message processing encounters an error, an error log is generated. The error log is used to record the offset and packet information of the subscription message and to send a feedback message to the producer.

6. The method according to claim 1, characterized in that, Also includes: When the processing time of the subscription message exceeds a preset time, an exception log is generated. The exception log is used to record the offset and packet information of the subscription message and to send a feedback message to the producer.

7. A multi-threaded data consumption device, characterized in that, include: The module comprises a receiving module, a processing module, a recording module, a comparison module, and a submission module. The receiving module is used to receive subscription messages from users on the subscription messaging system; The processing module is used to record the offset of the subscription message into the original sequence and put the subscription message into the thread pool for processing when it is determined that there is no first data packet containing the subscription message in the memory; The recording module is used to store the second data packet to which the subscription message belongs when the subscription message is processed normally, and the offset of the subscription message is recorded in the processing sequence; The comparison module is used to compare the processed sequence with the original sequence; The submission module is used to submit the offset to the subscription message system and send a feedback message to the producer of the subscription message when the processed sequence is successfully compared with the original sequence. The receiving module is also used to read the configuration file of the first extended type to obtain the consumer subscription configuration of the subscription messaging system, wherein the offset commit configuration is manual commit; The processing module is further configured to: when it is determined that there is no first data packet containing the subscription message in memory, read the configuration file of the first extended type to obtain the number of processing queues for the subscription message; Parse the subscription message and obtain the organization code of the data packet corresponding to the subscription message; Based on the organization's internal code, obtain a unique numerical code; Based on the unique digital code and the number of processing queues, the remainder M of the processing queues is obtained, wherein the unique digital code is modulo the number of processing queues, and the remainder M is obtained. The subscribed message is placed into the Mth processing queue for processing.

8. A computer device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-6.

Images