A method and system for processing high-concurrency or large-volume business
By generating unique sequence numbers and processing state mappings, and splitting and multi-threading high-concurrency data request packets, the problem of business interruption and data loss caused by logical processing failures in the queue is solved, achieving efficient data recovery and accuracy.
Patent Information
- Application Number
- CN202111468596.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-12-03
AI Technical Summary
When processing high-concurrency data in queues, existing technologies may experience business interruptions and data inaccuracies or loss due to logical processing failures. There is a lack of effective recovery solutions, resulting in a significant workload.
By generating unique serial numbers and processing status mappings, data request packets are split into modules and associated units. Multi-threaded processing and caching mechanisms are used to achieve distributed processing of the logical processing queue, and logical processing is performed again in case of failure until success.
It avoids the impact of a single module failure on the overall business, reduces the workload of recovery, improves data accuracy and queue consumption capacity, and supports automatic or manual data recovery.
Smart Images

Figure CN114217985B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer data processing, and in particular to a method and system for processing high-concurrency or large-volume services. Background Art
[0002] Currently, queue middleware is widely used to process high-concurrency data. For example, the mainstream queues include rabbitMQ, kafak, RocketMQ, etc. However, a large amount of logical processing is performed in the queue. If one of the logical errors occurs, the queue or business will be interrupted, resulting in inaccurate and lost business data. There is no better backup solution for correcting inaccurate data and recovering lost data, and re-processing the logic will cause a lot of workload. Summary of the invention
[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for processing high-concurrency or large-volume services, which solves the problem that a large amount of workload will be caused once the logical processing of high-concurrency data fails.
[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:
[0005] A method for processing high-concurrency or large-volume services is provided, comprising the following steps:
[0006] S1, obtain data request packet;
[0007] S2. Generate a unique serial number for each data request packet, map the processing status of the data request packet to a value and store it in a cache, and build a data request packet processing queue;
[0008] S3. Based on the business type, split the request data in the current request data packet into several modules, generate a unique serial number for each module, and map the processing status of each module into a value and store it in the cache;
[0009] S4. For each module, determine whether it is a simple module. If so, directly add it to the logic processing queue and go to step S6; otherwise, go to step S5;
[0010] S5, split the current module into several associated units according to the logic type, and add the associated units belonging to the same module to the logic processing queue in sequence according to the logical order, and enter step S6;
[0011] S6, logically processing the logical processing queues belonging to the same request data packet according to the sequence of the module serial numbers, and modifying the processing status of the module that has successfully processed the logic in the current request data packet;
[0012] S7, determine whether there is a module with failed logic processing in the current request data packet, if yes, proceed to step S8; otherwise, delete all caches of the current data packet and proceed to step S10;
[0013] S8, re-logically process the module that failed in logical processing in the current request data packet;
[0014] S9, determine whether the logical processing of the current request data packet is successful, if so, delete all caches of the current data packet, and enter step S10; otherwise, use the same method as steps S4 to S8 to process the module that has not successfully processed the logical processing of the current request data packet until the processing is successful, and enter step S10;
[0015] S10, determine whether there are still data packets to be processed in the data request packet processing queue, if so, return to step S3; otherwise, end the processing.
[0016] Furthermore, the specific method of mapping the processing status of each module to a value in step S3 is:
[0017] The initial value of the processing status mapping of each module is set to 0, indicating that the processing is not successful; the processing status value of the module that is successfully processed is mapped to 1.
[0018] Furthermore, in step S6, different simple modules belonging to the same request data packet are synchronously processed in a multi-threaded manner.
[0019] Furthermore, in step S7, the specific method for determining whether there is a module whose logic processing fails in the current request data packet is as follows:
[0020] It is determined whether the processing status value of each module in the current request data packet is 1. If so, it means that there is no module with logical processing failure in the current request data packet; otherwise, there is a module with logical processing failure.
[0021] Furthermore, in step S9, the specific method of processing the module that has not successfully processed the current request data packet logic using the same method as steps S4 to S8 is:
[0022] According to the serial number of the module that has not been successfully processed by the logic of the current request data packet, the corresponding module is re-acquired from the several modules obtained in step S3, and then the re-acquired module is processed using the same method as steps S4 to S8.
[0023] Furthermore, in step S10, the specific method for determining whether there are any data packets to be processed in the data request packet processing queue is:
[0024] It is determined whether there is still a data request packet with the processing status mapped as a value in the cache. If so, it is determined that there is still a data packet to be processed; otherwise, it is determined that there is no data packet to be processed.
[0025] A system for processing high-concurrency or large-volume services is provided, comprising:
[0026] A data acquisition component, used to acquire a data request packet;
[0027] A data request packet processing queue building component is used to generate a unique serial number for each data request packet, and map the processing status of the data request packet to a value and store it in a cache to build a data request packet processing queue;
[0028] A splitting component is used to split the request data in the current request data packet into several modules based on the business type, generate a unique serial number for each module, and map the processing status of each module into a value and store it in the cache;
[0029] A logic processing queue building component is used to list the simple module of the current request data packet into the logic processing queue, split the current module into several associated units according to the logic type, and add the associated units belonging to the same module to the logic processing queue in sequence according to the logical order;
[0030] A logic preprocessing component, used to perform logic processing on the logic processing queues belonging to the same request data packet according to the sequence of the module serial numbers, and modify the processing status of the module that has successfully performed logic processing in the current request data packet;
[0031] The logic reprocessing component is used to delete the cache of the request data packets that have been successfully processed logically, and to re-process the modules in the current request data packets that have failed to be logically processed;
[0032] The logic final processing component is used to determine whether the processing results of the logic reprocessing component are all successful. If so, all caches of the current data packet are deleted; otherwise, the logic processing queue construction component, the logic preprocessing component and the logic reprocessing component are run again to process the modules that have not successfully processed the logic of the current request data packet until the processing is successful;
[0033] The end processing component is used to determine whether there are still data packets to be processed in the data request packet processing queue. If so, the splitting component, the logic processing queue construction component, the logic pre-processing component and the logic re-processing component are run again for processing; otherwise, the processing is terminated.
[0034] Furthermore, the logic pre-processing component includes a multi-thread processor for synchronously processing different simple modules belonging to the same request data packet in a multi-thread processing manner.
[0035] The beneficial effects of the present invention are:
[0036] 1. Distributed processing of business logic under high concurrency prevents the failure of one module from affecting the next module. The failed module can be re-executed to avoid business interruption, reduce recovery workload, and improve data accuracy.
[0037] 2. The present invention can also obtain the step of the entire business by obtaining the serial number, which is convenient for re-execution and problem troubleshooting, and automatic or manual data recovery.
[0038] 3. The present invention is applicable to a distributed cluster architecture, which can improve the consumption capacity of the queue and avoid congestion caused by untimely consumption of the queue. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 Schematic diagram of the process of this method. DETAILED DESCRIPTION
[0040] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0041] like Figure 1 As shown, the method for processing high concurrency or large-volume business includes the following steps:
[0042] S1, obtain data request packet;
[0043] S2. Generate a unique serial number for each data request packet, map the processing status of the data request packet to a value and store it in a cache, and build a data request packet processing queue;
[0044] S3. Based on the business type, split the request data in the current request data packet into several modules, generate a unique serial number for each module, and map the processing status of each module into a value and store it in the cache;
[0045] S4. For each module, determine whether it is a simple module. If so, directly add it to the logic processing queue and go to step S6; otherwise, go to step S5;
[0046] S5, split the current module into several associated units according to the logic type, and add the associated units belonging to the same module to the logic processing queue in sequence according to the logical order, and enter step S6;
[0047] S6, logically processing the logical processing queues belonging to the same request data packet according to the sequence of the module serial numbers, and modifying the processing status of the module that has successfully processed the logic in the current request data packet;
[0048] S7, determine whether there is a module with failed logic processing in the current request data packet, if yes, proceed to step S8; otherwise, delete all caches of the current data packet and proceed to step S10;
[0049] S8, re-logically process the module that failed in logical processing in the current request data packet;
[0050] S9, determine whether the logical processing of the current request data packet is successful, if so, delete all caches of the current data packet, and enter step S10; otherwise, use the same method as steps S4 to S8 to process the module that has not successfully processed the logical processing of the current request data packet until the processing is successful, and enter step S10;
[0051] S10, determine whether there are still data packets to be processed in the data request packet processing queue, if so, return to step S3; otherwise, end the processing.
[0052] The specific method of mapping the processing status of each module to a value in step S3 is: setting the initial value of the processing status mapping of each module to 0, indicating that the processing is unsuccessful; and mapping the processing status value of the module that has been processed successfully to 1.
[0053] In step S6, different simple modules belonging to the same request data packet are synchronously processed in a multi-threaded manner.
[0054] The specific method for determining whether there is a module with a logical processing failure in the current request data packet in step S7 is: determining whether the processing status value of each module in the current request data packet is 1, if so, it means that there is no module with a logical processing failure in the current request data packet; otherwise, there is a module with a logical processing failure.
[0055] The specific method for processing the module that has not successfully processed the current request data packet logic in step S9 using the same method as steps S4 to S8 is: according to the serial number of the module that has not successfully processed the current request data packet logic, re-acquire the corresponding module from the several modules obtained in step S3, and then use the same method as steps S4 to S8 to process the re-acquired module.
[0056] The specific method for determining whether there are still data packets to be processed in the data request packet processing queue in step S10 is: determining whether there are still data request packets with processing status mapped as values in the cache, if so, determining that there are still data packets to be processed; otherwise, determining that there are no data packets to be processed.
[0057] The system for processing high concurrency or large-volume business includes:
[0058] A data acquisition component, used to acquire a data request packet;
[0059] A data request packet processing queue building component is used to generate a unique serial number for each data request packet, and map the processing status of the data request packet to a value and store it in a cache to build a data request packet processing queue;
[0060] A splitting component is used to split the request data in the current request data packet into several modules based on the business type, generate a unique serial number for each module, and map the processing status of each module into a value and store it in the cache;
[0061] A logic processing queue building component is used to list the simple module of the current request data packet into the logic processing queue, split the current module into several associated units according to the logic type, and add the associated units belonging to the same module to the logic processing queue in sequence according to the logical order;
[0062] A logic preprocessing component, used to perform logic processing on the logic processing queues belonging to the same request data packet according to the sequence of the module serial numbers, and modify the processing status of the module that has successfully performed logic processing in the current request data packet;
[0063] The logic reprocessing component is used to delete the cache of the request data packets that have been successfully processed logically, and to re-process the modules in the current request data packets that have failed to be logically processed;
[0064] The logic final processing component is used to determine whether the processing results of the logic reprocessing component are all successful. If so, all caches of the current data packet are deleted; otherwise, the logic processing queue construction component, the logic preprocessing component and the logic reprocessing component are run again to process the modules that have not successfully processed the logic of the current request data packet until the processing is successful;
[0065] The end processing component is used to determine whether there are still data packets to be processed in the data request packet processing queue. If so, the splitting component, the logic processing queue construction component, the logic pre-processing component and the logic re-processing component are run again for processing; otherwise, the processing is terminated.
[0066] The logic pre-processing component includes a multi-thread processor for synchronously processing different simple modules belonging to the same request data packet in a multi-thread processing manner.
[0067] In one embodiment of the present invention, the unique serial number generated by each data request packet can adopt a universal unique identification code; each module can add a number based on the unique serial number of the data request packet in which it is located to form a corresponding unique serial number. It is well known to those skilled in the art that different data packets will contain different numbers of modules, so the number of modules will not be determined when splitting according to the business type. The "several modules" in the text cannot be specified in number. In theory, it can be 1 module or any number of modules. This statement does not cause ambiguity. Similarly, "several associated units" can theoretically be at least 2 associated units, or any number of associated units, and this statement will not cause ambiguity.
[0068] In summary, the present invention distributes the business logic under high concurrency, avoiding the failure of one module affecting the next module. The failed module can be re-executed again, avoiding business interruption, reducing the recovery workload, and improving data accuracy. It is also possible to obtain the step of the entire business by obtaining the sequence number, which is convenient for re-execution and problem troubleshooting, and automatic or manual data recovery. The present invention is applicable to a distributed cluster architecture, which can improve the consumption capacity of the queue and avoid blocking caused by untimely consumption of the queue.
Claims
1. A method for processing high-concurrency or large-volume services, characterized in that: The following steps are involved: S1, obtain data request packet; S2. Generate a unique serial number for each data request packet, map the processing status of the data request packet to a value and store it in a cache, and build a data request packet processing queue; S3. Based on the business type, split the request data in the current request data packet into several modules, generate a unique serial number for each module, and map the processing status of each module into a value and store it in the cache; S4. For each module, determine whether it is a simple module. If so, directly add it to the logic processing queue and go to step S6; otherwise, go to step S5; S5, split the current module into several associated units according to the logic type, and add the associated units belonging to the same module to the logic processing queue in sequence according to the logical order, and enter step S6; S6, logically processing the logical processing queues belonging to the same request data packet according to the sequence of the module serial numbers, and modifying the processing status of the module that has successfully processed the logic in the current request data packet; S7, determine whether there is a module whose logic processing fails in the current request data packet, if so, proceed to step S8; Otherwise, delete all caches of the current data packet and proceed to step S10; S8, re-logically process the module that failed in logical processing in the current request data packet; S9, determine whether the logical processing of the current request data packet is successful, if so, delete all caches of the current data packet, and enter step S10; otherwise, use the same method as steps S4 to S8 to process the module that has not successfully processed the logical processing of the current request data packet until the processing is successful, and enter step S10; S10, determine whether there are still data packets to be processed in the data request packet processing queue, if so, return to step S3; otherwise, end the processing.
2. The method for processing high-concurrency or large-volume services according to claim 1, characterized in that: The specific method of mapping the processing status of each module to a value in step S3 is: The initial value of the processing status mapping of each module is set to 0, indicating that the processing is not successful; the processing status value of the module that is successfully processed is mapped to 1.
3. The method for processing high-concurrency or large-volume services according to claim 1, characterized in that: In step S6, different simple modules belonging to the same request data packet are synchronously processed in a multi-threaded manner.
4. The method for processing high-concurrency or large-volume services according to claim 2, characterized in that: The specific method for determining whether there is a module whose logic processing fails in the current request data packet in step S7 is: It is determined whether the processing status value of each module in the current request data packet is 1. If so, it means that there is no module with logical processing failure in the current request data packet; otherwise, there is a module with logical processing failure.
5. The method for processing high-concurrency or large-volume services according to claim 1, characterized in that: The specific method of processing the module that failed to successfully process the current request data packet in step S9 using the same method as steps S4 to S8 is: According to the serial number of the module that has not been successfully processed by the logic of the current request data packet, the corresponding module is re-acquired from the several modules obtained in step S3, and then the re-acquired module is processed using the same method as steps S4 to S8.
6. The method for processing high-concurrency or large-volume services according to claim 1, characterized in that: The specific method for determining whether there are any data packets to be processed in the data request packet processing queue in step S10 is: It is determined whether there is still a data request packet with the processing status mapped as a value in the cache. If so, it is determined that there is still a data packet to be processed; otherwise, it is determined that there is no data packet to be processed.
7. A system for processing high-concurrency or large-volume services, characterized in that: include: A data acquisition component, used to acquire a data request packet; A data request packet processing queue building component is used to generate a unique serial number for each data request packet, and map the processing status of the data request packet to a value and store it in a cache to build a data request packet processing queue; A splitting component is used to split the request data in the current request data packet into several modules based on the business type, generate a unique serial number for each module, and map the processing status of each module into a value and store it in the cache; A logic processing queue building component is used to list the simple module of the current request data packet into the logic processing queue, split the current module into several associated units according to the logic type, and add the associated units belonging to the same module to the logic processing queue in sequence according to the logical order; A logic preprocessing component, used to perform logic processing on the logic processing queues belonging to the same request data packet according to the sequence of the module serial numbers, and modify the processing status of the module that has successfully performed logic processing in the current request data packet; The logic reprocessing component is used to delete the cache of the request data packets that have been successfully processed logically, and to re-process the modules in the current request data packets that have failed to be logically processed; The logic final processing component is used to determine whether the processing results of the logic reprocessing component are all successful. If so, all caches of the current data packet are deleted; Otherwise, the logic processing queue building component, the logic pre-processing component and the logic re-processing component are run again to process the module whose logic processing of the current request data packet has not been successful until the processing is successful; The end processing component is used to determine whether there are still data packets to be processed in the data request packet processing queue. If so, the splitting component, the logic processing queue construction component, the logic pre-processing component and the logic re-processing component are run again for processing; otherwise, the processing is terminated.
8. The system for processing high-concurrency or large-volume services according to claim 7, characterized in that: The logic pre-processing component includes a multi-thread processor for synchronously processing different simple modules belonging to the same request data packet in a multi-thread processing manner.
