A method and system for handling message queue consumption exceptions

By monitoring the probability of database write failures and adjusting the way consumers obtain data, the problem of increased system load and vicious cycle caused by database anomalies was solved, and the stability and efficiency of the system were improved.

CN115827292BActive Publication Date: 2026-05-01PING AN BANK CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
PING AN BANK CO LTD
Filing Date
2022-11-04
Publication Date
2026-05-01

AI Technical Summary

Technical Problem

During the consumption of message queue data, database anomalies leading to failures and retries, as well as increased system load, create a vicious cycle that affects other services, and failure retries further burden the database.

Method used

By monitoring the failure rate of writing data to the database, we can adjust how consumers retrieve data from the message queue, reduce the amount of data retrieved when the database is abnormal, and increase the amount of data retrieved when the database is recovered. We can also use different consumption rates and thread numbers to optimize data processing.

Benefits of technology

It reduces system load, avoids database burden, improves system stability and efficiency, and prevents system crashes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115827292B_ABST
    Figure CN115827292B_ABST
Patent Text Reader

Abstract

The application provides a method for processing message queue consumption exception, which comprises the following steps: controlling a consumer to acquire data to be processed from a message queue according to a preset first consumption mode; processing the format of the data to be processed; writing the processed data into a database; acquiring an actual failure probability value of the processed data written into the database; acquiring a probability difference value between the actual failure probability value and a preset failure probability value; acquiring a preset second consumption mode corresponding to the probability difference value according to the probability difference value; and acquiring data to be processed from the message queue according to the preset second consumption mode. The application improves work efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of message queue data processing, and more particularly to a method for handling message queue consumption anomalies and a system for controlling the handling of message queue consumption anomalies. Background Technology

[0002] When consuming and processing data from a message queue, we may encounter exceptions during the processing, such as database anomalies preventing messages from being successfully written. This results in numerous errors and retries to try and ensure successful data writing. However, this presents two problems: First, without effective handling until the database recovers, the system will continue to report errors, increasing system load and impacting other services. Second, retries can double the number of database requests, potentially leading to database unrecovery and a vicious cycle. Summary of the Invention

[0003] In view of this, it is necessary to provide a method and system for handling message queue consumption exceptions that can improve work efficiency, as well as a computer-readable storage medium.

[0004] In a first aspect, this application provides a method for handling message queue consumption anomalies, the method comprising: controlling a consumer to retrieve data to be processed from a message queue according to a preset first consumption method; processing the format of the data to be processed; writing the processed data into a database; obtaining an actual failure probability value for writing the processed data into the database; obtaining a probability difference between the actual failure probability value and a preset failure probability value; obtaining a preset second consumption method corresponding to the probability difference; and the consumer retrieving the data to be processed from the message queue according to the preset second consumption method.

[0005] Secondly, this application provides a system for controlling message queue consumption anomalies. The system includes an acquisition module, a consumption control module, a data processing module, and a probability calculation module. The acquisition module is used for a consumer to acquire data to be processed from a producer; the consumption control module is used to control the consumer to acquire data to be processed from the message queue according to a preset first consumption method; the data processing module is used to process the format of the data to be processed and write the processed data into a database; the probability calculation module is used to calculate the actual failure probability of writing the processed data into the database and the probability difference between the actual failure probability and the preset failure probability, and sends the probability difference to the consumption control module so that the consumption control module selects a preset second consumption method based on the probability difference.

[0006] Thirdly, this application also provides a computer device for handling message queue consumption anomalies, characterized in that the computer device comprises:

[0007] Memory is used to store computer program instructions;

[0008] A processor for executing the computer program instructions to implement the method for handling message queue consumption exceptions as described in any one of claims 1 to 4.

[0009] This application obtains the corresponding consumption method by calculating the difference between the failure rate of business data being written to the database and the preset failure rate. Each difference has a pre-set consumption method, and the data in the message queue is consumed according to the new consumption method, that is, less data is acquired, so as to reduce the impact of database problems on system operation; and after the database is repaired, a new consumption method is acquired to increase the amount of data acquired. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the architecture of a system for controlling message queue consumption exceptions, provided as an embodiment of this application.

[0012] Figure 2 This is a flowchart illustrating a method for handling message queue consumption exceptions provided in an embodiment of this application.

[0013] Figure 3 A list of business data presented in natural language format for embodiments of this application.

[0014] Figure 4 This is a flowchart illustrating another sub-step of step S106.

[0015] Figure 5 A table is provided to set up different consumption methods for different probability value ranges provided in this application.

[0016] Figure 6 This is a schematic diagram of the internal structure of a server used by a control system to handle message queue consumption anomalies, as provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0018] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data are interchangeable where appropriate; in other words, the described embodiments are implemented according to a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, may also include other content; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0020] This application provides a method for handling message queue consumption exceptions, wherein the method runs in a system for handling message queue consumption exceptions. Figure 1As shown, the system includes: an acquisition module 1, a consumption control module 2, a data processing module 3, and a probability calculation module 4. The probability calculation module further includes an output module 5, a comparison module 6, a calculation module 7, a threshold acquisition module 8, a consumption method acquisition module 9, an anomaly detection module 10, and a calculation module 11. The system acquires messages from a message queue, which in turn acquires messages from producers. These producers include several upstream systems that generate business data, such as bank payment systems, ticketing systems, and Taobao APP. Producers send business data to the message queue. The business data is in code format and cannot be understood by the client. The system consists of a server cluster.

[0021] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a method for handling message queue consumption exceptions provided in an embodiment of this application.

[0022] Step S101: Obtain the data to be processed from the message queue according to the first preset consumption method.

[0023] The consumption method includes: a consumption rate and a number of consumption threads. The consumption rate is the speed at which the consumer retrieves messages from the message queue, such as consuming one piece of data every second or one piece of data every three seconds. The number of consumption threads refers to the number of channels for retrieving messages from the message queue, such as opening two channels to consume data simultaneously or opening three channels to consume data simultaneously. The consumer includes a business data processing system. The preset first consumption method can be set by the developers. The data to be processed includes transaction data, notification data, etc., and the business format of the data to be processed is incomprehensible to the customer.

[0024] For example, transaction data generated after a customer completes a transfer in the banking system, or data generated when purchasing train tickets on the Zhixing Ticketing APP, etc., are in code format. The system that generates business data can communicate with the message queue. The message queue obtains the aforementioned transaction data. The message queue provides three configurable interfaces for data retrieval. The business data processing system retrieves data from the three configurable interfaces of the message queue every two seconds.

[0025] Step S102: Process the format of the data to be processed.

[0026] The data to be processed is in code format. The business data in code format is then processed into a format that the user can understand. For example, ... Figure 3 The business data list shown includes the time of the business transaction, the amount of the business transaction, and the parties involved in the business transaction. The business data processing system processes the business data from code format into natural language that users can recognize.

[0027] Step S103: Write the processed data into the database.

[0028] Business data processed into natural language is written into the database.

[0029] Step S104: Obtain the actual failure probability value of writing the processed data into the database.

[0030] When a database encounters an anomaly, not all business data in natural language format will be written to the database; only a portion of the business data will be written. The percentage of data successfully written to the database out of all data is calculated, which is the actual failure probability value, for example, 50%.

[0031] Step S105: Obtain the probability difference between the actual failure probability value and the preset failure probability value.

[0032] The preset failure probability value is a pre-set failure probability value, which represents the acceptable failure rate for writing to the database. Under this preset failure probability, the system will not crash and can continue to run. The actual failure probability value is output and compared with the preset failure probability value. If the actual failure probability value is not equal to the preset failure probability value, the probability difference between the actual failure probability value and the preset failure probability value is calculated.

[0033] Step S106: Based on the probability difference, obtain the second preset consumption method corresponding to the probability difference from the preset consumption method configuration scheme. The second preset consumption method is different from the first preset consumption method. The preset consumption method configuration scheme is configured with a consumption method list. The consumption method list includes a correspondence between several probability difference ranges and several second preset consumption methods. Different probability difference ranges correspond to different second preset consumption methods.

[0034] Different probability differences correspond to different consumption methods. A preset second consumption method is obtained based on the calculated probability difference. For example, if the probability difference is 20%, the corresponding consumption method is to consume one piece of data every second, and the message queue consumption interface is reduced by 10%.

[0035] Step S107: The consumer retrieves the data to be processed from the message queue according to the preset second consumption method.

[0036] According to the preset second consumption method, data is obtained from the message queue, so that the system can reduce the amount of data obtained from the message queue when the database has problems, and increase the amount of data obtained from the message queue when the database receives data normally.

[0037] For example, if the current probability difference is 50%, the corresponding consumption method is to consume one piece of data, sleep for 3 seconds, and reduce the number of consumption threads by 50%. By consuming data according to this consumption method, less data is acquired, thereby reducing the probability difference. When the probability difference is reduced to 20%, the consumption method is to consume one piece of data, sleep for 1 second, and reduce the number of consumption threads by 10%, thereby acquiring more data and improving work efficiency.

[0038] The system monitors in real time whether all the processed data has been written to the database; if not all the processed data has been written to the database, it sends a message to the controller to control the consumer to retrieve data from the message queue according to the second consumption method.

[0039] Figure 4 This is a flowchart illustrating another sub-step of step S106. Step 106 includes steps S1061-S1062.

[0040] Step S1061: Obtain the threshold range to which the probability difference belongs. Different threshold ranges correspond to different consumption rates and the number of consumption threads.

[0041] like Figure 5 As shown, the preset consumption method configuration scheme is configured with a consumption method list, which has different probability value ranges for setting different consumption methods, including consumption rate and number of consumption threads.

[0042] Step S1062: Obtain the consumption rate and the number of consumption threads corresponding to the threshold range.

[0043] like Figure 1 As shown, Figure 1 This is a schematic diagram of the architecture of a system for controlling message queue consumption anomalies, provided in an embodiment of this application. The system includes: an acquisition module 1, a consumption control module 2, a data processing module 3, and a probability calculation module 4. The probability calculation module 4 further includes an output module 5, a comparison module 6, a calculation module 7, a threshold acquisition module 8, a consumption mode acquisition module 9, an anomaly detection module 10, and a calculation module 11.

[0044] Module 1 is used to retrieve data to be processed from the message pair column.

[0045] The consumption control module 2 is used to retrieve data to be processed from the message queue according to the preset first consumption method.

[0046] The consumption method includes: a consumption rate and a number of consumption threads. The consumption rate is the speed at which the consumer retrieves messages from the message queue, such as consuming one piece of data every second or one piece of data every three seconds. The number of consumption threads refers to the number of channels for retrieving messages from the message queue, such as opening two channels to consume data simultaneously or opening three channels to consume data simultaneously. The consumer includes a business data processing system. The preset first consumption method can be set by the developers. The data to be processed includes transaction data, notification data, etc., and the business format of the data to be processed is incomprehensible to the customer.

[0047] Data processing module 3 is used to process the format of the data to be processed and write the processed data into the database.

[0048] The data to be processed is in code format. The business data in code format is then processed into a format that the user can understand. For example, ... Figure 3 As shown, the business data includes the time of the business transaction, the amount of the business transaction, and the parties involved in the business transaction. The business data processing system processes the business data from code format into natural language that users can recognize, and writes the processed natural language business data into the database.

[0049] The probability calculation module 4 is used to calculate the actual failure probability value of writing the processed data into the database and the probability difference between the actual failure probability value and the preset failure probability value, and send the probability difference value to the consumption control module so that the consumption control module selects a preset second consumption mode based on the probability difference value.

[0050] Output module 5 is used to output the actual failure probability value.

[0051] The comparison module 6 is used to compare the actual failure probability value with the preset failure probability value.

[0052] The calculation module 7 is used to calculate the probability difference between the actual failure probability value and the preset failure probability value when the actual failure probability value is not equal to the preset failure probability value.

[0053] The preset failure probability value is a pre-set failure probability value, which represents the acceptable failure rate for writing to the database. Under this preset failure probability, the system will not crash and can continue to run. The actual failure probability value is output and compared with the preset failure probability value. If the actual failure probability value is not equal to the preset failure probability value, the probability difference between the actual failure probability value and the preset failure probability value is calculated.

[0054] The threshold acquisition module 8 is used to acquire the threshold range to which the probability difference belongs. Different threshold ranges correspond to different consumption rates and the number of consumption threads.

[0055] The consumption method acquisition module 9 is used to acquire the consumption rate and the number of consumption threads corresponding to the threshold range.

[0056] According to the preset second consumption method, data is obtained from the message queue. The system for handling message queue consumption anomalies can reduce the amount of data retrieved from the message queue when the database has problems, and increase the amount of data retrieved from the message queue when the database is receiving data normally.

[0057] For example, if the current probability difference is 50%, the corresponding consumption method is to consume one piece of data, sleep for 3 seconds, and reduce the number of consumption threads by 50%. By consuming data according to this consumption method, less data is acquired, thereby reducing the probability difference. When the probability difference is reduced to 20%, the consumption method is to consume one piece of data, sleep for 1 second, and reduce the number of consumption threads by 10%, thereby acquiring more data and improving work efficiency.

[0058] The anomaly detection module 10 is used to detect whether all the processed data has been written to the database. When it is detected that not all the processed data has been written to the database, an anomaly message is sent to the probability calculation module.

[0059] The system monitors in real time whether all the processed data has been written to the database; if not all the processed data has been written to the database, it sends a message to the controller to control the consumer to retrieve data from the message queue according to the second consumption method.

[0060] The calculation module 11 is used to receive information sent by the anomaly detection module. When the information indicates an anomaly, it sends information to the consumption control module to change the consumption mode so as to control the consumer to obtain the data to be processed from the message queue according to the preset second consumption mode.

[0061] Please check Figure 6 This is a schematic diagram of the internal structure of a server for handling message queue consumption anomalies in a control system, as provided in an embodiment of this application. The server 200 for handling message queue consumption anomalies in a control system is used for control. The computer device 200 includes a processor 201 and a memory 202. The processor 201 is used to execute an executable program to implement the method for handling message queue consumption anomalies as provided in the above embodiment, and the memory 202 is used to store the computer executable program and the system for handling message queue consumption anomalies in a control system provided in the above embodiment.

[0062] In some embodiments, processor 201 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip, used to run a computer-executable program stored in memory 202. Specifically, processor 201 executes the executable program to implement the method for handling message queue consumption exceptions provided in the above embodiments.

[0063] The memory 202 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 202 can be an internal storage unit of the computer device 200, such as the hard disk of the computer device 200. In other embodiments, the memory 202 can also be an external storage device of the computer device 200, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 200. Furthermore, the memory 202 can include both internal storage units and external storage devices of the computer device 200. The memory 202 can be used not only to store application software and various types of data installed on the computer device 200, such as code implementing methods for handling message queue consumption exceptions, but also to temporarily store data that has been output or will be output and modules of the system controlling the system to handle message queue consumption exceptions.

[0064] Computer device 200 also includes bus 203. Bus 203 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0065] Furthermore, the computer device 200 may also include a display component 204. The display component 204 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display component 204 may also be appropriately referred to as a display device or display unit, used to display information processed in the computer device 200 and to display a visual user interface.

[0066] Furthermore, the computer device 200 may also include a communication component 205. The communication component 205 may optionally include a wired communication component and / or a wireless communication component (such as a Wi-Fi communication component, a Bluetooth communication component, etc.), which is typically used to establish a communication connection between the computer device 200 and other computer devices.

[0067] Figure 6 Only a computer device 200 with partial components and a method for handling message queue consumption exceptions is shown. Those skilled in the art will understand that... Figure 6 The structure shown does not constitute a limitation on the computer device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0068] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0069] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to embodiments of the present invention is generated. The computer device may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0072] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0074] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard disks, read-only storage media (ROM), random access storage media (RAM), magnetic disks, or optical disks.

[0075] It should be noted that the sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0076] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for handling message queue consumption exceptions, characterized in that, The method includes: Data to be processed is retrieved from the message queue according to the first preset consumption method; The format of the data to be processed is processed, and the processed data is written into the database; Obtain the actual failure probability value of writing the processed data into the database; Obtain the probability difference between the actual failure probability value and the preset failure probability value; Based on the probability difference, a second preset consumption method corresponding to the probability difference is obtained from the preset consumption method configuration scheme. The second preset consumption method is different from the first preset consumption method. The preset consumption method configuration scheme is configured with a consumption method list. The consumption method list includes a correspondence between several probability difference ranges and several second preset consumption methods. Different probability difference ranges correspond to different second preset consumption methods. Data to be processed is retrieved from the message queue according to the second preset consumption method.

2. The method for handling message queue consumption exceptions as described in claim 1, characterized in that, The step of obtaining the probability difference between the actual failure probability value and the preset failure probability value includes: The actual failure probability value is calculated. Compare the actual failure probability value with the preset failure probability value; If the actual failure probability value is not equal to the preset failure probability value; Calculate the probability difference between the actual failure probability value and the preset failure probability value.

3. The method for handling message queue consumption exceptions as described in claim 1, characterized in that, The consumption method includes consumption rate and number of consumption threads, and obtaining the second consumption method corresponding to the probability difference based on the probability difference includes: Identify the threshold range to which the probability difference belongs; different threshold ranges correspond to different consumption rates and the number of consumption threads. The second consumption method is obtained by acquiring the consumption rate and the number of consumption threads corresponding to the threshold range.

4. The method for handling message queue consumption exceptions as described in claim 1, characterized in that, The data to be processed obtained from the message queue according to the second preset consumption method includes: Check whether all the processed data has been written to the database; If not all the processed data is written to the database, a message is sent to the controller to control the consumer to retrieve the data to be processed from the message queue according to the second preset consumption method.

5. A system for handling data consumption anomalies in a message queue, characterized in that, include: The retrieval module is used to retrieve data to be processed from the message pair column; The consumption control module is used to retrieve data to be processed from the message queue according to a first preset consumption method or a second preset consumption method; The data processing module is used to process the format of the data to be processed and write the processed data into the database; The probability calculation module is used to calculate the actual failure probability of writing the processed data into the database and the probability difference between the actual failure probability and the preset failure probability. Based on the probability difference, it retrieves a second preset consumption method corresponding to the probability difference from a preset consumption method configuration scheme. The second preset consumption method is different from the first preset consumption method. The preset consumption method configuration scheme is configured with a consumption method list. The consumption method list includes several probability difference ranges and several correspondences between second preset consumption methods. Different probability difference ranges correspond to different second preset consumption methods.

6. The system for handling data consumption anomalies in a message queue as described in claim 5, characterized in that, The probability calculation module also includes: The output module is used to output the actual failure probability value; The comparison module is used to compare the actual failure probability value with the preset failure probability value; The calculation module is used to calculate the probability difference between the actual failure probability value and the preset failure probability value when the actual failure probability value is not equal to the preset failure probability value.

7. The system for handling data consumption anomalies in a message queue as described in claim 5, characterized in that, The consumption method includes consumption rate and the number of consumption threads. The probability calculation module further includes: The threshold acquisition module is used to determine the threshold range to which the probability difference belongs. Different threshold ranges correspond to different consumption rates and the number of consumption threads. The consumption method acquisition module is used to obtain the consumption rate and the number of consumption threads corresponding to the threshold range to obtain the second preset consumption method.

8. The system for handling data consumption anomalies in a message queue as described in claim 5, characterized in that, The system also includes: The anomaly detection module is used to detect whether all the processed data has been written to the database. When it is detected that the processed data has not been written to the database, an anomaly message is sent to the probability calculation module. The probability calculation module is used to receive information sent by the anomaly detection module. When the information indicates an anomaly, it sends information to the consumption control module to change the consumption mode so as to control the consumer to obtain the data to be processed from the message queue according to the second preset consumption mode.

9. A computer device for handling message queue consumption exceptions, characterized in that, The computer device includes: Memory is used to store computer program instructions; A processor for executing the computer program instructions to implement the method for handling message queue consumption exceptions as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed by a processor to implement the method for handling message queue consumption exceptions as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Data processing method and device

    CN111488625A

  • Message queue-based message consumption method and device, computer equipment and a medium

    CN112667414A