Service message processing method and system based on message fingerprints and intelligent delay scheduling

Abstract

The invention provides a service message processing method and system based on a message fingerprint and intelligent delay scheduling. The method comprises the following steps: generating a unique fingerprint based on a key field of a service message; duplicate checking is performed on the generated unique fingerprint based on Redis, and automatic anti-duplicate and idempotent control of service processing is realized by adopting a distributed lock; recording a unique fingerprint and a state thereof through a cache, and setting expiration time based on business tolerance; the abnormal types capable of being retried are screened, dynamic delay retry is carried out on the abnormal types capable of being retried on the basis of expiration time corresponding to service setting, and delay retry is delayed and triggered through a time wheel; and finally verifying the current state of the service and updating the state of the unique fingerprint. The problems of message loss, repeated processing and retry strategy stiffness are solved, and message processing is not lost, repeated and overstocked.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer software, and particularly relates to a business message processing method and system based on message fingerprints and intelligent delay scheduling. BACKGROUND

[0002] In a distributed system, when a business system processes asynchronous messages through a message queue (such as RabbitMQ, Kafka), the following technical defects exist:

[0003] 1. Message loss risk: When the business system processes messages, if an exception occurs (such as database crash, network interruption), and the consumption state is not correctly fed back, the message may not be re-delivered and permanently lost.

[0004] 2. Repetitive processing problem: The traditional scheme relies on the retry mechanism of the message queue, but cannot identify messages with the same content but different IDs (such as different systems generating the same business event), resulting in resource waste and data inconsistency.

[0005] 3. Rigid retry strategy: The fixed time interval retry mechanism (such as retrying every 5 seconds) cannot adapt to the business recovery time, which may cause invalid retries or processing delays. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a business message processing method and system based on message fingerprints and intelligent delay scheduling to solve at least one of the above problems existing in the prior art.

[0007] To achieve the above purpose, one or more embodiments of the present application provide a business message processing method based on message fingerprints and intelligent delay scheduling, which includes the following steps:

[0008] S1. Generating a unique fingerprint based on the key fields of the business message;

[0009] S2. Checking the generated unique fingerprint for duplication based on Redis, and using a distributed lock to realize automatic duplication prevention and idempotent control of business processing; recording the unique fingerprint and its state through caching, and setting an expiration time based on business tolerance;

[0010] S3. Screening the retryable exception types, and dynamically delaying the retry of the retryable exception types based on the corresponding expiration time set by the business, wherein the delay retry is triggered by a time wheel delay;

[0011] S4. Finally checking the current state of the business and updating the state of the unique fingerprint.

[0012] According to the above technical scheme of the present application, the following improvements can also be made;

[0013] Optionally, in the step S1, the unique fingerprint is generated by hashing after splicing the key field of the service message, and if the unique service ID is included in the service message, the unique service ID is directly reused as the unique fingerprint.

[0014] Optionally, in the step S2, the duplicate checking on the generated unique fingerprint based on Redis comprises: querying Redis to determine whether the unique fingerprint exists, if not, the unique fingerprint is used as a key to seize a distributed lock, the business logic is executed, and the distributed lock is released after the business processing is successful; if it already exists, it is directly skipped to avoid repeated processing.

[0015] Optionally, in the step S3, the dynamic delay is realized by the following logic: the time of the delay retry is increased based on the number of times of the service failure caused by the exception.

[0016] Optionally, before the delay retry is triggered, the actual state of the service is queried, if the actual state of the service shows that the service has been completed, the state of the unique fingerprint is updated and the delay retry is terminated; if the actual state of the service shows that the service has not been completed, the delay retry step of the step S3 is executed.

[0017] According to the second aspect of the present application, a service message processing system based on message fingerprint and intelligent delay scheduling is provided, comprising:

[0018] A message fingerprint generation module generates a unique fingerprint based on the key field of the service message;

[0019] A state marker management module records the service processing state through a distributed cache;

[0020] A dynamic delay scheduling module triggers the abnormal retry strategy of the different service according to the type of the exception;

[0021] A bottom-up verification module performs the final consistency verification based on the service data state.

[0022] The message fingerprint generation module, the state marker management module, the dynamic delay scheduling module and the bottom-up verification module are sequentially and communicatively connected.

[0023] According to the above technical solution of the present application, the following improvements can be made:

[0024] Optionally, it further comprises a message receiving / listening module, the message receiving / listening module is used for capturing the service message in real time, and the message receiving / listening module is communicatively connected with the message fingerprint generation module.

[0025] According to the third aspect of the present application, a computer readable storage medium is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, when the computer program is executed by the processor, the method steps of any one of the above are realized.

[0026] According to a fourth aspect of the present application, there is provided an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the method steps of any of the above.

[0027] The present application has the beneficial effect that the present application provides a business message processing method and system based on message fingerprint and intelligent delay scheduling, solves the problems of message loss, repeated processing, and rigid retry strategy, and realizes non-loss, non-repetition, and non-backlog of message processing. Through the business field hash fingerprint, content-level deduplication is realized, and resource waste caused by repeated pushing of the same business event by different systems is avoided. According to the retry strategy, the business system recovery time is adapted, and the number of invalid retries is reduced by more than 40%. Before retrying, the business data state is actively queried, the extreme scenario of "successful marking but incomplete business" is solved, and the data consistency reaches 99.999%. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The figure is a principle block diagram of the business message processing method and system based on message fingerprint and intelligent delay scheduling of the embodiments of the present application.

[0029] Figure 2 The figure is a running flowchart of the business message processing method and system based on message fingerprint and intelligent delay scheduling of the embodiments of the present application. DETAILED DESCRIPTION

[0030] In order to make the objects, technical solutions, and advantages of the present disclosure clearer, further detailed description will be made to the present disclosure with reference to specific embodiments and the accompanying drawings.

[0031] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood as the general meaning understood by those skilled in the art to which the present disclosure belongs. The terms "first", "second", and similar terms used in one or more embodiments of the present application do not represent any order, quantity, or importance, but are only used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms cover the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0032] As Figure 1 andFigure 2 A message fingerprint and intelligent delay scheduling-based service message processing method of one or more embodiments in the present application includes the following steps:

[0033] S1, generating a unique fingerprint based on the key field of the service message; specifically, the unique fingerprint is generated by hashing after splicing the key field of the service message, and if the unique service ID is already included in the service message, the unique service ID is directly reused as the unique fingerprint.

[0034] S2, checking the generated unique fingerprint based on Redis, and using a distributed lock to realize automatic duplicate prevention and idempotent control of service processing; the unique fingerprint and its state are recorded in the cache, and the expiration time is set based on the service tolerance; specifically, checking the generated unique fingerprint based on Redis includes: querying Redis to determine whether the unique fingerprint exists, if not, the unique fingerprint is used as a key to seize a distributed lock, and the business logic is executed, and the distributed lock is released after the service processing is successful; if it already exists: directly skip to avoid duplicate processing. Among them, the distributed lock is realized by using the open source framework Redission, and the background thread can automatically renew the lock to avoid the lock expiring when the service processing is not completed. The business logic (such as updating the contract signing state) is executed, and the distributed lock is released after the processing is successful. In addition, the fingerprint and the state (such as “processing” and “not processing”) are stored in the cache, and the expiration time is set according to the business tolerance (such as 5 minutes).

[0035] S3, screening the retryable exception type, and dynamically delaying the retry of the retryable exception type based on the corresponding expiration time set by the business, and the delay retry is triggered by a time wheel; specifically, the dynamic delay is realized by the following logic: the time of the delay retry is increased based on the number of times the business fails due to exceptions.

[0036] Retryable exception: refers to an exception situation that is temporary and has the possibility of retry after the system or external environment recovers.

[0037] Typical examples are as follows:

[0038] Database connection failure (such as connection timeout, connection pool depletion); network communication interruption or delay (such as HTTP timeout); third-party interface temporarily unavailable (such as payment platform maintenance); distributed lock competition failure; state cache temporarily inaccessible, etc.

[0039] Non-retryable exception: refers to the exception caused by business logic error, parameter exception, permission error and other problems, which has no retry significance. Typical examples are as follows: illegal parameters (such as missing contract ID, field format error); business state mismatch (such as a contract to be signed has completed signing); message structure error (such as JSON parsing failure); signature verification failure or request authentication failure; fingerprint generation logic error (such as the key field of business message is empty) and the like.

[0040] Dynamic retry strategy: first failure: retry after 5 seconds; second failure: retry after 30 seconds; third failure: retry after 10 minutes. The maximum number of retries can also be set: 3 times.

[0041] S4, final check on the current state of the business and update the state of the unique fingerprint. Before triggering the delay retry, query the actual state of the business, if the actual state of the business shows that the business has been completed, update the state of the unique fingerprint and terminate the delay retry; if the actual state of the business shows that the business has not been completed, execute the delay retry step of step S3. As an example, check the actual state of the business data (such as the contract signing state in the database) before retry; if the business has been successfully completed, update the fingerprint state to "success" and terminate the processing.

[0042] The problems of message loss, repeated processing and rigid retry strategy are solved, and the message processing is not lost, not repeated and not accumulated. Through the business field hash fingerprint, content-level deduplication is realized (compared with traditional Message ID deduplication), which avoids the waste of resources caused by repeated pushing of the same business event by different systems. According to the retry strategy (such as 5s→30s→10min), the business system recovery time is adapted, and the number of invalid retries is reduced by more than 40%. Before retry, the business data state is actively queried to solve the extreme scenario of "marked success but business not completed", so that the data consistency reaches 99.999%.

[0043] As shown in Figure 1 Another embodiment provides a business message processing system based on message fingerprint and intelligent delay scheduling, which comprises:

[0044] A message fingerprint generation module generates a unique fingerprint based on the key fields of the business message;

[0045] A state marking management module records the business processing state through a distributed cache;

[0046] A dynamic delay scheduling module triggers different business exception retry strategies according to the exception type;

[0047] A bottom-up verification module performs final consistency check based on the business data state;

[0048] The message fingerprint generation module, the state mark management module, the dynamic delay scheduling module and the bottom-checking module are sequentially and communicatively connected;

[0049] The message receiving / listening module is used for capturing service messages in real time, and is communicatively connected with the message fingerprint generation module.

[0050] In another embodiment, a computer readable storage medium is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is executed by the processor to implement the method steps of any one of the above.

[0051] In yet another embodiment, an electronic device is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is executed by the processor to implement the method steps of any one of the above.

[0052] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 a system that performs the functions specified in one or more blocks or flows.

[0054] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction system that implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 a system that performs the functions specified in one or more blocks or flows.

[0055] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operations steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide the function of realizing the processes specified in the flowcharts Figure 1 one flowchart or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0056] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional modifications and variations to these embodiments without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be construed to include all such modifications and variations as fall within the scope of the application.

[0057] Obviously, various modifications and changes are possible in the present application without departing from the spirit and scope of the application. It is to be understood that the application includes any such modifications and changes only insofar as they come within the scope of the appended claims and their equivalents.

Claims

1. A business message processing method based on message fingerprinting and intelligent delay scheduling, characterized in that, Includes the following steps: S1. Generate a unique fingerprint based on the key fields of the business message; S2. Based on Redis, the generated unique fingerprint is deduplicated, and a distributed lock is used to implement automatic deduplication prevention and idempotent control of business processing; the unique fingerprint and its status are recorded by caching, and an expiration time is set based on the business tolerance. S3. Filter retryable exception types and dynamically delay retry for retryable exception types based on the expiration time corresponding to the business settings. This delayed retry is triggered by the time wheel delay. S4. Perform a final verification of the current business status and update the status of the unique fingerprint.

2. The business message processing method based on message fingerprinting and intelligent delay scheduling as described in claim 1, characterized in that, In step S1, the unique fingerprint is generated by hashing the key fields of the business message. If the business message already contains a unique business ID, then the unique business ID is directly reused as its unique fingerprint.

3. The business message processing method based on message fingerprinting and intelligent delay scheduling as described in claim 2, characterized in that, In step S2, the deduplication of the generated unique fingerprint based on Redis includes: querying Redis to determine if the unique fingerprint exists; if it does not exist, the unique fingerprint is used as the key to preempt the distributed lock, execute the business logic, and release the distributed lock after the business processing is successful; if it already exists, it is skipped directly to avoid duplicate processing.

4. The business message processing method based on message fingerprinting and intelligent delay scheduling as described in claim 1, characterized in that, In step S3, the dynamic delay is implemented using the following logic: the delay retry time is increased based on the number of times the business fails due to anomalies.

5. The business message processing method based on message fingerprinting and intelligent delay scheduling as described in claim 1, characterized in that, Before the delayed retry is triggered, the actual status of the service is queried. If the actual status of the service shows that the service has been completed, the status of the unique fingerprint is updated and the delayed retry is terminated. If the actual status of the service shows that the service has not been completed, the delayed retry step in step S3 is executed.

6. A business message processing system based on message fingerprinting and intelligent delay scheduling, characterized in that, include: The message fingerprint generation module generates a unique fingerprint based on the key fields of the business message; The status flag management module records the business processing status through a distributed cache; The dynamic delay scheduling module triggers a differentiated retry strategy for poor services based on the type of exception. The fallback verification module performs eventual consistency verification based on the status of business data. The message fingerprint generation module, status mark management module, dynamic delay scheduling module, and fallback verification module are connected in sequence.

7. The business message processing system based on message fingerprinting and intelligent delay scheduling as described in claim 6, characterized in that, It also includes a message receiving / listening module, which is used to capture business messages in real time, and the message receiving / listening module is communicatively connected to the message fingerprint generation module.

8. A computer-readable storage medium, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1-5.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1-5.

Images