Implementation and use method of message middleware consuming idempotency

Abstract

The application discloses a kind of message middleware consumption idempotent implementation and use method, belong to message middleware and distributed system technical field.The method responds to the instance of consumption and pulls the message to be consumed, obtains its original byte stream sequence;Based on subject identifier and queue identifier, generate measurement matrix through chaotic mapping;After sparse transformation is carried out to byte stream using sparse transformation base, compression sensing measurement is executed, measurement vector is obtained and is quantized as binary hash signature sequence;The signature is used as query key in idempotent signature cache Existence query, if there is, it is judged as repeated message and discarded, otherwise, store signature and execute business logic.The application maps message content to compact binary fingerprint by compression sensing, realizes high-precision idempotent deduplication based on message full content, with extremely low memory overhead and no business intrusion;Combined with chaotic measurement matrix, online dictionary learning and deep reinforcement learning dynamic parameter adjustment, with high creativity and authorization stability.

Description

Technical Field

[0001] This invention relates to the field of message middleware and distributed system technology, and in particular to an implementation and usage method of message middleware idempotent consumption. Background Technology

[0002] In distributed systems, message middleware, as a core component of asynchronous communication, is widely used in scenarios such as decoupling, traffic shaping, and asynchronous processing. To ensure reliable message delivery in unreliable network environments, message middleware generally adopts "at least once" delivery semantics. This semantics may lead to consumer instances receiving the same message repeatedly in abnormal situations such as network jitter, consumer crashes and restarts, partition rebalancing, or consumption confirmation timeouts. If the consumer end does not have idempotent processing capabilities, duplicate consumption will directly cause serious business logic errors such as data inconsistency, duplicate deductions, duplicate records, and duplicate points issuance.

[0003] Current mainstream solutions to the idempotency problem in message consumption include: First, based on database unique key constraints, which involves setting a unique index in the business data table and using the database's transaction characteristics to prevent duplicate insertions. This solution heavily relies on database performance; unique index verification becomes a system bottleneck during high-concurrency writes and cannot handle complex transaction scenarios involving multiple table updates. Second, based on distributed locks, which lock the unique identifier of the message before consumption and release the lock after consumption. This solution introduces additional network requests and lock contention overhead and carries risks such as deadlocks and lock timeouts, significantly reducing system throughput. Third, based on state machines, which require business messages to carry a current state field, allowing consumers to determine whether processing has occurred. This solution is highly intrusive to business code, unsuitable for existing legacy systems, and lacks general applicability.

[0004] Existing technologies include message deduplication schemes based on Bloom filters, which store the identifiers of consumed messages in the Bloom filter. This method is memory efficient, but suffers from false positives, potentially discarding unconsumed messages as consumed, which is unacceptable for scenarios such as order payments and inventory deductions. Furthermore, conventional message identifiers such as message IDs, offsets, and UUIDs are typically generated at the message producer or middleware, failing to uniquely represent the integrity of message content—messages with identical content but different IDs cannot be identified, and messages with the same ID but tampered content cannot be detected. Therefore, there is an urgent need for a general-purpose idempotent consumption method that does not rely on external storage systems, is non-intrusive to business logic, can accurately deduplicate messages based on their entire content, and has extremely low memory overhead. Summary of the Invention

[0005] The purpose of this invention is to provide an implementation and usage method for idempotent message middleware consumption, so as to solve the technical problems in the prior art that idempotent processing depends on a specific storage structure, is highly intrusive to business operations, cannot accurately deduplicate based on the full content of the message, and has a large memory overhead.

[0006] To achieve the above objectives, this invention provides a method for implementing and using idempotent message middleware consumption, comprising the following steps: Step S1: In response to the consumer instance pulling the message to be consumed from the message middleware, obtain the original byte stream sequence of the message to be consumed; Step S2: Based on the topic identifier and queue identifier corresponding to the consumption instance, generate a measurement matrix using a chaotic mapping function; Step S3: Perform a sparse transformation on the original byte stream sequence using a sparse transformation basis to obtain a sparse coefficient vector; Step S4: Perform compressed sensing measurement on the measurement matrix and the sparse coefficient vector to obtain the measurement vector, and quantize the measurement vector into a binary hash signature sequence through a sign function; Step S5: Use the binary hash signature sequence as the query key to perform an existence query in the idempotent signature cache; if the query key exists, determine that the message to be consumed is a duplicate message, return a consumption confirmation signal to the message middleware and discard the message to be consumed; if the query key does not exist, proceed to step S6. Step S6: Store the binary hash signature sequence into the idempotent signature cache, and execute the business logic corresponding to the message to be consumed. After the business logic is successfully executed, return a consumption confirmation signal to the message middleware.

[0007] Preferably, in step S2, the chaotic mapping function is a logistic mapping, and its iterative formula is: ; in, Here is the chaotic parameter, and its value range is... ; The initial value is obtained by normalizing the hash value of the concatenated string of the topic identifier and the queue identifier; a length of [value] is generated through this iterative formula. The chaotic sequence is reshaped into the measurement matrix, where The length of the original byte stream sequence. To measure the dimension of the vector, and .

[0008] Preferably, the method further includes a step S0: before executing step S1, dynamically adjusting the compression sensing measurement rate based on the current consumption throughput and average message size of the message middleware. ,in The value range is from 0.05 to 0.5.

[0009] Preferably, the measurement rate is dynamically adjusted. Specifically: establish a depth Network model, this depth The input state space of the network model includes the current throughput, average message size, and CPU utilization, and the output action space is... Discretized values ​​are used; the depth is trained with a reward function based on a weighted sum of idempotent cache hit rate and average processing latency. Network model to select the optimal measurement rate .

[0010] Preferably, in step S4, the symbolic function Defined as: ; in, For the measurement vector of the th The component; the first component of the binary hash signature sequence. bits Calculated by the following formula: .

[0011] Preferably, in step S5, the idempotent signature cache is a distributed cache based on a Bloom filter, and the length of the Bloom filter is... and the number of hash functions Based on the expected maximum total number of messages And the expected false positive rate Dynamic configuration, satisfying the following relation: ; ; in, It represents the natural logarithm.

[0012] Preferably, it also includes an online dictionary learning step: every fixed time window T, all raw byte stream sequences processed within that time window are collected as training sample sets, and the sparse transformation basis is updated to an adaptive dictionary using the K singular value decomposition algorithm, so that the representation of the training sample set under the adaptive dictionary is more sparse.

[0013] Preferably, the objective function of this K-singular value decomposition algorithm is: ; in, For the first The original byte stream sequence of each training sample. For the adaptive dictionary to be learned, For this training sample In the dictionary Sparse coding under, This indicates the number of non-zero elements in the sparse coding. This is the preset upper limit for sparsity.

[0014] Preferably, the message to be consumed is a message object conforming to the Java Message Service specification. The original byte stream sequence is obtained by calling the getBody method of the message object and then converting it into a byte array.

[0015] Preferably, the sparse transform basis is a discrete cosine transform basis, and its matrix elements are given by the following equation: ; in, The length of the original byte stream sequence. This represents the first element in the discrete cosine transform basis. Line 1 The elements of the column.

[0016] Therefore, the implementation and usage method of message middleware consumption idempotency using the above structure of the present invention has the following beneficial effects: (1) The equidistant embedding property of compressed sensing in this invention ensures that the probability of collision between binary signatures generated by any two messages with different contents in the measurement domain is extremely low, with a theoretical value of less than 2 to the power of negative M, which is far superior to traditional hashing methods and achieves precise idempotency based on the full content of the message.

[0017] (2) The measurement matrix of the present invention is generated online through chaotic mapping, without the need to store the complete matrix; the signature is compressed to M bits per message, M is only 5% to 20% of the original length; combined with Bloom filter, it can support the deduplication of hundreds of billions of messages in tens of megabytes of memory.

[0018] (3) The present invention dynamically adjusts the measurement rate through a deep Q-network and continuously optimizes the sparse representation base through online dictionary learning, so that the system can adapt to different business scenarios and load changes, and maintain a high hit rate and low latency.

[0019] (4) The compressed sensing technology of this invention is mainly applied to image processing, signal acquisition and other fields, and has never been introduced into the idempotent consumption problem of message middleware. This invention organically integrates chaotic measurement matrix, compressed sensing, symbolic quantization, Bloom filter, deep reinforcement learning and K-SVD dictionary learning. When faced with the idempotent consumption problem, those skilled in the art would not think of using the above-mentioned uncommon algorithm combination in the field of signal processing. This technical solution has significant inventiveness and high licensing stability.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] Figure 1 This is an overall flowchart illustrating the implementation and usage method of an idempotent message middleware consumer according to the present invention. Detailed Implementation

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0024] Example This invention proposes an implementation and usage method for idempotent message middleware consumption, the overall process of which is as follows: Figure 1 As shown. After the consumer instance pulls a message to be consumed from the message middleware, this method performs the following operations in sequence: S1. Obtain the raw byte stream: Extract the message body of the message object (such as ConsumerRecord in Kafka), convert it into a byte array, and obtain the raw byte stream sequence. Its length is denoted as .

[0025] S2. Generate a measurement matrix: Based on the topic name and partition number (queue identifier) ​​subscribed to by the current consumption instance, generate a measurement matrix using chaotic mapping. OK Column measurement matrix The chaotic mapping uses a logistic mapping, with initial values ​​determined by the hash values ​​of the topic and queue identifiers. and ratio This is called the measurement rate, which can be fixed or dynamically adjusted.

[0026] S3. Sparse Transformation: Utilizing sparse transformation basis (Initially using the discrete cosine transform basis) for Perform a sparse transformation to obtain a sparse coefficient vector. Because the message content has statistical correlation, Most of the coefficients are close to zero, with only a few coefficients being significantly non-zero.

[0027] S4. Compressed Measurement and Quantization: Calculating Measurement Vectors , The length is Then on Each component Applying symbolic functions: If Output 1 if the output is 1, otherwise output 0, thus obtaining a value of length 1. binary hash signature sequence .

[0028] S5, Idempotency Query: Use the key to query a distributed idempotent signature cache (such as a Redis-based Bloom filter). If it already exists in the cache... This indicates that the message has already been consumed, and it is determined to be a duplicate message. An ACK is then returned directly to the message middleware, and the message is discarded.

[0029] S6. Storage and Business Execution: If If it does not exist, then first... Insert the data into the cache, and then call the business logic function to execute the specific business logic (such as database writes, RPC calls, etc.). After successful execution, return an ACK to the message broker.

[0030] In addition, the method includes two optional adaptive optimization mechanisms: one is dynamic measurement rate adjustment, which is achieved through depth... The network selects the optimal [system / mechanism] in real time based on system load. Secondly, online dictionary learning, using the K-SVD algorithm to periodically update the sparse transform basis. This makes it more consistent with the statistical characteristics of current messages.

[0031] This embodiment uses the order system of a large e-commerce platform as an example. The system's message middleware uses Apache Kafka 2.8. The order service is a consumer instance in a consumer group, subscribing to a topic named "order-created" which contains three partitions. Each message's value is a JSON string, for example: ``` {"orderId":"ORD-10001","userId":"USER-888","amount":199.99,"items":[{"sku":"A123","qty":1}],"timestamp":"2025-06-12T10:30:00Z"} ``` The following describes the six steps of the technical solution with specific numerical examples.

[0032] Step S1: Obtain the raw byte stream: The consumer instance retrieves a ConsumerRecord using the KafkaConsumer.poll method. This record belongs to partition 0 of the "order-created" topic. The record.value method is called to obtain the aforementioned JSON string, which is then converted to a byte array using String.getBytes(StandardCharsets.UTF_8). The calculated length of this byte array is 256 bytes, therefore the original byte stream sequence... , .

[0033] Step S2: Generate the measurement matrix: Assume the current measurement rate Then the measurement dimension (Round up). The topic identifier is "order-created", and the queue identifier is "0", concatenated to form the string "order-created-0". Calculate the SHA-256 hash value of this string, take the first 64 bits, convert it to a double-precision floating-point number, and normalize it to the (0,1) interval to obtain the initial chaotic value. Setting chaos parameters According to the logistic mapping iteration formula: ; Iteration Next, generate a chaotic sequence. The sequence is then reshaped into a 51-row, 256-column matrix in row-major order, yielding the measurement matrix. .

[0034] Step S3, Sparse Transformation: Using Discrete Cosine Transform Basis Its matrix elements are given by the following formula: ; ; For the original byte stream sequence Perform sparse transformation: The calculated sparse coefficient vector In the sample, there are approximately 26 components with an absolute value greater than 0.01 (about 10%), and the remaining components are close to zero, satisfying the sparsity assumption.

[0035] Step S4, Compressed Measurement and Binary Quantization: Perform matrix multiplication: , to obtain the measurement vector Assuming The specific values ​​of the 51 components are as follows (only the first 5 and the last 2 are shown): .

[0036] Apply the sign function to each component and calculate the binary bits according to the formula: ; Obtain the binary signature sequence It has a length of 51 bits.

[0037] Step S5, Idempotency query: Using the binary signature sequence h as the key, query the idempotent signature cache based on the Redis Bloom filter. The Bloom filter is pre-defined based on the expected total number of messages. (One billion records) and allowable false positive rate Configuration. The length L of the Bloom filter and the number of hash functions k are calculated by the following formula: ; .

[0038] The query result was "not found".

[0039] Step S6, Storage and Business Execution: The system first inserts `h` into a Bloom filter (mapping `h` to 13 positions out of L bits using 13 hash functions and setting these positions to 1). Then, it calls the order processing logic: parsing the JSON string, validating the `orderId` format and `userId` validity, writing the order information to the MySQL order table (`order_table`) using JDBC, and deducting inventory via the inventory service's RPC interface. After successful execution, it calls `consumer.commitSync` to commit the offset and returns a consumption confirmation to Kafka.

[0040] Validation of repeat consumption scenarios: Suppose that Kafka did not receive the aforementioned acknowledgment due to a network timeout, and therefore re-delivered the message to the same consumer instance (or another instance). When the same JSON string is retrieved again, steps S1 to S4 are repeated. Since the message content is exactly the same, the generated binary signature h is the same as before. When querying the Bloom filter in step S5, since h already exists in the cache, the query returns "exists". The system directly skips the business logic and returns an ACK to Kafka, thereby avoiding the repeated execution of business operations and achieving idempotent consumption.

[0041] Example of dynamic measurement rate adjustment: During system operation, the consumer instance monitored a current throughput of 8000 TPS (messages per second), an average message size of 1500 bytes, and a CPU utilization of 65%. (Depth) The network model's input state vector is (8000, 1500, 0.65). This model is a three-layer fully connected network (3 nodes in the input layer, 64 nodes in the hidden layer, and 5 nodes in the output layer). After initial training, the model outputs an action value of 0.3, meaning λ=0.3 is selected. Therefore, the system increases the measurement rate from the default 0.2 to 0.3, thereby increasing the signature length, improving the ability to distinguish larger messages, and reducing the risk of false positives.

[0042] Online dictionary learning examples: Every 5 minutes, the system collects the raw byte stream of 1 million order messages processed in the past 5 minutes as a training sample set. Set an upper limit for sparsity. The objective function is solved by iterating 20 times using the K-SVD algorithm: ; Obtain the new adaptive dictionary This dictionary will be used as the basis for the sparse transformation in subsequent step S3. The original discrete cosine transform basis is replaced with an adaptive dictionary. Experiments show that using an adaptive dictionary improves the average value of the sparse coefficient vector. The norm was reduced from approximately 26 to approximately 18, the energy retention rate of compressed sensing measurements was improved by 12%, and the probability of signature collisions was further reduced.

[0043] Summary of working principle: This invention treats message content as a high-dimensional sparse signal, projects it into a low-dimensional space using a chaotic measurement matrix, and obtains a binary fingerprint using a sign function. The minute differences between different message contents in the measurement domain are captured by the nonlinear amplification of the chaotic system and the discretization of the sign function, forming a unique and stable signature. A Bloom filter provides fast existence lookup with low memory usage, while deep reinforcement learning and online dictionary learning enable the system to be adaptive. Ultimately, this achieves idempotent message consumption that is independent of business fields, does not introduce strong external dependencies, and is highly efficient and reliable.

[0044] Therefore, the present invention provides an implementation and usage method for idempotent message middleware consumption using the above-described structure, which completely solves the data inconsistency problem caused by duplicate consumption in message middleware and has extremely high innovation and practical value.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An implementation and usage method of idempotent message middleware consumption, characterized in that, Includes the following steps: Step S1: In response to the consumer instance pulling the message to be consumed from the message middleware, obtain the original byte stream sequence of the message to be consumed; Step S2: Based on the topic identifier and queue identifier corresponding to the consumption instance, generate a measurement matrix using a chaotic mapping function; Step S3: Perform a sparse transformation on the original byte stream sequence using a sparse transformation basis to obtain a sparse coefficient vector; Step S4: Perform compressed sensing measurement on the measurement matrix and the sparse coefficient vector to obtain the measurement vector, and quantize the measurement vector into a binary hash signature sequence through a sign function; Step S5: Use the binary hash signature sequence as the query key to perform an existence query in the idempotent signature cache; if the query key exists, determine that the message to be consumed is a duplicate message, return a consumption confirmation signal to the message middleware and discard the message to be consumed; if the query key does not exist, proceed to step S6. Step S6: Store the binary hash signature sequence into the idempotent signature cache, and execute the business logic corresponding to the message to be consumed. After the business logic is successfully executed, return a consumption confirmation signal to the message middleware.

2. The implementation and usage method of idempotent message middleware consumption according to claim 1, characterized in that, In step S2, the chaotic mapping function is a logistic mapping, and its iterative formula is: ； in, Here is the chaotic parameter, and its value range is... ; The initial value is obtained by normalizing the hash value of the concatenated string of the topic identifier and the queue identifier; a length of [value] is generated through this iterative formula. The chaotic sequence is reshaped into the measurement matrix, where The length of the original byte stream sequence. To measure the dimension of the vector, and .

3. The implementation and usage method of idempotent message middleware consumption according to claim 1, characterized in that, It also includes a step S0: Before executing step S1, dynamically adjust the compression-aware measurement rate based on the current consumption throughput and average message size of the message middleware. ,in The value range is from 0.05 to 0.

5.

4. The implementation and usage method of idempotent message middleware consumption according to claim 3, characterized in that, Dynamically adjust measurement rate Specifically: establish a depth Network model, this depth The input state space of the network model includes the current throughput, average message size, and CPU utilization, and the output action space is... Discretized values ​​are used; the depth is trained with a reward function based on a weighted sum of idempotent cache hit rate and average processing latency. Network model to select the optimal measurement rate .

5. The implementation and usage method of idempotent message middleware consumption according to claim 1, characterized in that, The symbolic function in step S4 Defined as: ； in, For the measurement vector of the th The component; the first component of the binary hash signature sequence. bits Calculated by the following formula: 。 6. The implementation and usage method of idempotent message middleware consumption according to claim 1, characterized in that, In step S5, the idempotent signature cache is a distributed cache based on a Bloom filter, and the length of the Bloom filter is... and the number of hash functions Based on the expected maximum total number of messages And the expected false positive rate Dynamic configuration, satisfying the following relation: ； ； in, It represents the natural logarithm.

7. The implementation and usage method of idempotent message middleware consumption according to claim 1, characterized in that, It also includes an online dictionary learning step: every fixed time window T, all raw byte stream sequences processed within that time window are collected as training sample sets, and the sparse transformation basis is updated to an adaptive dictionary using the K singular value decomposition algorithm, so that the representation of the training sample set under the adaptive dictionary is more sparse.

8. The implementation and usage method of idempotent message middleware consumption according to claim 7, characterized in that, The objective function of the K-singular value decomposition algorithm is: ； in, For the first The original byte stream sequence of each training sample. For the adaptive dictionary to be learned, For this training sample In the dictionary Sparse coding under, This indicates the number of non-zero elements in the sparse coding. This is the preset upper limit for sparsity.

9. The implementation and usage method of idempotent message middleware consumption according to claim 1, characterized in that, The message to be consumed is a message object that conforms to the Java Message Service specification. The original byte stream sequence is obtained by calling the getBody method of the message object and then converting it into a byte array.

10. The implementation and usage method of idempotent message middleware consumption according to claim 1, characterized in that, This sparse transform basis is a discrete cosine transform basis, and its matrix elements are given by the following equation: ； in, The length of the original byte stream sequence. This represents the first element in the discrete cosine transform basis. Line 1 The elements of the column.

Images