Government affair oriented message queue full-link dynamic optimization method and system

Abstract

The application discloses a message queue full-link dynamic optimization method and system for government affairs scenes, belongs to the technical field of information processing, and aims to solve the technical problems of low message queue transmission efficiency, high sensitive data security risk, insufficient storage redundancy and processing reliability in government affairs scenes. The method comprises the following steps: selecting a compression strategy of government messages according to the message type, and determining the routing path of the government messages; for lightweight messages, storing the ciphertext subject and metadata corresponding to the lightweight messages into a message queue; for large messages, storing the ciphertext subject corresponding to the large messages into object storage, and storing the encryption path and metadata corresponding to the large messages into the message queue as index messages; for lightweight messages, performing decryption and verification operations on the ciphertext subject based on the metadata, and for large messages, pulling the corresponding ciphertext subject from the object storage based on the access path, and performing verification and decryption operations on the ciphertext subject based on the metadata.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, in particular to a message queue full-link dynamic optimization method and system for government affairs scenarios. BACKGROUND

[0002] In the government affairs scenario, the message queue is the core of cross-department data flow (such as approval files, livelihood service notifications, and emergency command information), and needs to process multiple types of messages such as text (structured data) and binary (attachments and other unstructured data). However, the existing solutions (including general message queues and object storage coordination solutions) have the following key problems:

[0003] 1. Lack of dynamic adaptation capability on the producer side: The existing general solution generally uses fixed thresholds to divide "queue transmission" and "OSS storage", without considering the bandwidth fluctuation of government links (such as the decrease of cross-level link bandwidth during peak hours): when the network is smooth, the OSS is still frequently stored (increasing the interaction cost), and when the network is congested, the large message blocks the queue (causing delay in approval); and there is no message preprocessing mechanism, the volume of text messages is not fully compressed, and the compression time of binary messages is too long, wasting government terminal resources;

[0004] 2. Index redundancy and insufficient security compliance in the storage layer: In the existing solution, after large messages are stored in the OSS, the queue storage easily leaks sensitive information (such as storage path), and the verification mechanism uses ordinary hash (such as MD5), which has the risk of being tampered with, and does not meet the requirements of "sensitive data encryption protection in the whole process" in the government affairs scenario;

[0005] 3. Poor adaptation of processing efficiency and reliability on the consumer side: The existing solution does not distinguish between light message (such as notification) and large message (such as attachment) processing channels, and when large messages are pulled, light messages are blocked (such as delay in livelihood service notification); and there is no fault tolerance mechanism, and after the storage object fails to be pulled, it needs to be manually triggered again, which easily leads to interruption of the cross-department approval process, and cannot meet the core demand of "continuous and reliable" in the government affairs scenario.

[0006] The low transmission efficiency of the message queue, the high security risk of sensitive data, the storage redundancy, and the insufficient processing reliability in the government affairs scenario are technical problems that need to be solved. SUMMARY

[0007] The technical task of the present application is to provide a message queue full-link dynamic optimization method and system for government affairs scenarios to solve the technical problems of low transmission efficiency of the message queue, high security risk of sensitive data, storage redundancy, and insufficient processing reliability in the government affairs scenario.

[0008] In a first aspect, the present application provides a message queue full-link dynamic optimization method for government affairs scenarios, comprising the following steps:

[0009] Producer dynamic preprocessing: Select the compression strategy for government messages based on message type, calculate the dynamic threshold based on real-time bandwidth and sensitivity level, compare the dynamic threshold with the volume of the compressed government message, and determine the routing path of the government message based on the comparison result. If the volume of the compressed government message is less than or equal to the dynamic threshold, the government message is determined to be a lightweight message, and the routing path of the lightweight message is routed to the message queue. If the volume of the compressed government message is greater than the dynamic threshold, the government message is determined to be a large message, and the routing path of the large message is routed to object storage.

[0010] Message storage: For lightweight messages, the compressed government messages are encrypted using national cryptographic standards to generate a ciphertext body. The ciphertext body and metadata corresponding to the lightweight message are stored in the message queue. For large messages, the compressed government messages are encrypted using national cryptographic standards to generate a ciphertext body. The ciphertext body corresponding to the large message is stored in object storage. The access path of the ciphertext body corresponding to the large message is encrypted using national cryptographic standards to generate an encrypted path. The encrypted path and metadata corresponding to the large message are stored as an index message in the message queue. The metadata includes data fingerprint, message type, compression algorithm identifier, and encryption strategy identifier. The data fingerprint is used for the consumer to verify the integrity of the government message. The encryption strategy for the compressed government messages and access paths corresponding to the large message is the same.

[0011] Consumer-side fault tolerance: For lightweight messages, the encrypted body and metadata are retrieved from the message queue, and decryption and verification operations are performed on the encrypted body based on the metadata. For large messages, the index message is retrieved from the message queue, the encrypted path in the index message is decrypted to obtain the access path of the corresponding encrypted body, the corresponding encrypted body is pulled from the object storage based on the access path, and verification and decryption operations are performed on the encrypted body based on the metadata.

[0012] As a preferred option, the message types include two types: text and binary. Text messages include approval instructions and notification texts, while binary messages include attachments and scanned copies.

[0013] For text-based government messages, high-density compression algorithms are used for compression; for binary-based government messages, fast compression modes are used for compression.

[0014] As a preferred option, the formula for calculating the dynamic threshold is as follows:

[0015] Dynamic threshold = Base threshold × (Real-time bandwidth adaptation coefficient × Sensitivity correction weight).

[0016] Among them, the real-time bandwidth adaptation coefficient is dynamically adjusted according to the real-time bandwidth of the government link. When the bandwidth is sufficient, the threshold is increased to reduce the transfer, and when the bandwidth is congested, the threshold is reduced to ensure the smooth flow of the queue.

[0017] The sensitivity correction weight is a correction coefficient configured based on the sensitivity level of government messages. Highly sensitive messages are given priority in ensuring transmission reliability, while lowly sensitive messages are given priority in releasing bandwidth resources.

[0018] Preferably, the data fingerprint is generated by encrypting the ciphertext body based on the national cryptographic encryption algorithm. After the consumer pulls the ciphertext body, it verifies the data integrity by checking the data fingerprint.

[0019] As a preferred approach, fault tolerance handling on the consumer side includes the following operations:

[0020] For lightweight messages, after the consumer obtains the corresponding encrypted body and metadata from the message queue, it decrypts the corresponding encrypted body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant encrypted body through the data fingerprint in the metadata.

[0021] For large messages, the consumer retrieves the corresponding encrypted path and metadata from the message queue, decrypts the encrypted path based on the encryption policy identifier in the metadata to obtain the access path for the corresponding ciphertext body, pulls the ciphertext body corresponding to the large message from the object storage based on the access path, decrypts the relevant ciphertext body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant ciphertext body through the data fingerprint in the metadata. After the verification is successful, business processing is performed and the message ACK status is updated.

[0022] Secondly, the present invention provides a message queue end-to-end dynamic optimization system for government affairs scenarios, including a producer dynamic pre-processing module, a message storage module, and a consumer-end fault-tolerant processing module.

[0023] The producer dynamic preprocessing module performs the following: selects the compression strategy for government messages based on message type, calculates a dynamic threshold based on real-time bandwidth and sensitivity level, compares the dynamic threshold with the volume of the compressed government message, and determines the routing path of the government message based on the comparison result. If the volume of the compressed government message is less than or equal to the dynamic threshold, the government message is determined to be a lightweight message, and the routing path of the lightweight message is routed to the message queue. If the volume of the compressed government message is greater than the dynamic threshold, the government message is determined to be a large message, and the routing path of the large message is routed to object storage.

[0024] The message storage module performs the following operations: For lightweight messages, it encrypts the compressed government messages using national cryptographic standards to generate a ciphertext body, and stores the ciphertext body and metadata corresponding to the lightweight message in the message queue; for large messages, it encrypts the compressed government messages using national cryptographic standards to generate a ciphertext body, stores the ciphertext body corresponding to the large message in object storage, and encrypts the access path of the ciphertext body corresponding to the large message using national cryptographic standards to generate an encrypted path, and stores the encrypted path and metadata corresponding to the large message as an index message in the message queue. The metadata includes data fingerprint, message type, compression algorithm identifier, and encryption strategy identifier. The data fingerprint is used for the consumer to verify the integrity of the government message. The encryption strategy for the compressed government messages and access paths corresponding to the large messages is the same.

[0025] The consumer-side fault tolerance processing module performs the following operations: For lightweight messages, it retrieves the encrypted body and metadata from the message queue, and performs decryption and verification operations on the encrypted body based on the metadata. For large messages, it retrieves the index message from the message queue, decrypts the encrypted path in the index message to obtain the access path of the corresponding encrypted body, pulls the corresponding encrypted body from the object storage based on the access path, and performs verification and decryption operations on the encrypted body based on the metadata.

[0026] As a preferred option, the message types include two types: text and binary. Text messages include approval instructions and notification texts, while binary messages include attachments and scanned copies.

[0027] For text-based government messages, high-density compression algorithms are used for compression; for binary-based government messages, fast compression modes are used for compression.

[0028] As a preferred option, the formula for calculating the dynamic threshold is as follows:

[0029] Dynamic threshold = Base threshold × (Real-time bandwidth adaptation coefficient × Sensitivity correction weight).

[0030] Among them, the real-time bandwidth adaptation coefficient is dynamically adjusted according to the real-time bandwidth of the government link. When the bandwidth is sufficient, the threshold is increased to reduce the transfer, and when the bandwidth is congested, the threshold is reduced to ensure the smooth flow of the queue.

[0031] The sensitivity correction weight is a correction coefficient configured based on the sensitivity level of government messages. Highly sensitive messages are given priority in ensuring transmission reliability, while lowly sensitive messages are given priority in releasing bandwidth resources.

[0032] Preferably, the data fingerprint is generated by encrypting the ciphertext body based on the national cryptographic encryption algorithm. After the consumer pulls the ciphertext body, it verifies the data integrity by checking the data fingerprint.

[0033] As a preferred option, the consumer-side fault-tolerant processing module is used to perform the following operations:

[0034] For lightweight messages, after the consumer obtains the corresponding encrypted body and metadata from the message queue, it decrypts the corresponding encrypted body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant encrypted body through the data fingerprint in the metadata.

[0035] For large messages, the consumer retrieves the corresponding encrypted path and metadata from the message queue, decrypts the encrypted path based on the encryption policy identifier in the metadata to obtain the access path for the corresponding ciphertext body, pulls the ciphertext body corresponding to the large message from the object storage based on the access path, decrypts the relevant ciphertext body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant ciphertext body through the data fingerprint in the metadata. After the verification is successful, business processing is performed and the message ACK status is updated.

[0036] The message queue end-to-end dynamic optimization method and system for government affairs scenarios of the present invention have the following advantages:

[0037] 1. Transmission efficiency adapted to the characteristics of government links: Adaptive compression reduces the size of text messages and the compression time of binary messages, reducing the resource consumption of government terminals; Compared with the existing fixed threshold scheme, dynamic threshold reduces the frequency of OSS transfer when the network is smooth and the queue blocking rate decreases when congested, significantly shortening the transmission delay of core government messages (such as approval notices).

[0038] 2. Enhanced security and simplification of storage layer: Lightweight indexes eliminate redundancy, metadata size is reduced, national cryptographic SM2 encryption (subject / path) + SM3 fingerprint verification, and SM4 encryption of OSS storage files meet the requirements of enhanced protection measures for sensitive government data, eliminating the risk of leakage and tampering;

[0039] 3. Consumer-side reliability adapts to government process requirements: The dual-mode processing mechanism ensures that lightweight message processing is not affected by large messages, improving the real-time performance of government services (such as public service notification responses); asynchronous retrieval + automatic retry + fingerprint verification improve the success rate, security and efficiency of large message processing, ensuring the continuity of government processes such as cross-departmental approvals. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] The invention will be further described below with reference to the accompanying drawings.

[0042] Figure 1This is a flowchart of a message queue end-to-end dynamic optimization method for government affairs scenarios, as shown in Example 1.

[0043] Figure 2 This is a flowchart of the dynamic preprocessing process on the producer side in a message queue end-to-end dynamic optimization method for government affairs scenarios, as shown in Example 1.

[0044] Figure 3 This is a flowchart of the intelligent fault-tolerant process on the consumer side in a message queue end-to-end dynamic optimization method for government affairs scenarios, as described in Example 1. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0046] This invention provides a method and system for dynamic optimization of the entire message queue chain in government affairs scenarios, which addresses the technical problems of low message queue transmission efficiency, high security risks of sensitive data, redundant storage, and insufficient processing reliability in government affairs scenarios.

[0047] Example 1: The present invention provides a dynamic optimization method for the entire message queue in government affairs scenarios, which includes three steps: dynamic preprocessing by the producer, message storage, and fault-tolerant processing by the consumer.

[0048] Step S100 Producer Dynamic Preprocessing: Select the compression strategy for government messages based on message type, calculate the dynamic threshold based on real-time bandwidth and sensitivity level, compare the dynamic threshold with the volume of the compressed government message, and determine the routing path of the government message based on the comparison result. If the volume of the compressed government message is less than or equal to the dynamic threshold, the government message is determined to be a lightweight message, and the routing path of the lightweight message is routed to the message queue. If the volume of the compressed government message is greater than the dynamic threshold, the government message is determined to be a large message, and the routing path of the large message is routed to object storage.

[0049] The message types include two categories: text and binary. Text messages include approval instructions and notification texts, while binary messages include attachments and scanned copies. Text-based government messages (such as approval instructions and notification texts) are compressed using high-density compression algorithms (such as the improved version of LZ77) to prioritize size reduction and minimize data transfer. Binary government messages (such as attachments and scanned copies) are compressed using fast compression modes (such as the Snappy algorithm) to balance compression ratio and processing time.

[0050] The formula for calculating the dynamic threshold is as follows:

[0051] Dynamic threshold = Base threshold × (Real-time bandwidth adaptation coefficient × Sensitivity correction weight).

[0052] Among them, the real-time bandwidth adaptation coefficient is dynamically adjusted according to the real-time bandwidth of the government link (not a fixed rule, but can be adaptively calculated by the link detection module during actual deployment). When the bandwidth is sufficient, the threshold is increased to reduce the transfer, and when the bandwidth is congested, the threshold is reduced to ensure the smooth flow of the queue.

[0053] The sensitivity correction weight is a correction coefficient configured based on the sensitivity level of government messages (such as "Level 1 / Level 2 / Level 3 / Level 4"). Highly sensitive messages are given priority in ensuring transmission reliability, while lowly sensitive messages are given priority in releasing bandwidth resources.

[0054] In this embodiment, the producer actively reduces message volume through "type-adaptive compression," dynamically adjusts routing priority based on the sensitivity of government messages, and adapts to link fluctuations and data security requirements through "bandwidth-sensitivity dual-perception thresholds." This addresses the shortcomings of existing solutions such as "rigid fixed thresholds," "lack of security classification," and "lack of preprocessing," and accurately adapts to the "efficiency and compliance are equally important" characteristics of government scenarios, as shown in Figure 2.

[0055] Step S200 Message Storage: For lightweight messages, the compressed government message is encrypted using national cryptographic standards to generate a ciphertext body. The ciphertext body and metadata corresponding to the lightweight message are stored in the message queue. For large messages, the compressed government message is encrypted using national cryptographic standards to generate a ciphertext body. The ciphertext body corresponding to the large message is stored in object storage. The access path of the ciphertext body corresponding to the large message is encrypted using national cryptographic standards to generate an encrypted path. The encrypted path and metadata corresponding to the large message are stored as an index message in the message queue. The metadata includes data fingerprint, message type, compression algorithm identifier, and encryption strategy identifier. The data fingerprint is used for the consumer to verify the integrity of the government message. The encryption strategy for the compressed government message and the access path corresponding to the large message is the same.

[0056] The data fingerprint is generated by encrypting the ciphertext body based on the national cryptographic encryption algorithm. After the consumer pulls the ciphertext body, it verifies the data integrity by checking the data fingerprint.

[0057] In this embodiment, lightweight messages refer to messages whose volume is ≤ a dynamic threshold, such as government notification texts and short command data. They do not need to be transferred to OSS, but the storage security within the queue must be ensured, and the metadata logic must be consistent with that of large messages.

[0058] (1) Ciphertext body: generated using the national standard SM2 encryption method, which can only be parsed by the consumer end;

[0059] (2) National Cryptographic SM3 Data Fingerprint: Generates the encrypted body. After the consumer pulls the message body, it verifies the data integrity by verifying the fingerprint to prevent the data from being tampered with during transmission.

[0060] (3) Message type marking: Mark the message text or binary type for the consumer to match the corresponding decompression algorithm. This design is not present in general solutions, which ensures the accuracy of decompression of large messages;

[0061] (4) Compression algorithm identifier: Records the compression algorithm used by the producer for large messages (such as "LZ77" or "Snappy"), and supports the consumer to automatically call the corresponding algorithm to decompress and restore, thereby improving the efficiency of large message processing;

[0062] (5) Encryption strategy mark: Marks the encryption algorithm used in the ciphertext (the default is SM2). The consumer calls the SM2 decryption process based on this mark to ensure data transmission security and achieve a unified encryption strategy across the entire link.

[0063] Large messages refer to messages whose size exceeds a dynamic threshold, such as attachments and long approval documents. These need to be transferred to OSS, where a lightweight security index is stored in the queue, and the metadata logic is unified with that of the lightweight message storage.

[0064] (1) OSS encrypted access path: generated using the national cryptographic SM2 encryption method, which can only be parsed by the consumer end, to avoid storage location leakage and protect the security of data storage location;

[0065] (2) National Cryptographic SM3 Data Fingerprint: Generated for the encrypted message body stored in OSS. After the consumer pulls the message body, it verifies the fingerprint to confirm the data integrity and prevents the data from being tampered with in the OSS storage and transmission process.

[0066] (3) Message type tag: tag the message text or binary type for the consumer to match the corresponding decompression algorithm;

[0067] (4) Compression algorithm identifier: Records the compression algorithm used by the producer for large messages (such as "LZ77" or "Snappy"), and supports the consumer to automatically call the corresponding algorithm to decompress and restore, thereby improving the efficiency of large message processing;

[0068] (5) Encryption strategy mark: Marks the encryption algorithm used in the message body (the default is SM2, which is consistent with the encryption strategy logic of lightweight messages). The consumer end calls the SM2 decryption process based on the mark to ensure data transmission security and achieve the consistency of the encryption strategy across the entire link.

[0069] Step S300 Consumer-side fault tolerance processing: For lightweight messages, retrieve the encrypted body and metadata from the message queue, and perform decryption and verification operations on the encrypted body based on the metadata. For large messages, retrieve the index message from the message queue, decrypt the encrypted path in the index message to obtain the access path of the corresponding encrypted body, pull the corresponding encrypted body from the object storage based on the access path, and perform verification and decryption operations on the encrypted body based on the metadata.

[0070] As a specific implementation of fault tolerance processing on the consumer side, this step includes the following operations:

[0071] (1) For lightweight messages, after the consumer obtains the corresponding encrypted body and metadata from the message queue, it decrypts the corresponding encrypted body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant encrypted body through the data fingerprint in the metadata.

[0072] (2) For large messages, the consumer obtains the corresponding encrypted path and metadata from the message queue, decrypts the encrypted path based on the encryption policy identifier in the metadata, obtains the access path of the corresponding ciphertext body, pulls the ciphertext body corresponding to the large message from the object storage based on the access path, decrypts the relevant ciphertext body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant ciphertext body through the data fingerprint in the metadata. After the verification is passed, business processing is performed and the message ACK status is updated.

[0073] In this embodiment, the consumer side avoids large messages blocking lightweight messages through "dual-mode traffic splitting," and ensures the continuity and reliability of government processes through optimizations such as asynchronous retrieval, automatic retries, fingerprinting, and encrypted verification. For example... Figure 3 As shown.

[0074] The specific execution process is as follows:

[0075] (1) Index reading and parsing: retrieve a lightweight index from the queue and parse the message type, compression algorithm, encryption flag, etc.;

[0076] (2) Dual-mode flow splitting: Based on the parsed message type, dual-mode flow splitting is performed. Lightweight messages are verified for confidentiality and integrity by fingerprinting with national cryptographic SM3 and encryption policy marking. Once passed, they are processed directly without OSS interaction, thus avoiding blocking. Large messages start an independent asynchronous pull channel, decrypt the OSS path by encryption policy marking, and send a request to OSS to download the corresponding object content. The object content is verified for confidentiality and integrity by fingerprinting with national cryptographic SM3 and decryption with national cryptographic SM4. Once passed, business processing is performed.

[0077] (3) Update the message ACK status after the business is completed to ensure that the business is idempotent.

[0078] The method in this embodiment provides the following mechanism:

[0079] (1) Producer-side dynamic preprocessing mechanism: optimize compression for government message types, adjust routing thresholds in conjunction with real-time bandwidth, and reduce invalid transfers and link blockages;

[0080] (2) Storage layer security and simplification design: After large messages are transferred to OSS, a lightweight index (containing only core metadata) is generated and security is ensured by national cryptographic encryption;

[0081] (3) Intelligent fault tolerance system for consumers: messages are processed by type, and large messages are retrieved and verified through asynchronous channels to ensure the continuity of government processes.

[0082] Example 2: The present invention provides a dynamic optimization system for the entire message queue in government affairs scenarios, including a producer dynamic pre-processing module, a message storage module, and a consumer-side fault-tolerant processing module.

[0083] The producer dynamic preprocessing module performs the following actions: selects a compression strategy for government messages based on message type, calculates a dynamic threshold based on real-time bandwidth and sensitivity level, compares the dynamic threshold with the volume of the compressed government message, and determines the routing path of the government message based on the comparison result. If the volume of the compressed government message is less than or equal to the dynamic threshold, the government message is determined to be a lightweight message, and the routing path for lightweight messages is to the message queue. If the volume of the compressed government message is greater than the dynamic threshold, the government message is determined to be a large message, and the routing path for large messages is to object storage.

[0084] The message types include two categories: text and binary. Text messages include approval instructions and notification texts, while binary messages include attachments and scanned copies. Text-based government messages (such as approval instructions and notification texts) are compressed using high-density compression algorithms (such as the improved version of LZ77) to prioritize size reduction and minimize data transfer. Binary government messages (such as attachments and scanned copies) are compressed using fast compression modes (such as the Snappy algorithm) to balance compression ratio and processing time.

[0085] The formula for calculating the dynamic threshold is as follows:

[0086] Dynamic threshold = Base threshold × (Real-time bandwidth adaptation coefficient × Sensitivity correction weight).

[0087] Among them, the real-time bandwidth adaptation coefficient is dynamically adjusted according to the real-time bandwidth of the government link (not a fixed rule, but can be adaptively calculated by the link detection module during actual deployment). When the bandwidth is sufficient, the threshold is increased to reduce the transfer, and when the bandwidth is congested, the threshold is reduced to ensure the smooth flow of the queue.

[0088] The sensitivity correction weight is a correction coefficient configured based on the sensitivity level of government messages (such as "Level 1 / Level 2 / Level 3 / Level 4"). Highly sensitive messages are given priority in ensuring transmission reliability, while lowly sensitive messages are given priority in releasing bandwidth resources.

[0089] In this embodiment, the producer actively reduces message volume through "type-adaptive compression," dynamically adjusts routing priority based on the sensitivity of government messages, and adapts to link fluctuations and data security requirements through "bandwidth-sensitivity dual-perception thresholds." This addresses the shortcomings of existing solutions such as "rigid fixed thresholds," "lack of security classification," and "lack of preprocessing," and precisely adapts to the "efficiency and compliance are equally important" characteristics of government scenarios.

[0090] The message storage module performs the following operations: For lightweight messages, it encrypts the compressed government message using national cryptographic standards to generate a ciphertext body, and stores the ciphertext body and metadata corresponding to the lightweight message in the message queue; for large messages, it encrypts the compressed government message using national cryptographic standards to generate a ciphertext body, stores the ciphertext body corresponding to the large message in object storage, and encrypts the access path of the ciphertext body corresponding to the large message using national cryptographic standards to generate an encrypted path. It stores the encrypted path and metadata corresponding to the large message as an index message in the message queue. The metadata includes data fingerprint, message type, compression algorithm identifier, and encryption strategy identifier. The data fingerprint is used for the consumer to verify the integrity of the government message. The encryption strategy for the compressed government message and the access path corresponding to the large message is the same.

[0091] The data fingerprint is generated by encrypting the ciphertext body based on the national cryptographic encryption algorithm. After the consumer pulls the ciphertext body, it verifies the data integrity by checking the data fingerprint.

[0092] In this embodiment, lightweight messages refer to messages whose volume is ≤ a dynamic threshold, such as government notification texts and short command data. They do not need to be transferred to OSS, but the storage security within the queue must be ensured, and the metadata logic must be consistent with that of large messages.

[0093] (1) Ciphertext body: generated using the national standard SM2 encryption method, which can only be parsed by the consumer end;

[0094] (2) National Cryptographic SM3 Data Fingerprint: Generates the encrypted body. After the consumer pulls the message body, it verifies the data integrity by verifying the fingerprint to prevent the data from being tampered with during transmission.

[0095] (3) Message type marking: Mark the message text or binary type for the consumer to match the corresponding decompression algorithm. This design is not present in general solutions, which ensures the accuracy of decompression of large messages;

[0096] (4) Compression algorithm identifier: Records the compression algorithm used by the producer for large messages (such as "LZ77" or "Snappy"), and supports the consumer to automatically call the corresponding algorithm to decompress and restore, thereby improving the efficiency of large message processing;

[0097] (5) Encryption strategy mark: Marks the encryption algorithm used in the ciphertext (the default is SM2). The consumer calls the SM2 decryption process based on this mark to ensure data transmission security and achieve a unified encryption strategy across the entire link.

[0098] Large messages refer to messages whose size exceeds a dynamic threshold, such as attachments and long approval documents. These need to be transferred to OSS, where a lightweight security index is stored in the queue, and the metadata logic is unified with that of the lightweight message storage.

[0099] (1) OSS encrypted access path: generated using the national cryptographic SM2 encryption method, which can only be parsed by the consumer end, to avoid storage location leakage and protect the security of data storage location;

[0100] (2) National Cryptographic SM3 Data Fingerprint: Generated for the encrypted message body stored in OSS. After the consumer pulls the message body, it verifies the fingerprint to confirm the data integrity and prevents the data from being tampered with in the OSS storage and transmission process.

[0101] (3) Message type tag: tag the message text or binary type for the consumer to match the corresponding decompression algorithm;

[0102] (4) Compression algorithm identifier: Records the compression algorithm used by the producer for large messages (such as "LZ77" or "Snappy"), and supports the consumer to automatically call the corresponding algorithm to decompress and restore, thereby improving the efficiency of large message processing;

[0103] (5) Encryption strategy mark: Marks the encryption algorithm used in the message body (the default is SM2, which is consistent with the encryption strategy logic of lightweight messages). The consumer end calls the SM2 decryption process based on the mark to ensure data transmission security and achieve the consistency of the encryption strategy across the entire link.

[0104] The consumer-side fault tolerance processing module performs the following operations: For lightweight messages, it retrieves the encrypted body and metadata from the message queue, and performs decryption and verification operations on the encrypted body based on the metadata. For large messages, it retrieves the index message from the message queue, decrypts the encrypted path in the index message to obtain the access path of the corresponding encrypted body, pulls the corresponding encrypted body from the object storage based on the access path, and performs verification and decryption operations on the encrypted body based on the metadata.

[0105] As a specific implementation of the consumer-side fault tolerance processing module, this module is used to perform the following operations:

[0106] (1) For lightweight messages, after the consumer obtains the corresponding encrypted body and metadata from the message queue, it decrypts the corresponding encrypted body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant encrypted body through the data fingerprint in the metadata.

[0107] (2) For large messages, the consumer obtains the corresponding encrypted path and metadata from the message queue, decrypts the encrypted path based on the encryption policy identifier in the metadata, obtains the access path of the corresponding ciphertext body, pulls the ciphertext body corresponding to the large message from the object storage based on the access path, decrypts the relevant ciphertext body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant ciphertext body through the data fingerprint in the metadata. After the verification is passed, business processing is performed and the message ACK status is updated.

[0108] In this embodiment, the consumer side avoids large messages blocking lightweight messages by using "dual-mode traffic splitting". Through optimizations such as asynchronous retrieval, automatic retry, fingerprint and encryption verification, the continuity and reliability of government processes are ensured.

[0109] The specific execution process is as follows:

[0110] (1) Index reading and parsing: retrieve a lightweight index from the queue and parse the message type, compression algorithm, encryption flag, etc.;

[0111] (2) Dual-mode flow splitting: Based on the parsed message type, dual-mode flow splitting is performed. Lightweight messages are verified for confidentiality and integrity by fingerprinting with national cryptographic SM3 and encryption policy marking. Once passed, they are processed directly without OSS interaction, thus avoiding blocking. Large messages start an independent asynchronous pull channel, decrypt the OSS path by encryption policy marking, and send a request to OSS to download the corresponding object content. The object content is verified for confidentiality and integrity by fingerprinting with national cryptographic SM3 and decryption with national cryptographic SM4. Once passed, business processing is performed.

[0112] (3) Update the message ACK status after the business is completed to ensure that the business is idempotent.

[0113] The system in this embodiment can execute the method disclosed in Embodiment 1 to achieve dynamic optimization and secure storage of the entire chain of government message queues.

[0114] The above provides a detailed description of the message queue end-to-end dynamic optimization method and system for government scenarios provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A dynamic optimization method for the entire message queue chain in government affairs scenarios, characterized in that, Includes the following steps: Producer dynamic preprocessing: Select the compression strategy for government messages based on message type, calculate the dynamic threshold based on real-time bandwidth and sensitivity level, compare the dynamic threshold with the volume of the compressed government message, and determine the routing path of the government message based on the comparison result. If the volume of the compressed government message is less than or equal to the dynamic threshold, the government message is determined to be a lightweight message, and the routing path of the lightweight message is routed to the message queue. If the volume of the compressed government message is greater than the dynamic threshold, the government message is determined to be a large message, and the routing path of the large message is routed to object storage. Message storage: For lightweight messages, the compressed government messages are encrypted using national cryptographic standards to generate a ciphertext body. The ciphertext body and metadata corresponding to the lightweight message are stored in the message queue. For large messages, the compressed government messages are encrypted using national cryptographic standards to generate a ciphertext body. The ciphertext body corresponding to the large message is stored in object storage. The access path of the ciphertext body corresponding to the large message is encrypted using national cryptographic standards to generate an encrypted path. The encrypted path and metadata corresponding to the large message are stored as an index message in the message queue. The metadata includes data fingerprint, message type, compression algorithm identifier, and encryption strategy identifier. The data fingerprint is used for the consumer to verify the integrity of the government message. The encryption strategy for the compressed government messages and access paths corresponding to the large message is the same. Consumer-side fault tolerance: For lightweight messages, the encrypted body and metadata are retrieved from the message queue, and decryption and verification operations are performed on the encrypted body based on the metadata. For large messages, the index message is retrieved from the message queue, the encrypted path in the index message is decrypted to obtain the access path of the corresponding encrypted body, the corresponding encrypted body is pulled from the object storage based on the access path, and verification and decryption operations are performed on the encrypted body based on the metadata.

2. The message queue end-to-end dynamic optimization method for government affairs scenarios according to claim 1, characterized in that, Message types include two types: text and binary. Text messages include approval instructions and notification texts, while binary messages include attachments and scanned copies. For text-based government messages, high-density compression algorithms are used for compression; for binary-based government messages, fast compression modes are used for compression.

3. The message queue end-to-end dynamic optimization method for government affairs scenarios according to claim 1, characterized in that, The formula for calculating the dynamic threshold is as follows: Dynamic threshold = Base threshold × (Real-time bandwidth adaptation coefficient × Sensitivity correction weight). Among them, the real-time bandwidth adaptation coefficient is dynamically adjusted according to the real-time bandwidth of the government link. When the bandwidth is sufficient, the threshold is increased to reduce the transfer, and when the bandwidth is congested, the threshold is reduced to ensure the smooth flow of the queue. The sensitivity correction weight is a correction coefficient configured based on the sensitivity level of government messages. Highly sensitive messages are given priority in ensuring transmission reliability, while lowly sensitive messages are given priority in releasing bandwidth resources.

4. The message queue end-to-end dynamic optimization method for government affairs scenarios according to claim 1, characterized in that, The data fingerprint is generated by encrypting the ciphertext body based on the national cryptographic encryption algorithm. After the consumer pulls the ciphertext body, it verifies the data integrity by checking the data fingerprint.

5. The message queue end-to-end dynamic optimization method for government affairs scenarios according to claim 1, characterized in that, Consumer-side fault tolerance handling includes the following operations: For lightweight messages, after the consumer obtains the corresponding encrypted body and metadata from the message queue, it decrypts the corresponding encrypted body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant encrypted body through the data fingerprint in the metadata. For large messages, the consumer retrieves the corresponding encrypted path and metadata from the message queue, decrypts the encrypted path based on the encryption policy identifier in the metadata to obtain the access path for the corresponding ciphertext body, pulls the ciphertext body corresponding to the large message from the object storage based on the access path, decrypts the relevant ciphertext body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant ciphertext body through the data fingerprint in the metadata. After the verification is successful, business processing is performed and the message ACK status is updated.

6. A message queue end-to-end dynamic optimization system for government affairs scenarios, characterized in that, This includes a producer dynamic pre-processing module, a message storage module, and a consumer-side fault tolerance processing module; The producer dynamic preprocessing module performs the following: selects the compression strategy for government messages based on message type, calculates a dynamic threshold based on real-time bandwidth and sensitivity level, compares the dynamic threshold with the volume of the compressed government message, and determines the routing path of the government message based on the comparison result. If the volume of the compressed government message is less than or equal to the dynamic threshold, the government message is determined to be a lightweight message, and the routing path of the lightweight message is routed to the message queue. If the volume of the compressed government message is greater than the dynamic threshold, the government message is determined to be a large message, and the routing path of the large message is routed to object storage. The message storage module performs the following operations: For lightweight messages, it encrypts the compressed government messages using national cryptographic standards to generate a ciphertext body, and stores the ciphertext body and metadata corresponding to the lightweight message in the message queue; for large messages, it encrypts the compressed government messages using national cryptographic standards to generate a ciphertext body, stores the ciphertext body corresponding to the large message in object storage, and encrypts the access path of the ciphertext body corresponding to the large message using national cryptographic standards to generate an encrypted path, and stores the encrypted path and metadata corresponding to the large message as an index message in the message queue. The metadata includes data fingerprint, message type, compression algorithm identifier, and encryption strategy identifier. The data fingerprint is used for the consumer to verify the integrity of the government message. The encryption strategy for the compressed government messages and access paths corresponding to the large messages is the same. The consumer-side fault tolerance processing module performs the following operations: For lightweight messages, it retrieves the encrypted body and metadata from the message queue, and performs decryption and verification operations on the encrypted body based on the metadata. For large messages, it retrieves the index message from the message queue, decrypts the encrypted path in the index message to obtain the access path of the corresponding encrypted body, pulls the corresponding encrypted body from the object storage based on the access path, and performs verification and decryption operations on the encrypted body based on the metadata.

7. The message queue end-to-end dynamic optimization system for government scenarios according to claim 6, characterized in that, Message types include two types: text and binary. Text messages include approval instructions and notification texts, while binary messages include attachments and scanned copies. For text-based government messages, high-density compression algorithms are used for compression; for binary-based government messages, fast compression modes are used for compression.

8. The message queue end-to-end dynamic optimization system for government scenarios according to claim 6, characterized in that, The formula for calculating the dynamic threshold is as follows: Dynamic threshold = Base threshold × (Real-time bandwidth adaptation coefficient × Sensitivity correction weight). Among them, the real-time bandwidth adaptation coefficient is dynamically adjusted according to the real-time bandwidth of the government link. When the bandwidth is sufficient, the threshold is increased to reduce the transfer, and when the bandwidth is congested, the threshold is reduced to ensure the smooth flow of the queue. The sensitivity correction weight is a correction coefficient configured based on the sensitivity level of government messages. Highly sensitive messages are given priority in ensuring transmission reliability, while lowly sensitive messages are given priority in releasing bandwidth resources.

9. The message queue end-to-end dynamic optimization system for government scenarios according to claim 6, characterized in that, The data fingerprint is generated by encrypting the ciphertext body based on the national cryptographic encryption algorithm. After the consumer pulls the ciphertext body, it verifies the data integrity by checking the data fingerprint.

10. The message queue end-to-end dynamic optimization system for government scenarios according to claim 6, characterized in that, The consumer-side fault tolerance module is used to perform the following operations: For lightweight messages, after the consumer obtains the corresponding encrypted body and metadata from the message queue, it decrypts the corresponding encrypted body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant encrypted body through the data fingerprint in the metadata. For large messages, the consumer retrieves the corresponding encrypted path and metadata from the message queue, decrypts the encrypted path based on the encryption policy identifier in the metadata to obtain the access path for the corresponding ciphertext body, pulls the ciphertext body corresponding to the large message from the object storage based on the access path, decrypts the relevant ciphertext body based on the encryption policy identifier in the metadata, and performs integrity verification on the relevant ciphertext body through the data fingerprint in the metadata. After the verification is successful, business processing is performed and the message ACK status is updated.

Images