Message filtering optimization method and system

Abstract

The invention relates to the technical field of Internet, in particular to a message filtering optimization method, which comprises the following steps that: a proxy layer containing a special bloom filter is additionally arranged at the upstream of a server message queue, and the bloom filter is configured with a preset bit data structure and at least two hash functions and has the functions of querying and updating a specified position value; calculating a to-be-processed message of a client through a hash function to obtain a corresponding target bit, judging whether the message is repeated or not according to a numerical value of the target bit, only writing a non-repeated message into a message queue, and updating the target bit as an existence identifier; complete information is backed up before message consumption, and after successful consumption, a target bit is recalculated through a hash function and is updated to a reset identification value to complete state reset; according to the method, rapid deduplication verification before message writing can be realized, server resources are prevented from being occupied by repeated messages, and the message processing efficiency and the system stability are improved.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to an optimization method and system for message filtering. Background Technology

[0002] In current internet business systems, message push systems, and other scenarios that require handling large amounts of message interactions, message middleware or queues are often used to decouple different modules and improve system flexibility and processing efficiency in order to meet high throughput and high concurrency requirements. Meanwhile, to ensure message reliability, the client will resend the message if it fails to send or is received but no response is received, ensuring that the message is successfully processed.

[0003] However, in practical applications, network instability or other anomalies can lead to a large number of duplicate messages. Upon receiving a message, the server typically writes it directly to a message queue for processing by a dedicated consumer thread. If a message already exists in the queue but has not yet been processed, the continuous writing of duplicate messages can severely consume the server's computing and storage resources, not only reducing message processing efficiency but also potentially affecting the stability and response speed of the entire system.

[0004] Currently, there are limited ways to handle message duplication. Some solutions perform deduplication verification during the message consumption stage, but this method cannot avoid the waste of resources caused by writing duplicate messages to the queue. Another solution uses a simple cache to record the received message identifier, but in high-concurrency scenarios, the query and update efficiency of the cache is low, and it is difficult to balance the relationship between memory usage and deduplication accuracy.

[0005] Therefore, there is an urgent need for an efficient and low-cost message filtering scheme that can achieve accurate deduplication before messages are written to the queue, in order to solve the problems existing in the current technology. Summary of the Invention

[0006] To overcome the problems of low message processing efficiency and resource waste caused by message duplication processing methods in high-throughput and high-concurrency scenarios in existing technologies, the purpose of this invention is to provide an optimized message filtering method and system that can achieve fast deduplication verification before message writing, avoid duplicate messages occupying server resources, and improve message processing efficiency and system stability.

[0007] This invention is implemented using the following scheme: An optimization method for message filtering, comprising the following steps: Step 1: Add a Bloom filter proxy layer before the server message queue. All messages sent by clients are first filtered by the Bloom filter proxy layer. The Bloom filter is a dedicated filtering component that supports data updates. Its data structure is configured with a preset length and has methods for querying and updating any bit value. Step 2: Configure at least two hash functions. Each hash function processes the message through different calculation logic. Each hash function can output a target bit that matches the number of bits in the Bloom filter data structure. That is, the result obtained by each hash function after processing the message is moduloed by the number of bits in the Bloom filter data structure to generate a unique target bit. Step 3: After receiving the client message, the proxy layer calculates the corresponding target bits using the hash functions in Step 2, and calls the query method of the Bloom filter to obtain the values ​​of the target bits. Step 4: If the values ​​of all target bits are equal to the preset values, the current message is determined to be a duplicate message, and the Bloom filter proxy layer abandons writing the current message to the server message queue; if at least one of the values ​​of several target bits is not equal to the preset value, the current message is determined to be a non-duplicate message, the proxy layer writes the current message to the server message queue, and updates the values ​​of several target bits to the preset values ​​through the update method of the Bloom filter. Step 5: The server-side message consumption thread reads messages from the server-side message queue and performs consumption operations. Before starting the consumption operation, it backs up and stores the complete information of the current message to form message backup data. Step 6: After the consumer thread successfully consumes the current message, it calls the message backup data, recalculates the corresponding target bits through the hash functions configured in Step 2, and calls the update method of the Bloom filter to reset the values ​​of the three target bits to the reset flag value.

[0008] Furthermore, the preset length is 64 bits or 128 bits.

[0009] Furthermore, each hash function in step 2 processes the message through different calculation logic, specifically including: at least one function that directly performs hash operation on the original message string, and one function that performs hash operation on the message string after encryption; the encryption operation includes MD5 encryption, SHA-1 encryption, and SHA-256 encryption.

[0010] Furthermore, step 2 includes at least one of the following combinations: Configure three hash functions to process the message respectively: Function 1: Perform a hash calculation on the message string, obtain the hash value, and then take the remainder after dividing by 128 to get the first target bit; Function 2: Perform MD5 encryption on the message string, perform hash calculation on the encryption result again, and take the remainder of the hash value by 128 to obtain the second target bit; Function 3: After concatenating the message string, obtain the overall Unicode encoding, and take the remainder of the encoding value as 128 to obtain the third target bit.

[0011] Furthermore, the preset value is 1, the reset flag value is 0, and when the Bloom filter data structure is initialized, the values ​​at all positions are set to 0 by default to ensure that non-duplicate messages can pass through the filter normally in the initial state.

[0012] An optimized message filtering system includes: a proxy layer deployment module, a hash function configuration module, a message filtering determination module, a message writing control module, a message backup module, and a filtering state reset module. The proxy layer deployment module is used to add a Bloom filter proxy layer before the server message queue. All messages sent by clients are first filtered by the Bloom filter proxy layer. The Bloom filter is a dedicated filtering component that supports data updates. Its data structure is configured with a preset length and has methods for querying and updating any bit value. The hash function configuration module is used to configure at least two hash functions. Each hash function processes the message through different calculation logic. Each hash function can output a target bit that matches the number of bits in the Bloom filter data structure. That is, the result obtained by each hash function after processing the message is moduloed by the number of bits in the Bloom filter data structure to generate a unique target bit. The message filtering and determination module is used by the proxy layer to receive client messages, calculate the corresponding target bits through several hash functions in the hash function configuration module, and call the query method of the Bloom filter to obtain the values ​​on several target bits. The message writing control module is used to determine that the current message is a duplicate message if the values ​​of all target bits are equal to the preset value, and the Bloom filter proxy layer abandons writing the current message to the server message queue; if at least one of the values ​​of several target bits is not the preset value, the current message is determined to be a non-duplicate message, the proxy layer writes the current message to the server message queue, and updates the values ​​of several target bits to the preset value through the update method of the Bloom filter. The message backup module is used by the server's message consumption thread to read messages from the server's message queue and perform consumption operations. Before starting the consumption operation, it backs up and stores the complete information of the current message to form message backup data. The filter state reset module is used to call the message backup data after the consumer thread successfully consumes the current message, recalculate the corresponding target bits through several hash functions of the hash function configuration module, call the update method of the Bloom filter, and reset the values ​​of the three target bits to the reset flag value.

[0013] Furthermore, the preset length is 64 bits or 128 bits.

[0014] Furthermore, each hash function in step 2 processes the message through different calculation logic, specifically including: at least one function that directly performs hash operation on the original message string, and one function that performs hash operation on the message string after encryption; the encryption operation includes MD5 encryption, SHA-1 encryption, and SHA-256 encryption.

[0015] Furthermore, the configuration content of the hash function configuration module should include at least one of the following combinations: Configure three hash functions to process the message respectively: Function 1: Perform a hash calculation on the message string, obtain the hash value, and then take the remainder after dividing by 128 to get the first target bit; Function 2: Perform MD5 encryption on the message string, perform hash calculation on the encryption result again, and take the remainder of the hash value by 128 to obtain the second target bit; Function 3: After concatenating the message string, obtain the overall Unicode encoding, and take the remainder of the encoding value as 128 to obtain the third target bit.

[0016] Furthermore, the preset value is 1, the reset flag value is 0, and when the Bloom filter data structure is initialized, the values ​​at all positions are set to 0 by default to ensure that non-duplicate messages can pass through the filter normally in the initial state.

[0017] The beneficial effects of this invention are as follows: (1) High deduplication efficiency: The message is preprocessed before being written to the queue by the Bloom filter proxy layer. Three different hash functions are used to achieve fast location and verification. The whole filtering process is short and can adapt to the message processing needs in high-concurrency scenarios. (2) Low resource consumption: It adopts a 128-bit Bloom filter data structure, which has a very small memory consumption and significantly reduces the system's memory overhead compared with traditional cache deduplication schemes. (3) High deduplication accuracy: By combining multiple hash functions, the false judgment rate of the Bloom filter is reduced. At the same time, the updatable feature of the dedicated filter ensures that it can be reset in time after message consumption, avoiding misjudgment of subsequent legitimate messages. (4) Strong system compatibility: The proxy layer is added on the basis of the existing message queue architecture, without the need for large-scale modification of the original system modules, making it easy to deploy and integrate; (5) Improved stability: Effectively prevents duplicate messages from being written to the queue, reduces the waste of server-side storage and computing resources, improves message processing efficiency, and ensures the stable operation of the system in high-throughput scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural block diagram of the system of the present invention. Detailed Implementation

[0019] The invention will now be further described with reference to the accompanying drawings.

[0020] See Figure 1 An optimization method for message filtering, the method comprising the following steps: Step 1: Add a Bloom filter proxy layer before the server message queue. All messages sent by clients are first filtered by the Bloom filter proxy layer. The Bloom filter is a dedicated filtering component that supports data updates. Its data structure is configured with a preset length and has methods for querying and updating any bit value. Step 2: Configure at least two hash functions. Each hash function processes the message through different calculation logic. Each hash function can output a target bit that matches the number of bits in the Bloom filter data structure. That is, the result obtained by each hash function after processing the message is moduloed by the number of bits in the Bloom filter data structure to generate a unique target bit. Step 3: After receiving the client message, the proxy layer calculates the corresponding target bits using the hash functions in Step 2, and calls the query method of the Bloom filter to obtain the values ​​of the target bits. Step 4: If the values ​​of all target bits are equal to the preset values, the current message is determined to be a duplicate message, and the Bloom filter proxy layer abandons writing the current message to the server message queue; if at least one of the values ​​of several target bits is not equal to the preset value, the current message is determined to be a non-duplicate message, the proxy layer writes the current message to the server message queue, and updates the values ​​of several target bits to the preset values ​​through the update method of the Bloom filter. Step 5: The server-side message consumption thread reads messages from the server-side message queue and performs consumption operations. Before starting the consumption operation, it backs up and stores the complete information of the current message to form message backup data. Step 6: After the consumer thread successfully consumes the current message, it calls the message backup data, recalculates the corresponding target bits through the hash functions configured in Step 2, and calls the update method of the Bloom filter to reset the values ​​of the three target bits to the reset flag value.

[0021] The present invention will be further described below with reference to a specific embodiment: An optimization method for message filtering, the method comprising the following steps: Step 1. Add a special Bloom filter proxy layer before the server message queue. All messages sent by clients will first be filtered and processed by this proxy layer.

[0022] Step 2. Set the Bloom filter to a 128-bit length. Simultaneously, set multiple hash functions for the filter, namely: Function 1: Hash the message string to get a number, and then take the remainder when divided by 128.

[0023] Function 2: After performing MD5 hash on the message string, hash it again and take the remainder when divided by 128.

[0024] Function 3: Concatenates the strings and obtains the overall Unicode encoding, then takes the remainder when divided by 128.

[0025] Step 3. Note: This Bloom filter needs to be developed separately and cannot use general Bloom filter functionality (because general Bloom filter functionality does not allow modification of its own data). Set up a 128-bit data structure and provide at least two methods: one for querying the value of a specific bit and one for updating the value of a specific bit.

[0026] Step 4. Each message needs to be calculated using the three functions in Step 2 to obtain the value of these three bits in the filter. When the value of these three bits is 1, it means that the message is already duplicated, and the message is discarded. Otherwise, as long as any bit is not 1, the message is continued to be written to the message queue.

[0027] Step 5. After the message is written to the queue, the consumer thread begins message consumption. Before consumption, a complete copy of the message information needs to be made in advance. After consuming a message, using the backed-up complete message information, the corresponding number of bits needs to be calculated using three functions according to the calculation method in Step 2. Then, the method in the filter to update the value of a certain bit is called to set the value of that bit to 0.

[0028] Step 6. By using the above method, we can reuse the function of the Bloom filter with a very small memory space. At the same time, we can ensure that messages are unique and non-repeating during message consumption, which can reduce the server resources occupied by a large number of duplicate messages.

[0029] See Figure 2 An optimized message filtering system, the system comprising: a proxy layer deployment module, a hash function configuration module, a message filtering determination module, a message writing control module, a message backup module, and a filtering status reset module; The proxy layer deployment module is used to add a Bloom filter proxy layer before the server message queue. All messages sent by clients are first filtered by the Bloom filter proxy layer. The Bloom filter is a dedicated filtering component that supports data updates. Its data structure is configured with a preset length and has methods for querying and updating any bit value. The hash function configuration module is used to configure at least two hash functions. Each hash function processes the message through different calculation logic. Each hash function can output a target bit that matches the number of bits in the Bloom filter data structure. That is, the result obtained by each hash function after processing the message is moduloed by the number of bits in the Bloom filter data structure to generate a unique target bit. The message filtering and determination module is used by the proxy layer to receive client messages, calculate the corresponding target bits through several hash functions in the hash function configuration module, and call the query method of the Bloom filter to obtain the values ​​on several target bits. The message writing control module is used to determine that the current message is a duplicate message if the values ​​of all target bits are equal to the preset value, and the Bloom filter proxy layer abandons writing the current message to the server message queue; if at least one of the values ​​of several target bits is not the preset value, the current message is determined to be a non-duplicate message, the proxy layer writes the current message to the server message queue, and updates the values ​​of several target bits to the preset value through the update method of the Bloom filter. The message backup module is used by the server's message consumption thread to read messages from the server's message queue and perform consumption operations. Before starting the consumption operation, it backs up and stores the complete information of the current message to form message backup data. The filter state reset module is used to call the message backup data after the consumer thread successfully consumes the current message, recalculate the corresponding target bits through several hash functions of the hash function configuration module, call the update method of the Bloom filter, and reset the values ​​of the three target bits to the reset flag value.

[0030] In one embodiment of the present invention, the preset length is 64 bits or 128 bits.

[0031] In one embodiment of the present invention, each hash function in step 2 processes the message through different calculation logic, specifically including: at least one function that directly performs hash operation on the original message string, and one function that performs hash operation on the message string after encryption; the encryption operation includes MD5 encryption, SHA-1 encryption, and SHA-256 encryption.

[0032] In one embodiment of the present invention, the configuration content of the hash function configuration module includes at least one of the following combinations: Configure three hash functions to process the message respectively: Function 1: Perform a hash calculation on the message string, obtain the hash value, and then take the remainder after dividing by 128 to get the first target bit; Function 2: Perform MD5 encryption on the message string, perform hash calculation on the encryption result again, and take the remainder of the hash value by 128 to obtain the second target bit; Function 3: After concatenating the message string, obtain the overall Unicode encoding, and take the remainder of the encoding value as 128 to obtain the third target bit.

[0033] In one embodiment of the present invention, the preset value is 1, the reset flag value is 0, and when the Bloom filter data structure is initialized, the values ​​at all positions are set to 0 by default to ensure that non-duplicate messages can pass through the filter normally in the initial state.

[0034] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. An optimization method for message filtering, characterized in that, The method steps are as follows: Step 1: Add a Bloom filter proxy layer before the server message queue. All messages sent by clients are first filtered by the Bloom filter proxy layer. The Bloom filter is a dedicated filtering component that supports data updates. Its data structure is configured with a preset length and has methods for querying and updating any bit value. Step 2: Configure at least two hash functions. Each hash function processes the message through different calculation logic. Each hash function can output a target bit that matches the number of bits in the Bloom filter data structure. That is, the result obtained by each hash function after processing the message is moduloed by the number of bits in the Bloom filter data structure to generate a unique target bit. Step 3: After receiving the client message, the proxy layer calculates the corresponding target bits using the hash functions in Step 2, and calls the query method of the Bloom filter to obtain the values ​​of the target bits. Step 4: If the values ​​of all target bits are equal to the preset values, the current message is determined to be a duplicate message, and the Bloom filter proxy layer abandons writing the current message to the server message queue; if at least one of the values ​​of several target bits is not equal to the preset value, the current message is determined to be a non-duplicate message, the proxy layer writes the current message to the server message queue, and updates the values ​​of several target bits to the preset values ​​through the update method of the Bloom filter. Step 5: The server-side message consumption thread reads messages from the server-side message queue and performs consumption operations. Before starting the consumption operation, it backs up and stores the complete information of the current message to form message backup data. Step 6: After the consumer thread successfully consumes the current message, it calls the message backup data, recalculates the corresponding target bits through the hash functions configured in Step 2, and calls the update method of the Bloom filter to reset the values ​​of the three target bits to the reset flag value.

2. The message filtering optimization method according to claim 1, characterized in that, The preset length is 64 bits or 128 bits.

3. The message filtering optimization method according to claim 1, characterized in that, In step 2, each hash function processes the message through different calculation logics, specifically including: at least one function that directly performs hash operation on the original message string, and one function that performs hash operation on the message string after encryption; the encryption operation includes MD5 encryption, SHA-1 encryption, and SHA-256 encryption.

4. The message filtering optimization method according to claim 1, characterized in that, Step 2 includes at least one of the following combinations: Configure three hash functions to process the message respectively: Function 1: Perform a hash calculation on the message string, obtain the hash value, and then take the remainder when divided by 128 to get the first target bit; Function 2: Perform MD5 encryption on the message string, hash the encryption result again, and take the remainder after dividing the hash value by 128 to obtain the second target bit; Function 3: After concatenating the message string, obtain the overall Unicode encoding, and take the remainder of the encoding value as 128 to obtain the third target bit.

5. The message filtering optimization method according to claim 1, characterized in that, The preset value is 1, the reset flag value is 0, and when the Bloom filter data structure is initialized, the values ​​at all positions are set to 0 by default to ensure that non-duplicate messages can pass through the filter normally in the initial state.

6. An optimized message filtering system, characterized in that, The system includes: a proxy layer deployment module, a hash function configuration module, a message filtering determination module, a message writing control module, a message backup module, and a filter status reset module; The proxy layer deployment module is used to add a Bloom filter proxy layer before the server message queue. All messages sent by clients are first filtered by the Bloom filter proxy layer. The Bloom filter is a dedicated filtering component that supports data updates. Its data structure is configured with a preset length and has methods for querying and updating any bit value. The hash function configuration module is used to configure at least two hash functions. Each hash function processes the message through different calculation logic. Each hash function can output a target bit that matches the number of bits in the Bloom filter data structure. That is, the result obtained by each hash function after processing the message is moduloed by the number of bits in the Bloom filter data structure to generate a unique target bit. The message filtering and determination module is used by the proxy layer to receive client messages, calculate the corresponding target bits through several hash functions in the hash function configuration module, and call the query method of the Bloom filter to obtain the values ​​on several target bits. The message writing control module is used to determine that the current message is a duplicate message if the values ​​of all target bits are equal to the preset value, and the Bloom filter proxy layer abandons writing the current message to the server message queue; if at least one of the values ​​of several target bits is not the preset value, the current message is determined to be a non-duplicate message, the proxy layer writes the current message to the server message queue, and updates the values ​​of several target bits to the preset value through the update method of the Bloom filter. The message backup module is used by the server's message consumption thread to read messages from the server's message queue and perform consumption operations. Before starting the consumption operation, it backs up and stores the complete information of the current message to form message backup data. The filter state reset module is used to call the message backup data after the consumer thread successfully consumes the current message, recalculate the corresponding target bits through several hash functions of the hash function configuration module, call the update method of the Bloom filter, and reset the values ​​of the three target bits to the reset flag value.

7. The message filtering optimization system according to claim 6, characterized in that, The preset length is 64 bits or 128 bits.

8. The message filtering optimization system according to claim 6, characterized in that, In step 2, each hash function processes the message through different calculation logics, specifically including: at least one function that directly performs hash operation on the original message string, and one function that performs hash operation on the message string after encryption; the encryption operation includes MD5 encryption, SHA-1 encryption, and SHA-256 encryption.

9. The message filtering optimization system according to claim 6, characterized in that, The configuration content of the hash function configuration module must include at least one of the following combinations: Configure three hash functions to process the message respectively: Function 1: Perform a hash calculation on the message string, obtain the hash value, and then take the remainder when divided by 128 to get the first target bit; Function 2: Perform MD5 encryption on the message string, hash the encryption result again, and take the remainder after dividing the hash value by 128 to obtain the second target bit; Function 3: After concatenating the message string, obtain the overall Unicode encoding, and take the remainder of the encoding value as 128 to obtain the third target bit.

10. The message filtering optimization system according to claim 6, characterized in that, The preset value is 1, the reset flag value is 0, and when the Bloom filter data structure is initialized, the values ​​at all positions are set to 0 by default to ensure that non-duplicate messages can pass through the filter normally in the initial state.

Images