Message detection method, device, computer equipment and storage medium

Abstract

The application relates to a message detection method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a to-be-detected message, and determining a target pre-filter currently used for pre-detection; processing the to-be-detected message through more than one first mapping function in the target pre-filter respectively to obtain corresponding first mapping values; querying corresponding target positions of each first mapping value in a bit array in the target pre-filter based on the bit array; when it is determined that the to-be-detected message does not appear for the first time based on element marks at the target positions, processing the to-be-detected message through a second mapping function in a re-filter to obtain a second mapping value; and when the second mapping value is queried from all historical mapping values in the re-filter, determining that the to-be-detected message is a repeated message. The method can guarantee the efficiency of message detection and improve the accuracy of message detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a message detection method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of computer technology, various communication technologies have emerged. Through various communication technologies, the sending end can send messages to the receiving end. In the communication process, the message of the sending end is sent to the server and then sent to the receiving end through the server. Generally, the server needs to detect the message to determine whether the message is a duplicate message. If the message is a duplicate message, it will not be sent to the receiving end. If the message is not a duplicate message, it will be sent to the receiving end.

[0003] In the traditional message detection method, the server performs hash processing on the message, compares the processing result with all nodes in the hash table, and determines whether the message is a duplicate message.

[0004] However, in the current message detection method, the server needs to traverse all nodes in the hash table when performing hash processing on each message, which has the problem of low detection efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a message detection method, device, computer equipment and storage medium capable of improving the message deduplication detection efficiency to solve the above technical problems.

[0006] A message detection method, the method comprising:

[0007] Obtaining a to-be-detected message, and determining a target pre-filter currently used for pre-detection;

[0008] Processing the to-be-detected message through more than one first mapping function in the target pre-filter respectively to obtain corresponding first mapping values;

[0009] Based on a bit array in the target pre-filter, querying the target positions of each first mapping value in the bit array respectively;

[0010] When it is determined that the to-be-detected message does not appear for the first time based on the element markers at the target positions, processing the to-be-detected message through a second mapping function in a re-filter to obtain a second mapping value;

[0011] When the second mapping value is queried from all historical mapping values in the re-filter, it is determined that the to-be-detected message is a duplicate message.

[0012] A message detection device, the device comprising:

[0013] determining module, configured to acquire a to-be-detected message, and determine a target pre-filter currently used for pre-detection;

[0014] a first mapping module, configured to process the to-be-detected message by a first mapping function in the target pre-filter, to obtain a corresponding first mapping value;

[0015] a querying module, configured to query a target position of each first mapping value in a bit array in the target pre-filter based on the bit array;

[0016] a second mapping module, configured to process the to-be-detected message by a second mapping function in a re-filter, to obtain a second mapping value, when it is determined that the to-be-detected message is not a first occurrence based on an element mark at the target position;

[0017] The determining module is further configured to determine that the to-be-detected message is a duplicate message when the second mapping value is queried from all historical mapping values in the re-filter.

[0018] A computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0019] A computer readable storage medium, storing a computer program, the computer program is executed by a processor to implement the steps of the above method.

[0020] The message detection method, device, computer device and storage medium, by the first mapping function in the target pre-filter, the to-be-detected message is processed to obtain the corresponding first mapping value, based on the bit array in the target pre-filter, the target position of each first mapping value in the bit array is queried, based on the element mark at each target position, the to-be-detected message can be quickly determined whether it is not a first occurrence. When the to-be-detected message is not a first occurrence, it indicates that the to-be-detected message may be a duplicate message, and then the re-filter is used for further targeted and accurate detection. Specifically, the second mapping value of the to-be-detected message is obtained by the second mapping function in the re-filter, and the second mapping value is more accurately queried from all historical mapping values in the re-filter whether it is a duplicate message. When the second mapping value is queried from the historical mapping value, it is determined that the to-be-detected message is a duplicate message. That is, the target pre-filter and the re-filter are used for coordinated processing of the to-be-detected message. First, the bit array is used to judge whether the message is duplicated, which can exclude most of the first occurrence messages. Further, the message that may be a duplicate message is further detected comprehensively, which can ensure the efficiency of message deduplication detection and improve the accuracy of message deduplication detection. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 a flowchart of a message detection method in an embodiment;

[0022] Figure 2 a flowchart of a message detection method in an embodiment;

[0023] Figure 3 a schematic diagram of a target position of a first mapping value of a message to be detected in an embodiment in a bit array;

[0024] Figure 4 a schematic diagram of coordination between a first pre-filter and a second pre-filter in an embodiment;

[0025] Figure 5 a schematic diagram of message detection in the related art;

[0026] Figure 6 a schematic diagram of message detection in the related art;

[0027] Figure 7 a flowchart of a message detection method in an embodiment;

[0028] Figure 8 a timing diagram of message detection in an embodiment;

[0029] Figure 9 a structural block diagram of a message detection apparatus in an embodiment;

[0030] Figure 10 an internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0032] The message detection method provided by the present application can be applied in an application environment as shown in FIG. 1. Figure 1 In the application environment, a sending terminal 102 communicates with a server 104 through a network, and a receiving terminal 106 communicates with the server 104 through the network. The sending terminal 102 sends a message to be detected (the message to be detected may be, for example, an email, an instant message, a short message, a multimedia message, a voice message, a video message, a file, a webpage, etc.) to the server 104 through the network. The server 104 receives the message to be detected from the sending terminal 102 through the network, and sends the message to be detected to the receiving terminal 106 through the network. The receiving terminal 106 receives the message to be detected from the server 104 through the network, and performs a message detection operation on the message to be detected. Figure 1The message 108 is sent to the server 104. The server 104 acquires the message to be detected, and determines a target pre-filter currently used for pre-detection. The message to be detected is processed by a first mapping function in the target pre-filter, to obtain a corresponding first mapping value. Based on a bit array in the target pre-filter, the server 104 queries a target position of the first mapping value in the bit array. When it is determined that the message to be detected is not a first occurrence based on an element mark at the target position, the message to be detected is processed by a second mapping function in a re-filter, to obtain a second mapping value. When the second mapping value is queried from all historical mapping values in the re-filter, it is determined that the message to be detected is a duplicate message. Further, when the server 104 determines that the message to be detected is a first occurrence, the message 108 is sent to the receiving terminal 106.

[0033] The sending terminal 102 and the receiving terminal 106 can be, but are not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server 104 can be implemented by a single server or a server cluster composed of multiple servers.

[0034] In an embodiment, as shown in FIG. 1, a message detection method is provided. The method is applied to the server 104 in FIG. 1, and includes the following steps. Figure 2 Figure 1 The method includes the following steps.

[0035] In step S202, the message to be detected is acquired, and a target pre-filter currently used for pre-detection is determined.

[0036] The form of the message to be detected is not limited, and can be one or a combination of multiple of a text message, a picture message, an audio message, and a video message. For example, the message to be detected can be a piece of text, a collection of multiple pictures, a piece of audio, or a combination of a piece of text and a piece of audio.

[0037] The target pre-filter is a filter currently used for pre-detection of the message to be detected. The target pre-filter can be a Bloom filter. The Bloom filter is a random data structure with high space efficiency, which can use a bit array to concisely represent a set, and is a fast probabilistic algorithm for judging whether an element belongs to the set.

[0038] ​The Bloom filter includes a bit array, and the length of the bit array can be set as needed. Each bit of the bit array stores an element mark, and the element mark can be set as needed. When the bit array is a binary bit array, the element mark stored in each bit can be 0 or 1. The element mark stored in each bit of the initialized bit array is 0, and when it is needed to store data in the bit array, the element mark of the storage position is set to 1. The Bloom filter can also initialize the bit array, that is, set the element mark stored in each bit of the bit array to 0.

[0039] Specifically, the server obtains the to-be-detected message sent by the sending terminal, and determines a target pre-filter currently used for pre-detection from at least two candidate pre-filters. In an embodiment, the server can obtain the working mode of each candidate pre-filter, and take the pre-filter with the working mode in the main mode as the target pre-filter. In another embodiment, the server can obtain the mark of each candidate pre-filter, and take the candidate pre-filter with the mark in the available state as the target pre-filter. In other embodiments, the server can also randomly determine one of the at least two candidate pre-filters as the target pre-filter. The manner in which the server determines the target pre-filter is not limited herein.

[0040] In an embodiment, when the server includes one pre-filter, the pre-filter is directly taken as the target pre-filter.

[0041] In step S204, the to-be-detected message is processed by more than one first mapping function in the target pre-filter respectively, and a corresponding first mapping value is obtained.

[0042] The bit array is a data structure of a hash table. The length of the bit array can be set as needed. For example, the length of the bit array can be 100 bits. In each bit of the bit array, an element mark is stored, and the element mark represents specific data. For example, the element marks of the 3rd bit, the 4th bit and the 5th bit in the bit array are all 1, when the to-be-detected message is processed by the three first mapping functions respectively, three first mapping values are obtained, and the positions of the three first mapping values in the bit array are the 3rd bit, the 4th bit and the 5th bit, the target pre-filter can confirm that the to-be-detected message is not the first time.

[0043] The first mapping function is a function in the target pre-filter, and is a function for processing the to-be-detected message to obtain a first mapping value. The first mapping function can be a hash function or other mathematical functions. The first mapping function ensures that the input data corresponds to unique output data.

[0044] The number of the first mapping functions in the target pre-filter is more than one. For example, there are three first mapping functions in the target pre-filter. The first mapping value is a value obtained by processing the message to be detected by the first mapping function. The first mapping value includes position information, and the first mapping value can be queried to the target position in the bit array.

[0045] The server inputs the message to be detected into more than one first mapping function in the target pre-filter, and each first mapping function processes the message to be detected to obtain a corresponding first mapping value. For example, the three first mapping functions in the target pre-filter are all hash functions, which are A, B and C. The server inputs the message to be detected into the first mapping functions A, B and C, respectively. The first mapping value obtained by the first mapping function A is binary value 1, and the position information is the 3rd bit in the bit array. The first mapping value obtained by the first mapping function B is binary value 1, and the position information is the 7th bit in the bit array. The first mapping value obtained by the first mapping function C is binary value 1, and the position information is the 8th bit in the bit array.

[0046] In step S206, based on the bit array in the target pre-filter, the target position of each first mapping value in the bit array is queried.

[0047] When the first mapping function is a hash function, the bit array is a data structure of a hash table. The bit array can include M bits, and each bit stores an element mark. The longer the length of the bit array, the more data the bit array stores, and the higher the accuracy of the target pre-filter in detecting the message to be detected.

[0048] The target position refers to the position of the first mapping value in the bit array. The first mapping value includes information of the target position. For example, the first mapping value includes an index, and the index indicates the position of the message to be detected in the bit array.

[0049] The server obtains the position information from the first mapping value, and queries the target position corresponding to the position information of the first mapping value in the bit array from the target pre-filter.

[0050] Figure 3 A schematic diagram for querying the target position of each first mapping value of the message to be detected in the bit array in an embodiment. The bit array 302 is a binary bit array, which includes 18 bits, and each bit stores an element mark 0 or 1. After the server successfully issues the message to be detected, the target pre-filter maps the first mapping value corresponding to the message to be detected to the bit array. For example, after the server successfully issues the messages to be detected x, y and z, the target pre-filter maps the first mapping value corresponding to the messages to be detected x, y and z to the bit array, which is used for pre-detection of subsequent messages to be detected.

[0051] When the server receives a new message w to be detected, the message w to be detected is processed by three first mapping functions in the target pre-filter, and three first mapping values corresponding to the message w to be detected are obtained. Each first mapping value includes position information, and the target positions of each first mapping value in the bit array 302 can be queried, which are the 5th position, the 14th position and the 17th position, and the element markers at the target positions are binary values 1.

[0052] From Figure 3 It can be known that although the element markers at the target positions of the 5th position, the 14th position and the 17th position are binary values 1, the element markers at the target positions of the 5th position, the 14th position and the 17th position are not mapped by a first mapping value corresponding to one message, but are mapped by first mapping values corresponding to different messages.

[0053] In step S208, when it is determined that the message to be detected is not first appeared based on the element markers at the target positions, the message to be detected is processed by a second mapping function in the re-filter to obtain a second mapping value.

[0054] The element marker is a marker of a specific element stored in the bit array. For example, the element marker can be a binary value 0 or 1, and can also be a specific identifier such as “+”, “-”, “@” and the like. The bit array includes M bits, and each bit stores an element marker.

[0055] The re-filter is a filter for re-detecting the message to be detected. The re-filter can be a hash filter. The second mapping function is a mapping function in the re-filter, which is used for mapping the message to be detected.

[0056] The second mapping function is a function in the re-filter, which is used for processing the message to be detected to obtain a second mapping value. The second mapping function can be a hash function or other mapping function. The second mapping function ensures that the input data corresponds to unique output data. When the re-filter is a hash filter, the second mapping function is a hash function.

[0057] Based on the element markers at the target positions, it can be determined whether the message to be detected is first appeared, and the first appeared message to be detected can be quickly screened out. However, based on the element markers at the target positions, it is determined that the message to be detected is not first appeared, and there may be a problem of false detection. Therefore, when it is determined that the message to be detected is not first appeared based on the element markers at the target positions, the message to be detected needs to be re-detected by the re-filter to more accurately determine whether the message to be detected is not first appeared.

[0058] As Figure 3As shown, when the first mapping value corresponding to the to-be-detected message is mapped to the element markers at the target positions in the bit array, and the element markers at the target positions are all 1, it is considered that the to-be-detected message is not the first occurrence. For example, the first mapping values corresponding to the historical message x are mapped to the target positions of the 2nd bit, the 6th bit and the 14th bit in the bit array, and the element markers at the 2nd bit, the 6th bit and the 14th bit in the bit array are set to 1. When the server receives a new to-be-detected message, the first mapping value corresponding to the new to-be-detected message is also mapped to the target positions of the 2nd bit, the 6th bit and the 14th bit in the bit array, and it is considered that the new to-be-detected message is not the first occurrence, i.e., the new to-be-detected message is x.

[0059] When the server receives a new to-be-detected message w, the first mapping value corresponding to the to-be-detected message w is mapped to the target positions of the 5th bit, the 14th bit and the 17th bit in the bit array. Although the element markers at the target positions of the 5th bit, the 14th bit and the 17th bit in the bit array are all 1, the element markers 1 at the target positions are not mapped by the first mapping values corresponding to one historical message, but are mapped by the first mapping values corresponding to different historical messages. The new to-be-detected message w is actually the first occurrence. Therefore, when it is determined that the to-be-detected message is not the first occurrence based on the element markers at the target positions, the to-be-detected message needs to be detected again by the re-filter, which can avoid the false detection of the pre-filter and more accurately determine whether the to-be-detected message is not the first occurrence.

[0060] In step S210, when the second mapping value is queried from all the historical mapping values in the re-filter, it is determined that the to-be-detected message is a repeated message.

[0061] It can be understood that after the server receives a message, the mapping value of the message is stored in the data storage structure of the re-filter, so as to detect the subsequent messages and determine whether the subsequent messages are repeated messages.

[0062] Specifically, in the data storage structure of the re-filter, the historical mapping values are stored. The server compares the second mapping value with all the historical mapping values in the re-filter, and when the historical mapping value in the re-filter is the second mapping value, it is determined that the to-be-detected message is a repeated message.

[0063] The message detection method processes the to-be-detected message through more than one first mapping function in the target pre-filter respectively to obtain corresponding first mapping values, queries the corresponding target positions of each first mapping value in the bit array based on the bit array in the target pre-filter, and determines whether the to-be-detected message is a first occurrence based on the element markers at the target positions. When the to-be-detected message is a non-first occurrence, it indicates that the to-be-detected message may be a repeated message, and further targeted and accurate detection is performed through the re-filter. Specifically, the second mapping value of the to-be-detected message can be obtained by processing the to-be-detected message through the second mapping function in the re-filter, and it can be more accurately queried from all historical mapping values in the re-filter whether the second mapping value is a repeated message. When the second mapping value is queried from the historical mapping values, it is determined that the to-be-detected message is a repeated message. That is, the to-be-detected message is processed by the target pre-filter and the re-filter in coordination. First, it is judged whether the message is repeated by using the bit array, which can exclude most first occurrence messages. Further comprehensive detection is performed on the messages that may be repeated, which can ensure the efficiency of message deduplication detection and improve the accuracy of message deduplication detection.

[0064] According to the statistical results, about 4% of the messages are repeated messages, and most of the messages are not repeated. Therefore, by using the above message detection method, 96% of the non-repeated messages can be quickly screened out by the target pre-filter, and the re-filter is used for further detection of the 4% repeated messages, which can greatly reduce the CPU (central processing unit) occupancy rate of the re-filter in the server, improve the efficiency of message detection, and improve the performance of server message detection.

[0065] In one embodiment, the target pre-filter currently used for pre-detection is determined, including: determining the working mode of the first pre-filter and the second pre-filter respectively in the current time period, and taking the pre-filter with the main mode as the target pre-filter; wherein the working mode includes the main mode and the standby mode, and the first pre-filter and the second pre-filter will be exchanged between the main mode and the standby mode once every time period.

[0066] The time period can be set as required. For example, the time period can be 8 minutes or 10 minutes. The server includes two candidate pre-filter, which are a first pre-filter and a second pre-filter. The first pre-filter and the second pre-filter can both be Bloom filters, i.e., the first pre-filter is a first Bloom filter and the second pre-filter is a second Bloom filter, and the target pre-filter is a target Bloom filter. The working mode of the first pre-filter can be a primary mode or a backup mode, and the working mode of the second pre-filter can also be a primary mode or a backup mode.

[0067] In the current time period, one of the first pre-filter and the second pre-filter is in the primary mode and the other is in the backup mode. When entering the next time period from the current time period, the working modes of the first pre-filter and the second pre-filter are exchanged.

[0068] For example, in the current time period, the working mode of the first pre-filter is the primary mode and the working mode of the second pre-filter is the backup mode. When entering the next time period from the current time period, the working modes of the first pre-filter and the second pre-filter are exchanged, i.e., the working mode of the first pre-filter is the backup mode and the working mode of the second pre-filter is the primary mode.

[0069] It can be understood that the pre-filter in the primary mode is in the working state, and the pre-filter in the primary mode is used as the target pre-filter for pre-detection of the to-be-detected message. The pre-filter in the backup mode is in the standby state, and when entering the next time period from the current time period, the working mode of the pre-filter is switched from the backup mode to the primary mode, and the pre-filter is used as the new target pre-filter for pre-detection of the to-be-detected message.

[0070] In the embodiment, when the first pre-filter and the second pre-filter exist, in the current time period, the pre-filter in the primary mode is determined as the target pre-filter, and the target pre-filter can accurately determine the pre-detection. And every time period, the primary mode and the backup mode are exchanged, and any pre-filter is used for work, which can balance the working pressure of each pre-filter.

[0071] In one embodiment, the above method further includes: whenever the pre-filter in the primary mode is switched to the backup mode, the element mark stored in the bit array of the pre-filter in the backup mode is cleaned up; and the pre-filter in the backup mode after the cleaning is used for mapping processing of the first mapping value of the first appearing to-be-detected message.

[0072] When the primary mode pre-filter switches to the backup mode, the server sets each element mark stored in the bit array of the backup mode pre-filter to a second mark. The second mark is the element mark stored in the bit array in the initial state. The server sets each element mark stored in the bit array of the backup mode pre-filter to the second mark, i.e. initializes the bit array of the backup mode pre-filter.

[0073] For example, when the primary mode pre-filter switches to the backup mode, the server sets each element mark stored in the bit array of the backup mode pre-filter to 0, obtaining an initialized bit array.

[0074] In this embodiment, when the primary mode pre-filter switches to the backup mode, the element marks stored in the bit array of the backup mode pre-filter are cleaned up, which can clean up outdated data in time, avoid storing too much data, and provide more memory space for the backup mode pre-filter to store new data after cleaning up.

[0075] In addition, the backup mode pre-filter after cleaning up can be used to map the first mapping value of the first appearing to-be-detected message, i.e. update the first mapping value of the first appearing to-be-detected message to the bit array; when entering the next time period from the current time period, the working mode of the backup mode pre-filter switches to the primary mode, serving as a new target pre-filter, and the bit array of the new target pre-filter stores data of the last time period, which can be used to more accurately pre-filter the received to-be-detected message.

[0076] Figure 4 A schematic diagram of the coordination between the first pre-filter and the second pre-filter in one embodiment is shown. Taking a time period of 8 minutes as an example, in the time period of 0-8 minutes, the working mode of the first pre-filter is the primary mode, the working mode of the second pre-filter is the backup mode, and the first pre-filter is the target pre-filter. The first pre-filter is used to pre-filter the to-be-detected message. When the server sends the to-be-detected message to the receiving terminal, the first mapping value corresponding to the to-be-detected message is mapped to the bit array of the first pre-filter and the bit array of the second pre-filter.

[0077] When entering the time period of 8-16 minutes from the time period of 0-8 minutes, the working modes of the first pre-filter and the second pre-filter are interchanged, that is, the working mode of the first pre-filter is the standby mode, and the working mode of the second pre-filter is the main mode, and the element marks stored in the bit array of the pre-filter in the standby mode, that is, the first pre-filter, are cleaned for mapping processing of the first mapping value of the first-occurred to-be-detected message. The bit array of the pre-filter in the main mode, that is, the second pre-filter, still stores the data stored in the time period of 0-8 minutes and can still be used for pre-detection of the new to-be-detected message.

[0078] In the time period of 8-16 minutes, the new to-be-detected message is pre-processed by the second pre-filter, and after the server delivers the to-be-detected message to the receiving terminal, the first mapping value corresponding to the to-be-detected message is mapped into the bit array of the first pre-filter and the bit array of the second pre-filter.

[0079] Similarly, when entering the time period of 16-24 minutes from the time period of 8-16 minutes, the working modes of the first pre-filter and the second pre-filter are interchanged, that is, the working mode of the first pre-filter is the main mode, and the working mode of the second pre-filter is the standby mode, and the element marks stored in the bit array of the pre-filter in the standby mode, that is, the second pre-filter, are cleaned for mapping processing of the first mapping value of the first-occurred to-be-detected message. The bit array of the pre-filter in the main mode, that is, the first pre-filter, still stores the data stored in the time period of 8-16 minutes and can still be used for pre-detection of the new to-be-detected message.

[0080] In the time period of 16-24 minutes, the new to-be-detected message is pre-processed by the first pre-filter, and after the server delivers the to-be-detected message to the receiving terminal, the first mapping value corresponding to the to-be-detected message is mapped into the bit array of the first pre-filter and the bit array of the second pre-filter.

[0081] In an embodiment, the above method further comprises: when all the element marks at the target positions are the first marks, determining that the to-be-detected message is not first-occurred; and when there is one target position at which the element mark is the second mark among the more than one target positions, determining that the to-be-detected message is first-occurred.

[0082] The first mark is a mark corresponding to the mapping of the to-be-detected message to the bit array by a first mapping function. The second mark is a mark in the bit array which is not mapped by the first mapping function, that is, an element mark stored in the bit array in an initial state. For example, the first mark can be 1 in binary, and the second mark can be 0 in binary.

[0083] It can be understood that the target pre-filter includes more than one first mapping function, the more than one first mapping function processes the to-be-detected message to obtain more than one first mapping value, and when elements at target positions corresponding to the more than one first mapping value are all first marks, it is determined that the to-be-detected message is not first occurrence.

[0084] When any one historical message corresponding to a first mapping value is mapped into the bit array, the mapped target position is marked as a first mark, and when an element at one of the more than one target positions is marked as a second mark, it can be determined that there is no message in the historical message that is the same as the to-be-detected message, and the to-be-detected message is first occurrence.

[0085] In the embodiment, when elements at all target positions are marked as first marks, it is determined that the to-be-detected message is not first occurrence, and the to-be-detected message can be detected again by the re-filter; when an element at one of the more than one target positions is marked as a second mark, it is determined that the to-be-detected message is first occurrence, and the to-be-detected message of first occurrence can be quickly screened out.

[0086] In one embodiment, the re-filter includes a multi-stage data chain, at least a part of historical mapping values are stored in each stage data chain, and the historical mapping values stored in all stages of data chains collectively constitute all historical mapping values; after the second mapping value of the to-be-detected message is obtained by the second mapping function in the re-filter, the method further includes: sequentially querying the second historical mapping value from the data chain corresponding to the order from the initial stage data chain in the multi-stage data chain to the highest stage data chain; when the second historical mapping value is queried from one of the multi-stage data chains, it is determined that the second mapping value is queried from the re-filter; when the second historical mapping value is not queried from all stages of data chains in the multi-stage data chain, it is determined that the to-be-detected message is first occurrence.

[0087] The data chain is a chain-shaped data structure for storing data. The re-filter can include one stage data chain or multi-stage data chains. When the re-filter includes multi-stage data chains, the multi-stage data chains also constitute a data table. When the second mapping function in the re-filter is a hash function, the multi-stage data chains are multi-stage hashes, and the one-stage data chain is one-stage hash.

[0088] When the re-filter includes a first-order data chain, the length of the data chain can be set as needed. When the re-filter includes multiple-order data chains, an initial length of an initial-order data chain is first set, and then prime numbers smaller than the initial length are sequentially determined, and the prime numbers are respectively taken as lengths of subsequent data chains in descending order.

[0089] For example, when the re-filter includes 6-order data chains, the initial length of the initial-order data chain is 100, and the prime numbers smaller than 100 are sequentially determined as 97, 89, 83, 79, 73, and the prime numbers are respectively taken as lengths of the subsequent 5-order data chains in descending order, i.e., the length of the second-order data chain is 97, the length of the third-order data chain is 89, the length of the fourth-order data chain is 83, the length of the fifth-order data chain is 79, and the length of the sixth-order data chain is 73.

[0090] The order from the initial-order data chain to the highest-order data chain, i.e., the order from the first-order data chain, then the second-order data chain, the third-order data chain, and so on, to the highest-order data chain.

[0091] In one embodiment, the server sequentially determines, in the order from the initial-order data chain to the highest-order data chain in the multiple-order data chains, an addressing position of the second mapping value in a current-order data chain, queries the second mapping value from the addressing position of the current-order data chain, matches the second mapping value with a historical mapping value of the addressing position, and when the historical mapping value of the addressing position does not match the second mapping value, it is determined that the second mapping value does not exist in the current-order data chain, and then the addressing position in a next-order data chain is determined, and the historical mapping value of the addressing position in the next-order data chain is matched with the second mapping value. The determination of the addressing position of the current-order data chain includes: taking the second mapping value modulo the length of the current-order data chain to obtain the addressing position of the second mapping value in the current-order data chain.

[0092] When the second historical mapping value is queried from one of the multi-stage data chains, i.e., there is a historical mapping value matching the second mapping value, it is determined that the second mapping value is queried from the re-filter; when the second historical mapping value is not queried from all the multi-stage data chains, i.e., the historical mapping values at the addressing positions in all the multi-stage data chains do not match the second historical mapping value, it is determined that the to-be-detected message appears for the first time. In this embodiment, the second historical mapping value is sequentially queried from the data chains corresponding to the respective orders in the order from the initial-stage data chain to the highest-stage data chain in the multi-stage data chains; when the second historical mapping value is queried from one of the multi-stage data chains, it is determined that the second mapping value is queried from the re-filter; when the second historical mapping value is not queried from all the multi-stage data chains, it is determined that the to-be-detected message appears for the first time, and whether the second mapping value exists in the re-filter can be accurately queried, so that whether the to-be-detected message is a repeated message can be accurately determined.

[0093] In one embodiment, the method further includes: establishing a process for the to-be-detected message; and processing the to-be-detected message through the second mapping function in the re-filter to obtain the second mapping value, including: accessing the re-filter through the process, and locking the process by using an atomic operation, processing the to-be-detected message through the second mapping function in the re-filter to obtain the second mapping value.

[0094] The process is a running activity of a program on a certain data set in a computer, is a basic unit of resource allocation and scheduling of a system, and is a basis of an operating system structure.

[0095] In one implementation, the server establishes a process for the to-be-detected message when the to-be-detected message is obtained. In another implementation, the server establishes a process for the to-be-detected message when it is determined that the to-be-detected message does not appear for the first time based on the element markers at the target positions.

[0096] It should be noted that the atomic operation refers to an operation that cannot be interrupted by a thread scheduling mechanism; once the atomic operation starts, it runs until the end without any context switch (switching to another thread) in between. The atomic operation can be one step or multiple operation steps, but its order cannot be disturbed or cut to execute only a part of it. Viewing the entire operation as a whole is the core feature of atomicity.

[0097] In the related art, when a server processes a message, multiple processes are included, and each process writes in the same shared memory. When one process writes in the shared memory, it is necessary to ensure that no other process is in the writing operation, so a lock is used to lock the process, and the lock is released after execution. However, when multiple processes find the same space node in the shared memory at the same time, a lock problem occurs, resulting in inaccurate message detection.

[0098] As shown in Figure 5 , when a server processes a message, process 1 and process 2 are included, and process 1 and process 2 find the same space node N in the shared memory at the same time. Process 1 gets the lock before writing, and process 2 is blocked. Process 1 is locked, data is written to data node N, and the lock is released. During the locking of process 1, process 2 waits for the lock; when process 1 releases the lock, process 2 gets the lock and writes data to node N. At this time, process 2 writes data to node N, which will overwrite the result written by process 1. Then, if the same message of process 1 is repeatedly uploaded, the message cannot be detected as a repeated message from the shared memory of the duplicate filter, resulting in inaccurate message detection.

[0099] In the embodiment, when the process corresponding to the to-be-detected message accesses the duplicate filter, the process is locked by using an atomic operation, and then the to-be-detected message is processed by using the second mapping function in the duplicate filter to obtain a second mapping value, so that the lock problem can be avoided, the accuracy of data detection is improved, and the message miss-delivery problem is solved.

[0100] In one embodiment, the above method further includes: when it is determined that the to-be-detected message appears for the first time based on the element mark at each target position, performing a message delivery operation on the to-be-detected message; or when the second mapping value is not queried from all historical mapping values in the duplicate filter, determining that the to-be-detected message appears for the first time, and performing a message delivery operation on the to-be-detected message.

[0101] When the element mark of one target position is the second mark among the element marks at each target position of the bit array in the target pre-filter, it is determined that the to-be-detected message appears for the first time. For example, when the element mark of one target position is the second mark 0 among the element marks at each target position of the bit array in the target pre-filter, it is determined that the to-be-detected message appears for the first time.

[0102] When the second mapping value is not queried from all historical mapping values in the duplicate filter, it indicates that the to-be-detected message is not mapped to the data structure of the duplicate filter, and it can be determined that the to-be-detected message appears for the first time.

[0103] When the server determines that the to-be-detected message first appears, the server performs a message issuing operation on the to-be-detected message, and issues the to-be-detected message to the receiving terminal. After the to-be-detected message is issued to the receiving terminal, the to-be-detected message is processed by using the second mapping function to obtain a second mapping value, and the second mapping value is stored as a historical mapping value in the data structure of the re-filter, so as to be used for subsequent detection of to-be-detected messages.

[0104] As shown in FIG. 1, in the related art, when the server detects whether the to-be-detected message A is duplicated, the server inserts a mapping value corresponding to the to-be-detected message A into the shared memory, and then issues the to-be-detected message. When the issuing fails due to a backend timeout, the server re-detects whether the to-be-detected message A is duplicated. Since the mapping value corresponding to the to-be-detected message A has been inserted into the shared memory, if the mapping value in the shared memory is not deleted in advance, the detection result indicates that the to-be-detected message A is duplicated, which causes an inaccurate detection of the to-be-detected message. Figure 6 In the embodiment, when it is determined that the to-be-detected message first appears based on the element markers at the target positions, or when the second mapping value is not queried from all the historical mapping values in the re-filter, it can be accurately determined that the to-be-detected message first appears, and the message issuing operation is performed on the to-be-detected message, which avoids the inaccurate detection when the message issuing fails, and avoids the operation of deleting the data in the shared memory in advance, thereby saving computer resources.

[0105] In one embodiment, the above method further includes: when it is determined that the to-be-detected message is successfully issued, updating the element markers at the corresponding target positions in the bit array of the target pre-filter based on the to-be-detected message; and adding the second mapping value of the to-be-detected message to the re-filter to update the historical mapping values in the re-filter.

[0106] When it is determined that the to-be-detected message is successfully issued, the server sets the element markers at the target positions in the bit array to first markers. The first markers can be set as needed. For example, the first markers can be binary value 1, or specific markers.

[0107] It should be noted that when it is determined that the to-be-detected message is successfully issued, the server updates the element markers at the corresponding target positions in the bit array of the target pre-filter, and updates the historical mapping values in the re-filter, which can be performed simultaneously, or one step can be performed first and then the other step is performed, which is not limited herein.

[0108]

[0109] ​In the embodiment, when it is determined that the to-be-detected message is successfully issued, the element mark at the corresponding target position in the bit array of the target pre-filter is updated based on the to-be-detected message, and the second mapping value of the to-be-detected message is added to the re-filter to update the historical mapping value in the re-filter, so that the problem of deleting the stored data due to the failure of the message issuing can be avoided, resources are saved, and the to-be-detected message can be more accurately detected subsequently.

[0110] In one embodiment, the method further includes: when no acknowledgement message returned by the receiving terminal is received within a specified time length after the message issuing operation is performed, it is determined that the to-be-detected message is unsuccessfully issued; when the message issuing fails, the message issuing operation is re-performed within a time period after the message issuing operation is performed for the first time until the acknowledgement message returned by the receiving terminal is received, and it is determined that the to-be-detected message is successfully issued; and the specified time length is less than the time period.

[0111] The specified time length can be set as required. For example, the specified time length can be 10 seconds or 8 seconds. The time period can be set as required. For example, the time period can be 8 minutes or 5 minutes. The smaller the specified time length is and the larger the time period is, the more times the to-be-detected message can be re-issued.

[0112] An acknowledgement (ACK) is part of a communication protocol in computer networking, a message sent by a device or process to confirm that it has received a message.

[0113] It can be understood that when the receiving terminal successfully receives the to-be-detected message issued, an acknowledgement message is returned to the server, and the acknowledgement message is used to inform the server that the receiving terminal has received the to-be-detected message issued, that is, the to-be-detected message is successfully issued. When no acknowledgement message returned by the receiving terminal is received within a specified time length after the server performs the message issuing operation, the receiving terminal can fail to successfully receive the to-be-detected message due to network interruption or network timeout of the receiving terminal, and it is determined that the to-be-detected message is unsuccessfully issued.

[0114] It can be understood that when the message issuing fails, the server re-performs the message issuing operation within a time period after the message issuing operation is performed for the first time until the acknowledgement message returned by the receiving terminal is received, and it is determined that the to-be-detected message is successfully issued. The element mark at the corresponding target position in the bit array of the target pre-filter is updated based on the to-be-detected message, and the second mapping value of the to-be-detected message is added to the re-filter to update the historical mapping value in the re-filter.

[0115] When the current time exceeds the time period after the message issuing operation is performed for the first time, the message issuing operation is no longer performed, and resources are saved.

[0116] It should be noted that the server re-executes the message sending operation within a time period after the first execution of the message sending operation, and also detects the message to be detected for the execution of the sending operation. When entering the next time period from the current time period, the working mode of the pre-filter is interchanged. The target pre-filter whose working mode is in the main mode still contains the data in the previous time period, and can still detect the message to be detected within a time period after the first execution of the message sending operation.

[0117] In one embodiment, Figure 7 As shown, the server obtains the message to be detected 702 and executes step 704 to perform pre-detection on the message to be detected, that is, executes step 708 through the target pre-filter 706 to determine whether the message to be detected 702 appears for the first time. In the current time period, the operating mode of the first pre-filter 720 and the second pre-filter 722 are determined respectively, and the pre-filter operating in the main mode is used as the target pre-filter 706.

[0118] Specifically, the message to be detected 702 is processed respectively by more than one first mapping function in the target pre-filter 706 to obtain corresponding first mapping values; based on the bit array in the target pre-filter 706, the target positions corresponding to each first mapping value in the bit array are queried; when the element marks at all target positions are the first mark, it is determined that the message to be detected is not the first appearance; when there is an element mark of the second mark at more than one target position, it is determined that the message to be detected appears for the first time.

[0119] If the target pre-filter 706 determines that the message to be detected appears for the first time, step 714 is executed to perform a message delivery operation on the message to be detected 702. If the target pre-filter 706 determines that the message to be detected does not appear for the first time, step 712 is executed through the re-filter 712 to determine whether the message to be detected 702 appears for the first time.

[0120] Specifically, the message to be detected is processed by the second mapping function in the re-filter 712 to obtain a second mapping value; when the second mapping value is queried from all historical mapping values ​​in the re-filter, it is determined that the message to be detected is a duplicate message, and it is determined that the message to be detected 702 does not appear for the first time, and the operation of sending the message to be detected is not performed; when the second mapping value is not queried from all historical mapping values ​​in the re-filter, it is determined that the message to be detected 702 appears for the first time, then step 714 is executed, and the message sending operation is performed on the message to be detected 702.

[0121] When the server receives the acknowledgement message 716, indicating that the to-be-detected message 702 is successfully issued, the element mark at the corresponding target position in the bit array of the first pre-filter 720 is updated based on the to-be-detected message, the element mark at the corresponding target position in the bit array of the second pre-filter 722 is updated, and the second mapping value of the to-be-detected message is added to the re-filter 710 to update the historical mapping values in the re-filter.

[0122] Every time a time period elapses, step 724 is performed to interchange the working modes of the pre-filters, i.e., the working modes of the first pre-filter and the second pre-filter are interchanged once between the primary mode and the standby mode.

[0123] In the embodiment, the server first performs pre-detection on the to-be-detected message by the target pre-filter, which can quickly filter out the to-be-detected message that appears for the first time, and then directly performs the message issuing operation. When it is detected that the to-be-detected message does not appear for the first time, the to-be-detected message is detected by all the historical mapping values in the re-filter, which can accurately detect whether the to-be-detected message is a repeated message.

[0124] After the to-be-detected message is successfully issued, the server updates the data in the first pre-filter, the second pre-filter, and the re-filter based on the to-be-detected message, which can avoid the problem of inaccurate detection of the to-be-detected message when reissuing due to failure, improve the accuracy of detection, and also avoid the operation of deleting the data in the filter due to message issuing failure, thereby saving computer resources.

[0125] The first pre-filter and the second pre-filter are used to take turns as the target pre-filter, which can balance the working pressure of each pre-filter.

[0126] Figure 8 A timing diagram of message detection in an embodiment. A sending terminal sends a to-be-detected message 802 to a server. The server receives the to-be-detected message, performs step 804 to perform pre-processing on the to-be-detected message by a target pre-filter, step 806 to issue the to-be-detected message to a receiving terminal when it is determined by the target pre-filter that the to-be-detected message appears for the first time, step 808 to perform re-detection on the to-be-detected message by a re-filter when it is determined by the target pre-filter that the to-be-detected message does not appear for the first time, and step 810 to issue the to-be-detected message to the receiving terminal when it is determined by the re-filter that the to-be-detected message appears for the first time.

[0127] After the server issues the to-be-detected message to the receiving terminal, when an acknowledgement message 812 returned by the receiving terminal is received, step 814 is performed to update the bit array of the target pre-filter and update the historical mapping values in the re-filter.

[0128] In one embodiment, the message detection method comprises the following steps:

[0129] Step 1: The server acquires a message to be detected.

[0130] Step 2: In the current time period, the working mode of the first pre-filter and the second pre-filter is determined respectively, and the pre-filter with the working mode as the main mode is taken as the target pre-filter; wherein the working mode includes the main mode and the standby mode, and the first pre-filter and the second pre-filter will be exchanged between the main mode and the standby mode every time period.

[0131] Step 3: Whenever the pre-filter in the main mode is switched to the standby mode, the element mark stored in the bit array of the pre-filter in the standby mode is cleaned up; after cleaning up, the pre-filter in the standby mode is used to map the first mapping value of the first occurrence of the message to be detected.

[0132] Step 4: The message to be detected is processed by more than one first mapping function in the target pre-filter respectively, and the corresponding first mapping value is obtained.

[0133] Step 5: Based on the bit array in the target pre-filter, the target positions of each first mapping value in the bit array are queried.

[0134] Step 6: When the element marks at all target positions are the first mark, it is determined that the message to be detected is not the first occurrence, and step 7 is executed; when the element mark of one target position among more than one target positions is the second mark, it is determined that the message to be detected is the first occurrence, and step 10 is executed.

[0135] Step 7: A process is established for the message to be detected.

[0136] Step 8: When the process accesses the re-filter, the process is locked by using an atomic operation, and the message to be detected is processed by the second mapping function in the re-filter to obtain a second mapping value.

[0137] Step 9: According to the order from the initial stage data chain to the highest stage data chain in the multi-stage data chain, the second historical mapping value is queried from the data chain corresponding to the corresponding order in sequence; when the second historical mapping value is queried from one stage data chain in the multi-stage data chain, it is determined that the second mapping value is queried from the re-filter, and it is determined that the message to be detected is a repeated message, and the message delivery operation is not executed; when the second historical mapping value is not queried from all stage data chains in the multi-stage data chain, it is determined that the message to be detected is the first occurrence, and step 10 is executed.

[0138] Step 10: performing a message issuing operation on the to-be-detected message; detecting whether the to-be-detected message is successfully issued; when the to-be-detected message is successfully issued, performing step 11; when an acknowledgement message returned by the receiving terminal is not received within a specified time period after the message issuing operation is performed, it is confirmed that the to-be-detected message fails to be issued, and step 12 is performed.

[0139] Step 11: when it is determined that the to-be-detected message is successfully issued, updating an element mark at a corresponding target position in a bit array of the target pre-filter based on the to-be-detected message; and adding a second mapping value of the to-be-detected message to the re-filter to update a historical mapping value in the re-filter.

[0140] Step 12: re-performing the message issuing operation within a time period after the message issuing operation is performed for the first time, until an acknowledgement message returned by the receiving terminal is received, it is determined that the to-be-detected message is successfully issued, and step 11 is performed; the specified time period is less than the time period.

[0141] In the embodiment, the server first performs pre-detection on the to-be-detected message through the target pre-filter, can quickly filter out the to-be-detected message that appears for the first time, and directly performs the message issuing operation; when it is detected that the to-be-detected message appears for the first time, the to-be-detected message is detected through all historical mapping values in the re-filter, and it can be accurately detected whether the to-be-detected message is a repeated message.

[0142] After the to-be-detected message is successfully issued, the server updates the data in the first pre-filter, the second pre-filter and the re-filter based on the to-be-detected message, which can avoid the problem that the detection of the to-be-detected message is inaccurate when re-issuing after the message issuing fails, improve the accuracy of the detection, and also avoid the operation of deleting the data in the filter when the message issuing fails, saving computer resources.

[0143] The first pre-filter and the second pre-filter are used to rotate as the target pre-filter, which can balance the working pressure of each pre-filter.

[0144] The application further provides an application scenario, and the message detection method is applied in the application scenario.

[0145] The first user logs in a session application on a sending terminal, opens a session page with a second user in the session application, inputs a to-be-detected message in an information input window of the session page, triggers a sending operation, and then the sending terminal sends the to-be-detected message to the server. The to-be-detected message can be a session message.

[0146] The server receives the to-be-detected message sent by the sending terminal, performs preliminary detection on the to-be-detected message through the target preliminary filter, and when it is determined that the to-be-detected message appears for the first time, the to-be-detected message is sent to the receiving terminal where the second user is located; when it is determined that the to-be-detected message does not appear for the first time, the to-be-detected message is detected again through the re-filter.

[0147] When it is determined through the re-filter that the to-be-detected message appears for the first time, the to-be-detected message is sent to the receiving terminal where the second user is located; when it is determined that the to-be-detected message does not appear for the first time, the to-be-detected message is a repeated message, and the message sending operation is not performed.

[0148] When the to-be-detected message is successfully sent, the server updates the historical data in the target preliminary filter, the preliminary filter in the standby mode, and the re-filter.

[0149] It should be understood that, although Figure 2 and Figure 7 the flowcharts in the respective steps are displayed in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 2 and Figure 7 at least part of the steps in and can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0150] In one embodiment, as shown in Figure 9 , a message detection device is provided, which can be a software module or a hardware module, or a combination of the two as part of a computer device, and specifically includes a determination module 902, a first mapping module 904, a query module 906, and a second mapping module 908, wherein:

[0151] The determination module 902 is configured to obtain a to-be-detected message and determine a target preliminary filter currently used for preliminary detection.

[0152] The first mapping module 904 is configured to process the to-be-detected message through more than one first mapping function in the target preliminary filter to obtain corresponding first mapping values.

[0153] The query module 906 is configured to query the target positions of the respective first mapping values in the bit array based on the bit array in the target preliminary filter.

[0154] The second mapping module 908 is configured to, when it is determined that the to-be-detected message is not first appeared based on the element markers at the target positions, process the to-be-detected message by using a second mapping function in the re-filter to obtain a second mapping value.

[0155] The determining module 902 is further configured to, when the second mapping value is queried from all historical mapping values in the re-filter, determine that the to-be-detected message is a duplicate message.

[0156] The message detection apparatus described above processes the to-be-detected message by using more than one first mapping function in the target pre-filter to obtain corresponding first mapping values, queries the target positions corresponding to the first mapping values in the bit array in the target pre-filter based on the bit array, and determines whether the to-be-detected message is not first appeared based on the element markers at the target positions. When the to-be-detected message is not first appeared, it indicates that the to-be-detected message is likely to be a duplicate message, and further accurate detection is performed by using the re-filter. Specifically, the second mapping value is obtained by using a second mapping function in the re-filter, and it is more accurately queried from all historical mapping values in the re-filter whether the second mapping value is a duplicate message. When the second mapping value is queried from the historical mapping values, it is determined that the to-be-detected message is a duplicate message. That is, the to-be-detected message is processed by using the target pre-filter and the re-filter in coordination. First, it is determined whether the message is duplicated by using the bit array, which can exclude most of the first appeared messages. Then, further comprehensive detection is performed on the messages that are likely to be duplicated, which can ensure the efficiency of the message deduplication detection and improve the accuracy of the message deduplication detection.

[0157] In an embodiment, the determining module is further configured to determine the working mode of the first pre-filter and the second pre-filter respectively in a current time period, and use the pre-filter in the main mode as the target pre-filter; wherein the working mode includes the main mode and the standby mode, and the first pre-filter and the second pre-filter are exchanged between the main mode and the standby mode every time period.

[0158] In an embodiment, the apparatus further includes a working mode switching module configured to clean the element markers stored in the bit array of the pre-filter in the standby mode after the pre-filter in the main mode is switched to the standby mode; and the pre-filter in the standby mode is used to map the first mapping value of the first appeared to-be-detected message after the cleaning.

[0159] In an embodiment, the determining module is further configured to determine that the to-be-detected message is not first appeared when the element markers at all target positions are the first markers; and determine that the to-be-detected message is first appeared when the element marker of one target position is the second marker among more than one target positions.

[0160] In an embodiment, the re-filter comprises a plurality of data chains, each of which stores at least part of the historical mapping values, and all the data chains store all the historical mapping values. The apparatus further comprises a re-detection module configured to sequentially query the second historical mapping value from the data chains corresponding to the order from the initial data chain to the highest data chain in the plurality of data chains, and determine that the second mapping value is queried from the re-filter when the second historical mapping value is queried from one of the data chains in the plurality of data chains, and determine that the to-be-detected message is first appeared when the second historical mapping value is not queried from all the data chains in the plurality of data chains.

[0161] In an embodiment, the apparatus further comprises a process locking module configured to establish a process for the to-be-detected message, and lock the process by using an atomic operation when the process accesses the re-filter. The second mapping function processing module 908 is further configured to process the to-be-detected message by using the second mapping function in the re-filter to obtain the second mapping value.

[0162] In an embodiment, the apparatus further comprises a delivery module configured to perform a message delivery operation on the to-be-detected message when it is determined that the to-be-detected message is first appeared based on the element mark at the target position in the bit array of the target pre-filter. The delivery module is further configured to determine that the to-be-detected message is first appeared when the second mapping value is not queried from all the historical mapping values in the re-filter, and perform the message delivery operation on the to-be-detected message.

[0163] In an embodiment, the apparatus further comprises an updating module configured to update the element mark at the target position in the bit array of the target pre-filter based on the to-be-detected message when it is determined that the message delivery is successful, and add the second mapping value of the to-be-detected message to the re-filter to update the historical mapping value in the re-filter.

[0164] In an embodiment, the delivery module is further configured to determine that the message delivery is failed when an acknowledgement message returned by the receiving terminal is not received within a specified time period after the message delivery operation is performed, and perform the message delivery operation again within a time period after the message delivery operation is performed for the first time until the acknowledgement message returned by the receiving terminal is received, and then determine that the message delivery is successful. The specified time period is less than the time period.

[0165] The specific limitations of the message detection apparatus can refer to the limitations of the message detection method described above, which will not be repeated here. Each module in the above message detection apparatus can be implemented by software, hardware, and a combination thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor invokes and executes the operations corresponding to the above modules.

[0166] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 10 The computer device includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store target pre-filter, re-filter, to-be-detected message, historical message, and the like. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a message detection method.

[0167] Those skilled in the art can understand that Figure 10 The structure shown in the above figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0168] In one embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0169] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0170] In one embodiment, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the steps in the above method embodiments.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0172] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0173] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A message detection method, characterized in that: The method comprises: Obtain the message to be detected and determine the target pre-filter currently used for pre-detection; Establishing a process for the message to be detected; Processing the message to be detected respectively by using more than one first mapping function in the target pre-filter to obtain corresponding first mapping values; Based on the bit array in the target pre-filter, querying the target position corresponding to each of the first mapping values ​​in the bit array; When it is determined based on the element tags at each of the target positions that the message to be detected does not appear for the first time, accessing the refilter through the process, locking the process using an atomic operation, and processing the message to be detected through the second mapping function in the refilter to obtain a second mapping value; When the second mapping value is found from all historical mapping values ​​in the re-filter, it is determined that the message to be detected is a duplicate message.

2. The method according to claim 1, characterized in that The determining of the target pre-filter currently used for pre-detection includes: In a current time period, the working modes of the first pre-filter and the second pre-filter are respectively determined, and the pre-filter whose working mode is the main mode is used as the target pre-filter; wherein, the working mode includes a main mode and a backup mode, and the first pre-filter and the second pre-filter will exchange the main mode and the backup mode once every time period.

3. The method according to claim 2, characterized in that The method further comprises: Whenever the pre-filter in the main mode switches to the standby mode, the element tags stored in the bit array of the pre-filter in the standby mode are cleaned; after cleaning, the pre-filter in the standby mode is used to map the first mapping value of the message to be detected that appears for the first time.

4. The method according to claim 1, wherein The method further comprises: When all element marks at the target positions are first marks, determining that the message to be detected is not the first occurrence; When an element mark of one target location among more than one target locations is the second mark, it is determined that the message to be detected appears for the first time.

5. The method according to claim 1, wherein The re-filter includes a multi-order data chain, each order data chain stores at least a portion of historical mapping values, and the historical mapping values ​​stored in all orders of data chains together constitute all historical mapping values. After the message to be detected is processed by the second mapping function in the re-filter to obtain the second mapping value, the method further includes: In order from the initial order data chain to the highest order data chain in the multi-order data chains, query the second mapping value from the data chains corresponding to the corresponding order; When the second mapping value is queried from one of the first-order data chains in the multi-order data chains, it is determined that the second mapping value is queried from the heavy filter; When the second mapping value is not found in all order data links in the multi-order data links, it is determined that the message to be detected appears for the first time.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: When it is determined based on the element tags at each of the target positions that the message to be detected appears for the first time, executing a message sending operation on the message to be detected; or When the second mapping value is not found from all historical mapping values ​​in the re-filter, it is determined that the message to be detected appears for the first time, and a message sending operation is performed on the message to be detected.

7. The method according to claim 6, characterized in that The method further comprises: When it is determined that the message to be detected is successfully sent, updating the element mark at the corresponding target position in the bit array of the target pre-filter based on the message to be detected; The second mapping value of the message to be detected is added to the re-filter to update the historical mapping value in the re-filter.

8. The method according to claim 7, characterized in that The method further comprises: When no confirmation message returned by the receiving terminal is received within a specified time period after the message sending operation is executed, it is confirmed that the sending of the message to be detected has failed; When the message sending fails, the message sending operation is re-executed within a time period after the first execution of the message sending operation until a confirmation message returned by the receiving terminal is received, then it is determined that the message to be detected has been sent successfully; the specified time length is less than the time period.

9. A message detection device, characterized in that: The device comprises: A determination module is used to obtain the message to be detected and determine the target pre-filter currently used for pre-detection; A process locking module, configured to establish a process for the message to be detected; A first mapping module, configured to process the message to be detected respectively through more than one first mapping function in the target pre-filter to obtain corresponding first mapping values; a query module, configured to query, based on the bit array in the target pre-filter, target positions corresponding to the first mapping values ​​in the bit array; The process locking module is further configured to, when it is determined based on the element tags at each target position that the message to be detected is not the first occurrence, access the re-filter through the process and lock the process using an atomic operation; a second mapping module configured to, when it is determined based on the element tags at each of the target positions that the message to be detected does not appear for the first time, process the message to be detected using a second mapping function in the re-filter to obtain a second mapping value; The determining module is further configured to determine that the message to be detected is a duplicate message when the second mapping value is found from all historical mapping values ​​in the re-filter.

10. The message detection device according to claim 9, characterized in that: The determination module is further configured to determine, within a current time period, the operating modes of the first pre-filter and the second pre-filter, respectively, and to use the pre-filter whose operating mode is the main mode as the target pre-filter; wherein the operating modes include a main mode and a backup mode, and the first pre-filter and the second pre-filter will exchange between the main mode and the backup mode once every time period.

11. The message detection device according to claim 10, characterized in that: The device further includes a working mode switching module, wherein the working mode switching module is configured to clear element tags stored in the bit array of the pre-filter in the standby mode whenever the pre-filter in the main mode switches to the standby mode; The pre-filter in the standby mode after cleaning is used to perform mapping processing on the first mapping value of the message to be detected that appears for the first time.

12. The message detection device according to claim 9, characterized in that The determination module is also used to determine that the message to be detected is not the first occurrence when the element marks at all the target positions are the first mark; and to determine that the message to be detected is the first occurrence when the element mark at one of more than one target positions is the second mark.

13. The message detection device according to claim 9, characterized in that: The re-filter includes a multi-stage data chain, each stage data chain stores at least a portion of the historical mapping values, and the historical mapping values ​​stored in all stages of the data chain together constitute all the historical mapping values; The device also includes a re-detection module, which is used to query the second mapping value from the data chains corresponding to the corresponding order in the order from the initial order data chain to the highest order data chain in the multi-order data chain; when the second mapping value is queried from one of the first-order data chains in the multi-order data chain, it is determined that the second mapping value is queried from the re-filter; when the second mapping value is not queried from all order data chains in the multi-order data chain, it is determined that the message to be detected appears for the first time.

14. The message detection device according to any one of claims 9 to 13, characterized in that: The device further includes a sending module, configured to send a message to the message to be detected when it is determined based on the element tags at each target position that the message to be detected appears for the first time; Or when the second mapping value is not found from all historical mapping values ​​in the re-filter, it is determined that the message to be detected appears for the first time, and a message sending operation is performed on the message to be detected.

15. The message detection device according to claim 14, characterized in that: The device further includes an updating module, configured to update an element tag at a corresponding target position in the bit array of the target pre-filter based on the message to be detected when it is determined that the message to be detected is successfully sent; The second mapping value of the message to be detected is added to the re-filter to update the historical mapping value in the re-filter.

16. The message detection device according to claim 15, characterized in that: The sending module is further configured to confirm that sending of the message to be detected has failed when no confirmation message returned by the receiving terminal is received within a specified time period after the message sending operation is executed; When the message sending fails, the message sending operation is re-executed within a time period after the first execution of the message sending operation until a confirmation message returned by the receiving terminal is received, then it is determined that the message to be detected has been sent successfully; the specified time length is less than the time period.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

18. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

19. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images