A data filtering method and device

Abstract

The present invention discloses a data filtering method and device, and relates to the field of computer technology. A specific implementation of the method includes receiving a data storage request, obtaining a unique identifier of an object to be stored in the request; calling a first hash function, calculating a first array index according to the unique identifier of the object to be stored, and then judging whether the array data corresponding to the first array index is empty, if so, storing the unique identifier of the object to be stored in the corresponding first array, and the request processing is completed; if not, obtaining the unique identifier in the first array data, judging whether it is the same as the unique identifier of the object to be stored, if so, the request processing is completed; if not, calling a second hash function, calculating a second array index according to the unique identifier of the object to be stored, and storing the unique identifier of the object to be stored in the corresponding second array. Thus, the implementation of the present invention can solve the problems of low efficiency and high misjudgment rate of existing data filtering.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data filtering method and device. Background Art

[0002] Cuckoo filter is a function that performs deduplication and filtering on big data in order to save huge amounts of data storage space. The cuckoo filter is derived from the cuckoo hash algorithm, and its basic structure is an array, two hash algorithms, and a hash algorithm for calculating fingerprints.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] Currently, the Cuckoo filter needs to calculate the fingerprint information of the object through the fingerprint hash algorithm based on the inserted object. The fingerprint is only one byte with 256 possibilities. There is a high possibility that the same fingerprint will appear in the same position, which will lead to misjudgment in query and deletion. In addition, inserting multiple identical elements will lead to a run cycle. In addition, when querying at the same time during the kicking process, it is possible that the fingerprint that has just been kicked out and has not found a new position cannot be queried, resulting in misjudgment. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data filtering method and device, which can solve the problems of low efficiency and high misjudgment rate of existing data filtering.

[0006] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a data filtering method is provided, including receiving a data storage request, obtaining a unique identifier of an object to be stored in the request; calling a first hash function, calculating a first array subscript according to the unique identifier of the object to be stored, and then judging whether the array data corresponding to the first array subscript is empty, if so, storing the unique identifier of the object to be stored in the corresponding first array, and completing the request processing; if otherwise, obtaining the unique identifier in the first array data, judging whether it is the same as the unique identifier of the object to be stored, and if so, completing the request processing; if otherwise, calling a second hash function, calculating a second array subscript according to the unique identifier of the object to be stored, and storing the unique identifier of the object to be stored in the corresponding second array.

[0007] Optionally, calling the second hash function to calculate the second array index according to the unique identifier of the object to be stored includes:

[0008] Determine whether the array data corresponding to the second array index is empty, if so, store the unique identifier of the object to be stored in the corresponding second array, and the request processing is completed;

[0009] If not, obtain the unique identifier in the second array data, and determine whether it is the same as the unique identifier of the object to be stored. If so, the request processing is completed; if not, kick out the unique identifier in the second array data to store the unique identifier of the object to be stored in the second array.

[0010] Optionally, the unique identifier in the second array data is kicked out, including:

[0011] Obtain the query lock identifier, and determine whether the query lock identifier is locked. If so, kick out the unique identifier in the second array data and put it into a queue to wait for re-storage; otherwise, kick out the unique identifier in the second array data and re-store it.

[0012] Optionally, after kicking out the unique identifier in the second array data, the method further comprises:

[0013] Get the kick status flag and increment it;

[0014] After the unique identifier of the kickout is stored again, the kickout status identifier is obtained and decremented.

[0015] Optionally, it also includes:

[0016] Receive a data query request, and obtain a unique identifier of an object to be queried in the request;

[0017] Calling a first hash function to calculate a first array subscript according to the unique identifier of the object to be stored, and then determining whether the array data corresponding to the first array subscript is empty;

[0018] If yes, then mark the preset empty mark, call the second Hash function to execute the above process cyclically to obtain the empty mark, and return the result that it is not found after being marked twice;

[0019] Otherwise, obtain the unique identifier in the first array data, and determine whether it is the same as the unique identifier of the object to be queried. If it is, return the result as found; if otherwise, call the second hash function to loop and execute the above process and still get a unique identifier that is inconsistent with the object to be queried, then return the result as not found.

[0020] Optionally, after calling the second hash function to cyclically execute the aforementioned process and still obtaining a unique identifier that is inconsistent with the object to be queried, the method includes:

[0021] Obtaining a query lock flag, and determining whether the query lock flag is unlocked;

[0022] If so, determine whether the kick-out status flag is 0, and if so, return the result as not found; if not, change the query lock flag to locked, and set the kick-out status flag to 0, traverse the queue to find whether there is a unique flag of the object to be queried in the kicked unique flag, and if so, return the result as found, wait for all the kicked unique flags in the queue to be dequeued, change the query lock flag to unlocked, and restore the kick-out status flag; if not, return the result as not found, wait for all the kicked unique flags in the queue to be dequeued, change the query lock flag to unlocked, and restore the kick-out status flag;

[0023] If not, the first Hash function and the second Hash function are called cyclically to perform the query until the returned result is that the query has been completed or the query lock mark is unlocked and the returned result is that the query has not been completed.

[0024] Optionally, it also includes:

[0025] Executing the received request through a thread, if the unique identifier is re-stored, then increasing the preset re-storage times of the thread;

[0026] The number of restoring times of the thread is monitored, and if the number of restoring times is greater than a preset number threshold, the storage space of the first array and the second array is expanded.

[0027] In addition, the present invention also provides a data filtering device, including a receiving module, which is used to receive a data storage request and obtain the unique identifier of the object to be stored in the request; a processing module, which is used to call a first hash function, calculate a first array subscript according to the unique identifier of the object to be stored, and then determine whether the array data corresponding to the first array subscript is empty. If so, the unique identifier of the object to be stored is stored in the corresponding first array, and the request processing is completed; if otherwise, the unique identifier in the first array data is obtained, and it is determined whether it is the same as the unique identifier of the object to be stored. If so, the request processing is completed; if otherwise, the second hash function is called, and the second array subscript is calculated according to the unique identifier of the object to be stored to store the unique identifier of the object to be stored in the corresponding second array.

[0028] One embodiment of the above invention has the following advantages or beneficial effects: In order to avoid misjudgment and circular run problems, the present invention no longer uses fingerprints for recording, but can directly record through the configured object unique identifier. In addition, a locking mechanism is added when performing operations such as query, addition and deletion to ensure the absolute accuracy of the query, and the efficiency of the locked query is optimized, and only the kicked data is locked, which will not affect the normal progress of other processes.

[0029] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0031] Figure 1 is a schematic diagram of the main process of the data filtering method according to the first embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of the main process of the data filtering method according to the second embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of the main process of the data filtering method according to the third embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of main modules of a data filtering device according to an embodiment of the present invention;

[0035] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0036] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] Figure 1 FIG. 1 is a schematic diagram of the main process of the data filtering method according to the first embodiment of the present invention. Figure 1 As shown, the data filtering method includes:

[0039] Step S101: receiving a data storage request, and obtaining a unique identifier of an object to be stored in the request.

[0040] In an embodiment, a data storage request is received and parsed to obtain a unique identifier of the object to be stored carried in the request, wherein the unique identifier is generated according to preset configuration information, for example, the configuration information is user ID+event code.

[0041] That is to say, the caller directly generates a unique identifier for the object to be stored based on the preset configuration information, and puts the unique identifier into the data storage request. After receiving the data storage request, the called party can directly obtain the unique identifier of the object to be stored (if the unique identifiers are the same, the objects to be stored are considered to be the same) for storage and deduplication, without any processing, which greatly saves processing time.

[0042] Preferably, the configuration information can be set as a parameter of a unique String type specified by the user.

[0043] Step S102, calling the first hash function, calculating the first array subscript according to the unique identifier of the object to be stored, and then judging whether the array data corresponding to the first array subscript is empty, if so, storing the unique identifier of the object to be stored in the corresponding first array, and the request processing is completed; if otherwise, obtaining the unique identifier in the first array data, and judging whether it is the same as the unique identifier of the object to be stored, if so, the request processing is completed; if otherwise, calling the second hash function, calculating the second array subscript according to the unique identifier of the object to be stored, and storing the unique identifier of the object to be stored in the corresponding second array.

[0044] In the embodiment, the first hash function and the second hash function only need to be different, the first hash function corresponds to the first array, and the second hash function corresponds to the second array. Of course, one hash function and one array can also be set to perform a data filtering process, and the processing process is the same as that of each embodiment of the present invention, except that the secondary array judgment is not performed.

[0045] It should be noted that the same unique identifier may appear when receiving two data storage requests. This situation is extremely rare, but if the same unique identifier is stored in two locations in the array, it can be regarded as redundant data if it is not kicked out (redundant data is conducive to improving query efficiency). If it is kicked out, the kicked out unique identifier will inevitably use another hash function to continue looking for a foothold. At this time, another identical unique identifier will definitely be found. The duplicate unique identifier will be regarded as having found the location and the processing will end, and the deduplication operation will be completely completed.

[0046] In some embodiments, after the second hash function is called and the second array index is calculated based on the unique identifier of the object to be stored, it can be determined whether the array data corresponding to the second array index is empty. If so, the unique identifier of the object to be stored is stored in the corresponding second array, and the request processing is completed. If not, the unique identifier in the second array data is obtained, and it is determined whether it is the same as the unique identifier of the object to be stored. If so, the request processing is completed; if not, the unique identifier in the second array data is kicked out to store the unique identifier of the object to be stored in the second array.

[0047] It can be seen that after calculating another hash function, if there is no identical unique identifier or no array location can be stored, the old unique identifier will be kicked out and the storage will be re-executed.

[0048] In a further embodiment, kicking out the unique identifier in the second array data may also include obtaining a query lock identifier and determining whether the query lock identifier is locked. If so, kicking out the unique identifier in the second array data and placing it in a queue to wait for re-storage; otherwise, kicking out the unique identifier in the second array data and re-storing it.

[0049] It can be seen that the present invention sets a query lock mark. If the query lock mark is locked, it is considered that the data is being locked and queried, that is, the data query request has been received and the data query process is being performed. In order to ensure that the query is more accurate and the unique identifier kicked out can also be queried, the unique identifier kicked out of the second array data is put into the queue and waits for re-storage, so that the queue can be traversed to query the unique identifier. At the same time, parallel processing of data storage and data query is also achieved.

[0050] It is worth noting that, by executing the received request through the thread, the thread is pre-configured with a re-storage number, that is, during the process of executing the request, as long as the unique identifier is kicked out for re-storage, the re-storage number needs to be incremented. Moreover, the present invention monitors the re-storage number of the thread, and if the re-storage number is greater than a preset number threshold, the storage space of the first array and the second array is expanded.

[0051] In a further embodiment, after the unique identifier in the second array data is kicked out, the kick-out status identifier can be obtained and incremented, and after the unique identifier to be kicked out is restored, the kick-out status identifier is obtained and decremented. It can be seen that the present invention sets the kick-out status identifier of a global variable, and when a unique identifier in the array is kicked out, the kick-out status identifier is increased by 1, and when the unique identifier to be kicked out is restored, the kick-out status identifier is decremented by 1.

[0052] As some other embodiments, a data query request may also be received during the data filtering process, and the processing process includes: obtaining the unique identifier of the object to be queried in the request. Calling the first hash function, calculating the first array index according to the unique identifier of the object to be stored, and then determining whether the array data corresponding to the first array index is empty. If so, marking the preset empty identifier, calling the second hash function to cyclically execute the above process to obtain the empty identifier, and returning the result as not found if it is marked twice.

[0053] Otherwise, obtain the unique identifier in the first array data, and determine whether it is the same as the unique identifier of the object to be queried. If it is, return the result as found; if otherwise, call the second hash function to loop and execute the above process and still get a unique identifier that is inconsistent with the object to be queried, then return the result as not found.

[0054] It can be seen that when the first hash function cannot obtain a return result, the second hash function can be called (of course, if the second hash function, i.e., another hash function, is not configured, the result can be directly returned as not found) to query the data.

[0055] In a further embodiment, after the second Hash function is called to cyclically execute the aforementioned process and the unique identifier of the object to be queried is still inconsistent, a query lock identifier can be obtained to determine whether the query lock identifier is unlocked.

[0056] If so, determine whether the kick status flag is 0. If so, return the result as not found; if not, change the query lock flag to locked, and set the kick status flag to 0, traverse the queue to find whether the unique flag of the object to be queried exists in the kicked unique flag. If so, return the result as found, wait for all the kicked unique flags in the queue to be dequeued, change the query lock flag to unlocked, and restore the kick status flag; if otherwise, return the result as not found, wait for all the kicked unique flags in the queue to be dequeued, change the query lock flag to unlocked, and restore the kick status flag.

[0057] If not, the first Hash function and the second Hash function are called cyclically to perform the query until the returned result is that the query has been completed or the query lock mark is unlocked and the returned result is that the query has not been completed.

[0058] It can be seen that in the embodiment of the data query, if the two hash functions cannot find the unique identifier, it is necessary to check three identifiers: the empty identifier, the query lock identifier, and the kick status identifier, which are divided into several situations:

[0059] If the empty tag is marked twice, it is considered non-existent and the returned result is "Not Found".

[0060] The empty flag is not marked twice. If the query lock flag is locked, the first hash function and the second hash function are called cyclically to query until the return result is that the query is found or the query lock flag is unlocked and the return result is not found.

[0061] The empty flag is not marked twice. If the query lock flag is unlocked and the kick status flag is not 0, the query lock flag is changed to locked, and the kick status flag is set to 0. The queue is traversed to find whether the unique flag of the object to be queried exists in the kicked unique flag. If so, the return result is found. After all the kicked unique flags in the queue are dequeued, the query lock flag is changed to unlocked, and the kick status flag is restored. Otherwise, the return result is not found. After all the kicked unique flags in the queue are dequeued, the query lock flag is changed to unlocked, and the kick status flag is restored.

[0062] The empty flag is not marked twice. If the lock flag is unlocked and the kick status flag is 0, it is considered non-existent and the return result is not found.

[0063] Preferably, the processing of the request can be performed by threads.

[0064] It should be noted that when the unique identifier is kicked out from the array, the current kick-out status identifier is incremented, and when the kicked-out unique identifier is stored in the queue, the current kick-out status identifier is decremented. When the kicked-out unique identifier is taken out of the queue and stored again, the current kick-out status identifier is incremented, that is, the kick-out status identifier is restored.

[0065] It is also worth mentioning that if a request for data deletion is received, the processing process of the data query request can be reused, except that when the result is returned as found, the found unique identifier is removed from the array or kicked out to the preset queue.

[0066] Figure 2 is a schematic diagram of the main process of data storage according to the second embodiment of the present invention. The data filtering method may include:

[0067] Step S201: receiving a data storage request, and obtaining a unique identifier of an object to be stored in the request.

[0068] Step S202: calling a first hash function to calculate a first array index according to a unique identifier of the object to be stored.

[0069] Step S203, determining whether the array data corresponding to the first array index is empty, if so, proceeding to step S204, if not, proceeding to step S205.

[0070] Step S204: store the unique identifier of the object to be stored in the corresponding first array, and the request processing is completed.

[0071] Step S205, obtaining the unique identifier in the first array data, and determining whether it is the same as the unique identifier of the object to be stored, if yes, proceeding to step S206, if not, proceeding to step S207.

[0072] Step S206: The unique identifier of the object to be stored is not stored, and the request processing is completed.

[0073] Step S207: calling a second hash function to calculate a second array index according to the unique identifier of the object to be stored.

[0074] Step S208, determining whether the array data corresponding to the second array index is empty, if so, proceeding to step S209, if not, proceeding to step S210.

[0075] Step S209: store the unique identifier of the object to be stored in the corresponding second array, and the request processing is completed.

[0076] Step S210, obtaining the unique identifier in the second array data, and determining whether it is the same as the unique identifier of the object to be stored, if yes, proceeding to step S211, if not, proceeding to step S212.

[0077] Step S211: The unique identifier of the object to be stored is not stored, and the request processing is completed.

[0078] Step S212, obtaining the query lock mark, and determining whether the query lock mark is locked, if so, proceeding to step S213, if not, proceeding to step S214.

[0079] Step S213, kick out the unique identifier in the second array data, put it into the queue and record the number of re-storage times, and return to step S202.

[0080] Step S214, directly kick out the unique identifier in the second array data and record the number of re-storage times, and return to step S202.

[0081] In the embodiment, after the unique identifier in the second array data is kicked out in step S214, the kick-out state identifier can be obtained and incremented, and then, after the unique identifier to be kicked out is re-stored, the kick-out state identifier is obtained and decremented. When the unique identifier in the second array data is kicked out in step S213, the current kick-out state identifier is obtained and incremented, and when the unique identifier is put into the queue, the current kick-out state identifier is obtained and decremented, and all the kicked out unique identifiers in the queue are removed from the queue, and the kick-out state identifier is restored.

[0082] It is worth noting that the present invention pre-configures the number of re-storage times for the thread by executing the received request through the thread, that is, during the process of the thread executing the request, as long as the unique identifier is kicked out for re-storage, the number of re-storage times needs to be automatically increased. Moreover, the present invention monitors the number of re-storage times of the thread, and if the number of re-storage times is greater than the preset number threshold, the storage space of the first array and the second array is expanded. Thus, the dynamic expansion of the array storage space is realized, and resources are saved, and the most efficient resource utilization is achieved.

[0083] Preferably, if the unique identifier needs to be stored in the queue, the number of re-storage times will be automatically increased, and then the unique identifier will be kicked out of the queue for re-storage, and the number of re-storage times will be automatically increased again. For example: the unique identifier is kicked out and stored in the queue, and the number of re-storage times is automatically increased to 5. Then the unique identifier is kicked out of the queue for re-storage, and the number of re-storage times is automatically increased again to 10.

[0084] Figure 3 is a schematic diagram of the main process of data query according to the third embodiment of the present invention. The data filtering method may include:

[0085] Step S301: receiving a data query request and obtaining a unique identifier of an object to be queried in the request.

[0086] Step S302: calling a first hash function to calculate a first array index according to a unique identifier of the object to be stored.

[0087] Step S303, determine whether the array data corresponding to the first array index is empty, if so, proceed to step S304, if not, proceed to step S305.

[0088] Step S304, mark the preset empty mark, and proceed to step S307.

[0089] Step S305, obtaining the unique identifier in the first array data, and determining whether it is the same as the unique identifier of the object to be queried, if yes, proceeding to step S306, if not, proceeding to step S307.

[0090] Step S306, the returned result is "queried".

[0091] Step S307: calling a second hash function to calculate a second array index according to the unique identifier of the object to be stored.

[0092] Step S308, determining whether the array data corresponding to the second array index is empty, if so, proceeding to step S309, if not, proceeding to step S310.

[0093] Step S309: the empty tag is marked twice, and the returned result is not found.

[0094] Step S310, obtaining the unique identifier in the second array data, and determining whether it is the same as the unique identifier of the object to be queried, if yes, proceeding to step S311, if not, proceeding to step S312.

[0095] Step S311, the returned result is "queried".

[0096] Step S312, obtaining the query lock mark, and determining whether the query lock mark is unlocked, if so, proceeding to step S313, if not, returning to step S302.

[0097] Step S313, determine whether the kick-out status flag is 0, if yes, proceed to step S314, if not, proceed to step S315.

[0098] Step S314: Return result: Not found.

[0099] Step S315, change the query lock flag to locked, and set the kick status flag to 0, traverse the queue to find out whether the unique identifier of the object to be queried exists in the kicked unique identifiers, if so, proceed to step S316, if not, proceed to step S317.

[0100] Step S316, the result returned is that the query has been found, and all the unique identifiers kicked out of the queue are dequeued, the query lock identifier is changed to unlocked, and the kick status identifier is restored.

[0101] Step S317, the result returned is not found, wait for all the unique identifiers kicked out of the queue to be dequeued, change the query lock identifier to unlocked, and restore the kick status identifier.

[0102] Figure 4 is a schematic diagram of the main modules of the data filtering device according to an embodiment of the present invention, such as Figure 4 As shown, the data filtering device 400 includes a receiving module 401 that receives a data storage request and obtains a unique identifier of an object to be stored in the request; a processing module 402 calls a first hash function, calculates a first array index based on the unique identifier of the object to be stored, and then determines whether the array data corresponding to the first array index is empty. If so, the unique identifier of the object to be stored is stored in the corresponding first array, and the request processing is completed; if otherwise, the unique identifier in the first array data is obtained, and it is determined whether it is the same as the unique identifier of the object to be stored. If so, the request processing is completed; if otherwise, a second hash function is called, and a second array index is calculated based on the unique identifier of the object to be stored to store the unique identifier of the object to be stored in the corresponding second array.

[0103] In some embodiments, the processing module 402 calls the second hash function, and after calculating the second array index according to the unique identifier of the object to be stored, includes:

[0104] Determine whether the array data corresponding to the second array index is empty, if so, store the unique identifier of the object to be stored in the corresponding second array, and the request processing is completed;

[0105] If not, obtain the unique identifier in the second array data, and determine whether it is the same as the unique identifier of the object to be stored. If so, the request processing is completed; if not, kick out the unique identifier in the second array data to store the unique identifier of the object to be stored in the second array.

[0106] In some embodiments, the processing module 402 kicks out the unique identifier in the second array of data, including:

[0107] Obtain the query lock identifier, and determine whether the query lock identifier is locked. If so, kick out the unique identifier in the second array data and put it into a queue to wait for re-storage; otherwise, kick out the unique identifier in the second array data and re-store it.

[0108] In some embodiments, after the processing module 402 kicks out the unique identifier in the second array of data, the process includes:

[0109] Get the kick status flag and increment it;

[0110] After the unique identifier to be kicked out is restored, the kick status identifier is obtained and decremented.

[0111] In some embodiments, the receiving module 401 receives a data query request and obtains a unique identifier of an object to be queried in the request;

[0112] The processing module 402 calls the first hash function, calculates the first array index according to the unique identifier of the object to be stored, and then determines whether the array data corresponding to the first array index is empty; if so, marks the preset empty identifier, calls the second hash function to cyclically execute the above process to obtain the empty identifier to be marked twice, and returns the result of not found; if otherwise, obtains the unique identifier in the first array data, and determines whether it is the same as the unique identifier of the object to be queried, and if so, returns the result of found; if otherwise, calls the second hash function to cyclically execute the above process and still obtains the unique identifier that is inconsistent with the object to be queried, then returns the result of not found.

[0113] In some embodiments, after the processing module 402 calls the second hash function to cyclically execute the aforementioned process and still obtains a unique identifier that is inconsistent with the object to be queried, it includes:

[0114] Obtaining a query lock flag, and determining whether the query lock flag is unlocked;

[0115] If so, determine whether the kick-out status flag is 0, and if so, return the result as not found; if not, change the query lock flag to locked, and set the kick-out status flag to 0, traverse the queue to find whether there is a unique flag of the object to be queried in the kicked unique flag, and if so, return the result as found, wait for all the kicked unique flags in the queue to be dequeued, change the query lock flag to unlocked, and restore the kick-out status flag; if not, return the result as not found, wait for all the kicked unique flags in the queue to be dequeued, change the query lock flag to unlocked, and restore the kick-out status flag;

[0116] If not, the first Hash function and the second Hash function are called cyclically to perform the query until the returned result is that the query has been completed or the query lock mark is unlocked and the returned result is that the query has not been completed.

[0117] In some embodiments, the receiving module 401 is also used to: execute the received request through a thread; the processing module 402 is also used to: increase the preset restoring times of the thread if the unique identifier is re-stored; monitor the restoring times of the thread, and if the restoring times are greater than the preset times threshold, expand the storage space of the first array and the second array.

[0118] It should be noted that the data filtering method and the data filtering device of the present invention have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.

[0119] Figure 5 An exemplary system architecture 500 is shown to which the data filtering method or data filtering device according to the embodiment of the present invention can be applied.

[0120] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0121] The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 501, 502, 503.

[0122] The terminal devices 501 , 502 , and 503 may be various electronic devices having a data filtering screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0123] The server 505 may be a server that provides various services, such as a background management server (only an example) that provides support for users to use the terminal devices 501, 502, and 503. The background management server may analyze and process the received data such as the product information query request, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0124] It should be noted that the data filtering method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the computing device is generally disposed in the server 505 .

[0125] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0126] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0127] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0128] The following components are connected to the I / O interface 605: an input section 605 including a keyboard, a mouse, etc.; an output section 606 including a cathode ray tube (CRT), a liquid crystal data filter (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0129] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0130] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, it may be described as follows: a processor includes a receiving module and a processing module. The names of these modules do not, in some cases, limit the modules themselves.

[0133] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes receiving a data storage request, obtaining a unique identifier of the object to be stored in the request; calling a first hash function, calculating a first array subscript according to the unique identifier of the object to be stored, and then judging whether the array data corresponding to the first array subscript is empty, if so, storing the unique identifier of the object to be stored in the corresponding first array, and the request processing is completed; if not, obtaining the unique identifier in the first array data, judging whether it is the same as the unique identifier of the object to be stored, if so, the request processing is completed; if not, calling a second hash function, calculating a second array subscript according to the unique identifier of the object to be stored to store the unique identifier of the object to be stored in the corresponding second array.

[0134] The technical solution according to the embodiment of the present invention can solve the problems of low efficiency and high misjudgment rate of existing data filtering.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data filtering method, characterized in that: include: Receive a data storage request, and obtain a unique identifier of an object to be stored in the request; Calling a first hash function, calculating a first array index according to the unique identifier of the object to be stored, and then determining whether the array data corresponding to the first array index is empty, if so, storing the unique identifier of the object to be stored in the corresponding first array, and the request processing is completed; If not, obtain the unique identifier in the first array data, and determine whether it is the same as the unique identifier of the object to be stored. If so, the request processing is completed; if not, call the second hash function, calculate the second array subscript according to the unique identifier of the object to be stored, and store the unique identifier of the object to be stored in the corresponding second array.

2. The method according to claim 1, characterized in that Calling the second hash function to calculate the second array index according to the unique identifier of the object to be stored includes: Determine whether the array data corresponding to the second array index is empty, if so, store the unique identifier of the object to be stored in the corresponding second array, and the request processing is completed; If not, obtain the unique identifier in the second array data, and determine whether it is the same as the unique identifier of the object to be stored. If so, the request processing is completed; if not, kick out the unique identifier in the second array data to store the unique identifier of the object to be stored in the second array.

3. The method according to claim 2, characterized in that Kick out the unique identifier in the second array of data, including: Obtain the query lock identifier, and determine whether the query lock identifier is locked. If so, kick out the unique identifier in the second array data and put it into a queue to wait for re-storage; otherwise, kick out the unique identifier in the second array data and re-store it.

4. The method according to claim 3, characterized in that After the unique identifier in the second array data is kicked out, the method includes: Get the kick status flag and increment it; After the unique identifier to be kicked out is restored, the kick status identifier is obtained and decremented.

5. The method according to claim 1, characterized in that Also includes: Receive a data query request, and obtain a unique identifier of an object to be queried in the request; Calling a first hash function to calculate a first array subscript according to the unique identifier of the object to be stored, and then determining whether the array data corresponding to the first array subscript is empty; If yes, then mark the preset empty mark, call the second Hash function to execute the above process cyclically to obtain the empty mark, and return the result that it is not found after being marked twice; Otherwise, obtain the unique identifier in the first array data, and determine whether it is the same as the unique identifier of the object to be queried. If it is, return the result as found; if otherwise, call the second hash function to loop and execute the above process and still get a unique identifier that is inconsistent with the object to be queried, then return the result as not found.

6. The method according to claim 5, characterized in that After calling the second hash function to cyclically execute the above process and still obtaining a unique identifier inconsistent with the object to be queried, it includes: Obtaining a query lock flag, and determining whether the query lock flag is unlocked; If so, determine whether the kick-out status flag is 0, and if so, return the result as not found; if not, change the query lock flag to locked, and set the kick-out status flag to 0, traverse the queue to find whether there is a unique flag of the object to be queried in the kicked unique flag, and if so, return the result as found, wait for all the kicked unique flags in the queue to be dequeued, change the query lock flag to unlocked, and restore the kick-out status flag; if not, return the result as not found, wait for all the kicked unique flags in the queue to be dequeued, change the query lock flag to unlocked, and restore the kick-out status flag; If not, the first Hash function and the second Hash function are called cyclically to perform the query until the returned result is that the query has been completed or the query lock mark is unlocked and the returned result is that the query has not been completed.

7. The method according to any one of claims 2 to 6, characterized in that: Also includes: Executing the received request through a thread, if the unique identifier is re-stored, then increasing the preset re-storage times of the thread; The number of restoring times of the thread is monitored, and if the number of restoring times is greater than a preset number threshold, the storage space of the first array and the second array is expanded.

8. A data filtering device, characterized in that: include: A receiving module, used to receive a data storage request and obtain a unique identifier of an object to be stored in the request; A processing module is used to call a first hash function, calculate a first array index according to the unique identifier of the object to be stored, and then determine whether the array data corresponding to the first array index is empty. If so, the unique identifier of the object to be stored is stored in the corresponding first array, and the request processing is completed; if not, obtain the unique identifier in the first array data, and determine whether it is the same as the unique identifier of the object to be stored. If so, the request processing is completed; if not, call a second hash function, calculate a second array index according to the unique identifier of the object to be stored, and store the unique identifier of the object to be stored in the corresponding second array.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images