Data screening method and device based on cache, electronic equipment and medium

Abstract

The invention discloses a cache-based data screening method and device, electronic equipment and a medium, and the method comprises the steps: responding to a first resource obtaining instruction of a target object, and randomly generating a random number in a preset range; based on the random number, obtaining a picked resource primary key list and a resource pool resource primary key list of the corresponding data block from the target cache; performing OR operation on the picked resource primary key list and the resource pool resource primary key list to obtain a target data list; randomly acquiring a target data bit of which the numerical value is a second numerical value from the target data list; and querying in the target cache based on the target data bit to obtain the target data. According to the method, full-amount comparison is not needed, only comparison of data with the same block number is needed, and it is also guaranteed that the speed is higher during data comparison and screening. The method can efficiently realize data screening, and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cache-based data screening method, device, electronic equipment and medium. Background Art

[0002] In daily life, people are always performing various search and filtering operations. When the amount of data in the filtering source is too large, it is no longer possible to use simple manual troubleshooting and filtering methods to operate. At this time, some auxiliary tools are needed to search. The existing database search method usually directly adds the query condition not in statement after the SQL query statement to exclude. For a single table with a small amount of data and a small amount of data to be excluded, this is often the best choice. However, when the total amount of data is too large and the amount of data to be excluded increases, this method will have performance problems (when the amount of data in not in reaches the specified number, the database will not use the index, which will result in a full table scan). Summary of the invention

[0003] The present invention proposes a cache-based data screening method, device, electronic device and medium, aiming to solve one of the technical problems in the related art at least to a certain extent. The embodiments of the present invention can efficiently perform data screening.

[0004] On the one hand, an embodiment of the present invention provides a cache-based data screening method, comprising:

[0005] In response to a first resource acquisition instruction of a target object, randomly generating a random number within a preset range;

[0006] Based on the random number, obtain the extracted resource primary key list and resource pool resource primary key list of the corresponding data block from the target cache;

[0007] The target cache includes a preset number of data blocks; the extracted resource master key list and the resource pool resource master key list both include a preset number of data bits; in the extracted resource master key list, a first data state of a data bit represented by a first value is extracted, and a first data state of a data bit represented by a second value is not extracted; in the resource pool resource master key list, a second data state of a data bit represented by a first value is not present, and a second data state of a data bit represented by a second value is not present;

[0008] Performing an OR operation on the extracted resource primary key list and the resource pool resource primary key list to obtain a target data list; randomly obtaining a target data bit whose value is the second value in the target data list;

[0009] The target data is obtained by searching the target cache based on the target data bit.

[0010] Optionally, the method further comprises the following steps:

[0011] The ratio of the total amount of data in all data blocks to the preset number of bits is rounded up to obtain the preset number.

[0012] Optionally, the method further comprises the following steps:

[0013] The preset range is determined based on an integer value range from 1 to a preset number.

[0014] Optionally, a plurality of data to be shared is stored in the target cache, and the primary key values ​​of the resource primary key list of the resource pool are associated with the data to be shared and stored in the target cache using a hash type data structure; querying the target cache based on the target data position to obtain the target data includes the following steps:

[0015] Obtaining a target primary key value according to the product of the first sequence number of the data block corresponding to the resource pool resource primary key list and the preset number of bits and the sum of the second sequence number of the target data bit in the resource pool resource primary key list;

[0016] The target data is retrieved from the target cache based on the target primary key value.

[0017] Optionally, when there is no target data position with the second value in the target data list, the method further comprises the following steps:

[0018] Increment or decrement the random number to update the random number;

[0019] Add 1 to the number of loops, and return to execute the step of obtaining the extracted resource master key list and the resource pool resource master key list of the corresponding data block from the target cache based on the random number, until the value of the target data bit of the resource pool resource master key list is not empty, or the number of loops is greater than a preset loop number threshold;

[0020] When the number of loops is greater than the loop number threshold, resource acquisition ends.

[0021] Optionally, after executing the step of querying the target cache to obtain the target data based on the target data bit, the method further comprises the following steps:

[0022] The value of the target data bit of the extracted resource primary key list is set to the first value.

[0023] Optionally, the target cache stores a plurality of data to be shared, and the primary key value of the resource pool resource primary key list corresponding to each data block is associated with the data to be shared using a hash type data structure and stored in the target cache; the method further comprises the following steps:

[0024] Acquire a second resource acquisition instruction of the target object; the second resource acquisition instruction includes a primary key value of the resource request;

[0025] Determine whether the primary key value exists in the resource primary key list of all resource pools in the target cache through the preset request interception filter;

[0026] When the primary key value does not exist in the primary key list of all resource pool resources in the target cache, the resource acquisition request of the target object is filtered out.

[0027] On the other hand, an embodiment of the present invention provides a cache-based data screening device, comprising:

[0028] A first module is used to randomly generate a random number within a preset range in response to a first resource acquisition instruction of a target object;

[0029] The second module is used to obtain the extracted resource primary key list and the resource pool resource primary key list of the corresponding data block from the target cache based on the random number;

[0030] The target cache includes a preset number of data blocks; the extracted resource master key list and the resource pool resource master key list both include a preset number of data bits; in the extracted resource master key list, a first data state of a data bit represented by a first value is extracted, and a first data state of a data bit represented by a second value is not extracted; in the resource pool resource master key list, a second data state of a data bit represented by a first value is not present, and a second data state of a data bit represented by a second value is not present;

[0031] The third module is used to perform an OR operation on the extracted resource primary key list and the resource pool resource primary key list to obtain a target data list; randomly obtain a target data bit whose value is the second value in the target data list;

[0032] The fourth module is used to query the target cache to obtain the target data based on the target data bit.

[0033] Optionally, the device further comprises:

[0034] The fifth module is used to perform an upward rounding operation on the ratio of the total amount of data of all data blocks to the preset number of bits to obtain a preset number.

[0035] Optionally, the device further comprises:

[0036] A sixth module is used to determine a preset range based on an integer value range from 1 to a preset number.

[0037] Optionally, when there is no target data position whose value is the second value in the target data list, the device further includes:

[0038] The seventh module is used to perform increment or decrement processing on the random number, thereby updating the random number;

[0039] The eighth module is used to increase the number of loops by 1 and return to execute the second module until the value of the target data bit of the resource pool resource master key list is not empty, or the number of loops is greater than a preset loop number threshold;

[0040] The ninth module is used to end resource acquisition when the number of cycles is greater than the cycle number threshold.

[0041] Optionally, the device further comprises:

[0042] The tenth module is used to set the value of the target data bit of the extracted resource master key list to the first value; the tenth module is executed after the fourth module.

[0043] Optionally, a plurality of data to be shared is stored in the target cache, and a primary key value of a resource pool resource primary key list corresponding to each data block is associated with the data to be shared using a hash type data structure and stored in the target cache; the device further comprises:

[0044] The eleventh module is used to obtain a second resource acquisition instruction of the target object; the second resource acquisition instruction includes a primary key value of the resource request;

[0045] The twelfth module is used to determine whether the primary key value exists in the resource primary key list of all resource pools in the target cache through a preset request interception filter;

[0046] The thirteenth module is used to filter out the resource acquisition request of the target object when the primary key value does not exist in the resource primary key list of all resource pools in the target cache.

[0047] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used to store programs; the processor executes the program to implement the above-mentioned cache-based data screening method.

[0048] On the other hand, an embodiment of the present invention provides a computer storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned cache-based data screening method.

[0049] The embodiment of the present invention randomly generates a random number within a preset range in response to a first resource acquisition instruction of a target object; based on the random number, obtains a corresponding data block's extracted resource master key list and a resource pool resource master key list from a target cache; wherein the target cache includes a preset number of data blocks; the extracted resource master key list and the resource pool resource master key list both include a preset number of data bits; in the extracted resource master key list, a first data state of a data bit represented by a first value is extracted, and a first data state of a data bit represented by a second value is not extracted; in the resource pool resource master key list, a second data state of a data bit represented by a first value is non-existent, and a second data state of a data bit represented by a second value is non-existent; an OR operation is performed on the extracted resource master key list and the resource pool resource master key list to obtain a target data list; randomly obtains a target data bit whose value is a second value in the target data list; and obtains target data by querying in the target cache based on the target data bit. The present invention divides the data into blocks of a specified length, and then realizes rapid confirmation of the data status by performing an OR operation on the extracted resource master key list and the resource pool resource master key list, so that each comparison does not require a full comparison, and only the data with the same block number needs to be compared, which also ensures faster data comparison and screening. The present invention can efficiently realize data screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0051] Figure 1 A schematic diagram of an implementation environment for cache-based data screening provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a flow chart of a cache-based data screening method provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of an example of a data block storage bit structure provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of an example of a hash type data structure provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the principle of request interception and filtering provided by an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of a process for request interception and filtering provided by an embodiment of the present invention;

[0057] Figure 7Schematic diagram of the main process of obtaining random resources for existing users;

[0058] Figure 8 A schematic diagram of the implementation principle of cache-based data screening provided by an embodiment of the present invention;

[0059] Fig. 9 A schematic diagram of an implementation flow of cache-based data screening provided by an embodiment of the present invention;

[0060] Fig.10 A schematic diagram of the principle of data comparison provided by an embodiment of the present invention;

[0061] Fig.11 A schematic diagram of the structure of a cache-based data screening device provided by an embodiment of the present invention;

[0062] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0064] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Reference to "embodiments" in the present invention means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present invention may be combined with other embodiments.

[0066] To facilitate the understanding of the technical solution of the present invention, the following is an explanation of the specific names of the technical features that may appear in the technical solution of the present invention:

[0067] Data filtering: specifically refers to excluding a specified part from a bunch of data, similar to taking the difference between two sets.

[0068] like Figure 1FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0069] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0070] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0071] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention.

[0072] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a cache-based data screening method. The following is explained using the cache-based data screening method applied in the terminal 102 as an example. It can be understood that the cache-based data screening method can also be applied to the server 101.

[0073] Reference Figure 2 , Figure 2 The flowchart of the cache-based data screening method applied to the terminal provided by the embodiment of the present invention, the execution subject of the cache-based data screening method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method comprises the following steps:

[0074] S100, in response to a first resource acquisition instruction of a target object, randomly generating a random number within a preset range;

[0075] It should be noted that, in some embodiments, the method may further include the following steps: determining a preset range based on an integer value range from 1 to a preset number.

[0076] Exemplarily, in some specific implementations, a random number r can be first generated in the code, with a value range of 1 to total / 31+(total%31>0?1:0); wherein total represents the total amount of data of all data blocks, and 31 is an optional example of a preset number of bits (which can be adjusted according to actual application requirements).

[0077] S200, based on the random number, obtaining a corresponding data block's extracted resource primary key list and a resource pool resource primary key list from the target cache;

[0078] The target cache includes a preset number of data blocks; the extracted resource master key list and the resource pool resource master key list both include a preset number of data bits; in the extracted resource master key list, a first data state of a data bit represented by a first value is extracted, and a first data state of a data bit represented by a second value is not extracted; in the resource pool resource master key list, a second data state of a data bit represented by a first value is not present, and a second data state of a data bit represented by a second value is not present;

[0079] It should be noted that, in some embodiments, the method may further include the following steps: performing an upward rounding operation on the ratio of the total amount of data of all data blocks to the preset number of bits to obtain a preset number.

[0080] In some embodiments, a plurality of data to be shared is stored in the target cache, and the primary key value of the resource primary key list of the resource pool is associated with the data to be shared using a hash type data structure and stored in the target cache;

[0081] For example, in some specific implementations, the embodiments of the present invention may adopt the following data cache storage method:

[0082] a. Use the bit position of binary value 1 to indicate the numerical state. For example, binary value 1010 indicates that the numbers 2 and 4 exist, and 1100 indicates that the numbers 3 and 4 exist.

[0083] b. To facilitate filtering, use block storage. Assume that the existing 1-100 values ​​need to be stored in the cache. If they are stored as a whole, 100 binary bits are required to represent them, which will make subsequent filtering more difficult. By dividing them into blocks, for example, int type can be used for block storage, and each block stores 31 bits, such as Figure 3As shown, an int type includes 4 bytes and 32 bits. If the sign bit is removed, 31 bits remain. In practical applications, it can also be adaptively adjusted according to actual needs, such as using char type, short type or other format types with a custom number of bits; taking the int type as an example for block division (the subsequent embodiments and related implementation methods uniformly take the int type as an example), it is divided into 100 / 31+(100%31>0?1:0)=4 blocks. Each time filtering is performed, it is only necessary to obtain the data for the block number.

[0084] c.redis uses hash type data structure, such as Figure 4 As shown, the number of bits with hashKey=0 can be used to store the maximum value (total, i.e., the total number of data) currently represented. The block where the value n is located is: n / 31+(n%31>0?1:0), and the i-th bit of the block is converted to the value: i=(n%31==0)?31:n%31, and the position i is counted from right to left, and the starting value is 1.

[0085] d. The data that the user has extracted (corresponding to the extracted resource master key list) and the shared total data (corresponding to the resource pool resource master key list) are both stored in the above manner, but the binary values ​​have different meanings. The binary 1 (i.e., the first value) of the data that the user has extracted represents the extracted data, and 0 (i.e., the second value) represents the data that has not been extracted. The binary 1 of the shared total data represents non-existent data, and 0 represents existing data. In other implementations, if a custom data type is used, the first value and the second value can also be set by yourself. It is only necessary to ensure that the second value corresponds to the unextracted state in the extracted data and exists in the shared total data, so that the subsequent OR operation results of the two lists can be consistent.

[0086] S300, performing an OR operation on the extracted resource primary key list and the resource pool resource primary key list to obtain a target data list; randomly obtaining a target data bit whose value is a second value in the target data list;

[0087] Exemplarily, in some specific implementations, taking a random number r as an example and taking binary 1 and 0 to represent data status as an example, first obtain the data n corresponding to the r block number of the total data (corresponding to the resource pool resource master key list) and the data m corresponding to the r block number of the extracted data (corresponding to the extracted resource master key list); then, perform an OR (|) operation on n and m, and then convert the result into binary data. At this time, in the binary data, 0 represents data that can be extracted, and 1 represents data that has been extracted or does not exist. Take any value as 0 and record its position i.

[0088] S400 , querying the target cache based on the target data bit to obtain the target data.

[0089] It should be noted that a plurality of data to be shared are stored in the target cache, and the primary key values ​​of the resource pool resource primary key list are associated with the data to be shared and stored in the target cache using a hash type data structure; in some embodiments, obtaining the target data based on the target data bit by querying in the target cache may include the following steps: obtaining the target primary key value based on the product of the first serial number of the data block corresponding to the resource pool resource primary key list and the number of preset bits and the sum of the second serial number of the target data bit in the resource pool resource primary key list; obtaining the target data based on the target primary key value by querying in the target cache.

[0090] For example, in some specific implementations, taking the application scenario of the above specific implementation as an example, i can then be converted into a real value (i.e., the key value corresponding to the hash type data structure): (r-1)*31+i. Then the database (corresponding to the target cache) can be queried for the data associated with the corresponding value through this value.

[0091] Optionally, in some embodiments, when there is no target data bit with the second value in the target data list, the method may further include the following steps: incrementing or decrementing the random number to update the random number; adding 1 to the number of loops, returning to execute the step of obtaining the corresponding data block's extracted resource master key list and resource pool resource master key list from the target cache based on the random number, until the value of the target data bit in the resource pool resource master key list is not empty, or the number of loops is greater than a preset loop number threshold; when the number of loops is greater than the loop number threshold, ending resource acquisition.

[0092] Exemplarily, in some specific implementations, taking the application scenario of the aforementioned specific implementation as an example, if the binary data corresponding to the result of the OR operation is all 1, it indicates that all the data corresponding to the current data block have been obtained, then r=r±1 or a new random number r can be generated to re-screen the data.

[0093] Optionally, in some embodiments, the method may further include the following steps: setting the value of the target data bit of the extracted resource primary key list to a first value.

[0094] For example, in some specific implementations, taking the application scenario of the above specific implementation as an example, the user has extracted data cache (corresponding to the extracted resource primary key list): each time the user extracts new data, obtain its resource primary key n, calculate its block number m = n / 31 + (n%31>0?1:0) and position i in the block = (n%31==0)?31:n%31. Obtain the cache value corresponding to block m, and set the value of position i to 1.

[0095] In some preferred implementations, when the cache of the total shared data is newly added: after each new share is added, the value of block 0 of the total data cache is changed to the current shared primary key. The block number m1 and the position i1 in the block are calculated (the same principle as the change of the primary key list of the extracted resources mentioned above), the cache value corresponding to the m1 block is obtained, and the value of the i1 position (i.e. the key corresponding to the data block) is set to 0.

[0096] It should be understood that when the cache of the shared total data is deleted: its logic is consistent with the newly added step, the difference is that when setting the value of the binary specified bit value, it needs to be set to 1.

[0097] Optionally, in some embodiments, a plurality of data to be shared are stored in the target cache, and the primary key value of the resource pool resource primary key list corresponding to each data block is associated with the data to be shared and stored in the target cache using a hash type data structure; the method may further include the following steps: obtaining a second resource acquisition instruction for the target object; the second resource acquisition instruction includes the primary key value of the resource request; determining whether a primary key value exists in all resource pool resource primary key lists in the target cache through a preset request interception filter; when no primary key value exists in all resource pool resource primary key lists in the target cache, filtering out the resource acquisition request of the target object.

[0098] For example, in some specific implementations, data can be directly filtered according to the data query instructions for specific data bits of the target object. However, in some specific application scenarios, there is often a risk of malicious attacks for interfaces facing the public network, especially for some interfaces that obtain details by data primary keys. If the attacker changes the primary key value every time a request is made, even if the data details are cached, each request will still access the database, causing excessive pressure on the database and chain effects on service availability. Figure 5 and Figure 6 As shown, at this time, you can add a request interception filter to filter out requests whose values ​​do not exist in the database, so as to protect the security of the application.

[0099] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0100] First of all, it should be noted that currently when retrieving and excluding specific parts of data from a relational database (for example, assuming there are two tables A and B, table A stores the shared data of all users, and table B stores the records of all users obtaining other people's shared data. When a user obtains shared resources again, it is necessary to exclude the resources that he has already obtained). Figure 7As shown in the figure, the simple way is to add the query condition not in statement directly after the SQL query statement to exclude. This is often the best choice for a single table with a small amount of data and a small amount of data to be excluded. However, when the total amount of data is too large and the amount of data to be excluded increases, this method will have performance problems (when the amount of data in not in reaches the specified number, the database will not use the index, which will lead to a full table scan), which is not a good solution.

[0101] In view of this, the present invention proposes a new data screening scheme to avoid the slow interface response speed caused by the full table scan during the program running process, so as to ensure the quick response and normal operation of the program. Specifically, when a user shares resources, a shared resource record will be added to the sharing table, and the table primary key uses a continuous long integer value (this is to ensure that the primary key data sparsity rate is low and save storage space when all primary keys are cached). When a user extracts a shared resource, an extraction record will be inserted into the extraction table. When subsequent users continue to extract resources, the extracted resources must be excluded. The present invention stores all sharing record IDs and each user's extracted resource records in the cache, so that each time a user extracts a resource, it can quickly respond and return new resources. Figure 8 and Fig. 9 As shown, its specific implementation is as follows:

[0102] 1) How data is stored in the cache:

[0103] a. Use the bit position of binary value 1 to indicate the numerical state. For example, binary value 1010 indicates that the numbers 2 and 4 exist, and 1100 indicates that the numbers 3 and 4 exist.

[0104] b. To facilitate filtering, use block storage. Assume that the existing 1-100 values ​​need to be stored in the cache. If they are stored as a whole, 100 binary bits are required to represent them, which will make subsequent filtering more difficult. By dividing them into blocks, for example, int type can be used for block storage, and each block stores 31 bits, such as Figure 3 As shown, an int type includes 4 bytes and 32 bits. If the sign bit is removed, 31 bits remain. In practical applications, it can also be adaptively adjusted according to actual needs, such as using char type, short type or other format types with a custom number of bits; taking the int type as an example for block division (the subsequent embodiments and related implementation methods uniformly take the int type as an example), it is divided into 100 / 31+(100%31>0?1:0)=4 blocks. Each time filtering is performed, it is only necessary to obtain the data for the block number.

[0105] c.redis uses hash type data structure, such as Figure 4As shown, the number of bits with hashKey=0 can be used to store the maximum value (total, i.e., the total number of data) currently represented. The block where the value n is located is: n / 31+(n%31>0?1:0), and the i-th bit of the block is converted to the value: i=(n%31==0)?31:n%31, and the position i is counted from right to left, and the starting value is 1.

[0106] d. The data that the user has extracted (corresponding to the extracted resource master key list) and the shared total data (corresponding to the resource pool resource master key list) are both stored in the above manner, but the binary values ​​have different meanings. The binary 1 (i.e., the first value) of the data that the user has extracted represents the extracted data, and 0 (i.e., the second value) represents the data that has not been extracted. The binary 1 of the shared total data represents non-existent data, and 0 represents existing data. In other implementations, if a custom data type is used, the first value and the second value can also be set by yourself. It is only necessary to ensure that the second value corresponds to the unextracted state in the extracted data and exists in the shared total data, so that the subsequent OR operation results of the two lists can be consistent.

[0107] 2) Get a random resource:

[0108] a. First generate a random number r in the code, with a value range of 1 to total / 31+(total%31>0?1:0).

[0109] b. Obtain the data n corresponding to the block number r of the total data (i.e. the total shared data) and the data m corresponding to the block number r of the extracted data.

[0110] c. Perform an OR (|) operation on n and m, and convert the result into binary data. In this binary data, 0 indicates data that can be extracted, and 1 indicates data that has been extracted or does not exist. Take any value as 0 and record its position i.

[0111] d. Convert i to a real value: (r-1)*31+i. Then use this value to query the database.

[0112] e. If the binary numbers obtained in step c are all 1, you can set r = r ± 1 or generate a new random number r to repeat steps b and c.

[0113] 3) Real-time changes to cached data:

[0114] a. Changes to the user's extracted data cache: Each time the user extracts new data, obtain its resource primary key n, calculate its block number m = n / 31 + (n%31>0?1:0) and position i in the block = (n%31==0)?31:n%31. Obtain the cache value corresponding to block m and set the value of position i to 1.

[0115] b. Addition of total data cache: After each new share is added, change the value of block 0 of the total data cache to the primary key of the current share. Calculate the block number m1 and the position i1 in the block (same as step a), obtain the cache value corresponding to block m1, and set the value of position i1 to 0.

[0116] c. Cache deletion of total data: Its logic is the same as step b, except that when setting the value of the binary specified bit value, it needs to be set to 1.

[0117] The technical solution of the present invention is further described below in conjunction with some specific application scenarios:

[0118] 1. Data comparison and screening: In program applications, each time a user extracts resources from a resource pool, the user must exclude the resources that the user has already extracted and only select one resource that the user has not extracted (which can be abstracted as the difference between two sets). When a user has extracted a large number of resources, the filtering speed using the database not in statement will be slow. Therefore, the design is to store the resource pool primary key and the primary key of the resource that the user has extracted in the cache. Each time a resource is extracted, the primary key of the resource that the user has not extracted is directly obtained from the cache for comparison, and the database is searched and returned based on it.

[0119] like Fig.10 As shown, each block stores n binary numbers, with binary 1 indicating the existence of a value and 0 indicating the absence of a value. By comparing the data of the same block number of the two data sets, the difference set can be obtained. Finally, the true value can be obtained by conversion: the i-th bit of the n-th block is converted to a numerical value: (n-1)*31+i.

[0120] 2. Request interception and filtering: For interfaces facing the public network, there is often a risk of malicious attacks, especially for some interfaces that use data primary keys to obtain details. If the attacker changes the primary key value every time a request is made, even if the data details are cached, each request will still access the database, causing excessive pressure on the database and affecting the availability of the service. Figure 5 and Figure 6 As shown, at this time, you can add a request interception filter to filter out requests whose values ​​do not exist in the database, so as to protect the security of the application.

[0121] In summary, the purpose of the present invention is to solve the hidden danger that the speed of users acquiring random resources slows down when the amount of data increases. Using cache to store data can quickly achieve data acquisition; the special binary bit representation of numerical value helps to save cache server memory resources and ensure the storage of more data; at the same time, the data is divided into blocks with specified lengths, and each data block is stored using the hash data structure of redis, so that each comparison does not require a full comparison, and only the comparison of data with the same block number is required, which also ensures faster data comparison and screening.

[0122] Compared with the prior art, the present invention has at least the following beneficial effects:

[0123] The special binary digit method of representing numerical values ​​saves the memory resources of the cache server, and the hash data structure of the redis framework is used, so that each comparison does not require a full comparison, but only a part of it. This not only reduces the complexity of the program code, but also greatly speeds up the response speed. In the future, a function similar to the Bloom filter can be expanded based on this method to identify which data is valid and which data is invalid or does not exist. This can effectively protect the system and prevent malicious requests from reaching the database directly and causing the application to crash.

[0124] On the other hand, Fig.11 As shown, an embodiment of the present invention provides a cache-based data screening device 900, comprising:

[0125] A first module is used to randomly generate a random number within a preset range in response to a first resource acquisition instruction of a target object;

[0126] The second module is used to obtain the extracted resource primary key list and the resource pool resource primary key list of the corresponding data block from the target cache based on the random number;

[0127] The target cache includes a preset number of data blocks; the extracted resource master key list and the resource pool resource master key list both include a preset number of data bits; in the extracted resource master key list, a first data state of a data bit represented by a first value is extracted, and a first data state of a data bit represented by a second value is not extracted; in the resource pool resource master key list, a second data state of a data bit represented by a first value is not present, and a second data state of a data bit represented by a second value is not present;

[0128] The third module is used to perform an OR operation on the extracted resource primary key list and the resource pool resource primary key list to obtain a target data list; randomly obtain a target data bit whose value is the second value in the target data list;

[0129] The fourth module is used to query the target cache to obtain the target data based on the target data bit.

[0130] In some embodiments, the apparatus may further include:

[0131] The fifth module is used to perform an upward rounding operation on the ratio of the total amount of data of all data blocks to the preset number of bits to obtain a preset number.

[0132] In some embodiments, the apparatus may further include:

[0133] A sixth module is used to determine a preset range based on an integer value range from 1 to a preset number.

[0134] In some embodiments, when there is no target data bit with the second value in the target data list, the apparatus may further include:

[0135] The seventh module is used to perform increment or decrement processing on the random number, thereby updating the random number;

[0136] The eighth module is used to increase the number of loops by 1 and return to execute the second module until the value of the target data bit of the resource pool resource master key list is not empty, or the number of loops is greater than a preset loop number threshold;

[0137] The ninth module is used to end resource acquisition when the number of cycles is greater than the cycle number threshold.

[0138] In some embodiments, the apparatus may further include:

[0139] The tenth module is used to set the value of the target data bit of the extracted resource master key list to the first value; the tenth module is executed after the fourth module.

[0140] In some embodiments, a plurality of data to be shared is stored in the target cache, and a primary key value of a resource pool resource primary key list corresponding to each data block is associated with the data to be shared using a hash type data structure and stored in the target cache; the device further comprises:

[0141] The eleventh module is used to obtain a second resource acquisition instruction of the target object; the second resource acquisition instruction includes a primary key value of the resource request;

[0142] The twelfth module is used to determine whether the primary key value exists in the resource primary key list of all resource pools in the target cache through a preset request interception filter;

[0143] The thirteenth module is used to filter out the resource acquisition request of the target object when the primary key value does not exist in the resource primary key list of all resource pools in the target cache.

[0144] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] On the other hand, an embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above sensitive information method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0146] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] like Fig.12 As shown, Fig.12 A specific example of the hardware structure of an electronic device 1000 according to an embodiment is shown. The electronic device 1000 includes:

[0148] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0149] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the network node population optimization method of the embodiment of the present invention;

[0150] Input / output interface 1003, used to implement information input and output;

[0151] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0152] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0153] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0154] The electronic device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0156] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0157] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read to Only Memory, CD to ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0158] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0162] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD to ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.

[0163] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented by the present invention. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0164] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of various functional modules in the device disclosed in the present invention, the actual implementation of the module will be understood within the conventional skills of engineers. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0165] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0166] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution device, apparatus or device (such as a computer-based device, a device including a processor, or other device that can fetch instructions from an instruction execution device, apparatus or device and execute the instructions), or in conjunction with such instruction execution device, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution device, apparatus or device, or in conjunction with such instruction execution device, apparatus or device.

[0167] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0168] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0169] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0170] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0171] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A cache-based data screening method, characterized in that: The following steps are involved: In response to a first resource acquisition instruction of a target object, randomly generating a random number within a preset range; Based on the random number, obtaining a corresponding data block's extracted resource primary key list and a resource pool resource primary key list from the target cache; wherein the target cache includes a preset number of the data blocks; the extracted resource master key list and the resource pool resource master key list both include a preset number of data bits; in the extracted resource master key list, a first data state of the data bit represented by a first value is extracted, and a first data state of the data bit represented by a second value is not extracted; in the resource pool resource master key list, a second data state of the data bit represented by the first value is not present, and a second data state of the data bit represented by the second value is not present; Performing an OR operation on the extracted resource primary key list and the resource pool resource primary key list to obtain a target data list; randomly obtaining a target data bit whose value is the second value in the target data list; The target data is obtained by searching the target cache based on the target data bit.

2. The cache-based data screening method according to claim 1, characterized in that: The method further comprises the following steps: The ratio of the total amount of data of all the data blocks to the preset number of bits is rounded up to obtain the preset number.

3. The cache-based data screening method according to claim 1 or 2, characterized in that: The method further comprises the following steps: The preset range is determined based on an integer value range from 1 to the preset number.

4. The cache-based data screening method according to claim 1, characterized in that: The target cache stores a plurality of data to be shared, and the primary key value of the resource pool resource primary key list is associated with the data to be shared using a hash type data structure and stored in the target cache; The step of querying the target cache to obtain the target data based on the target data bit includes the following steps: Obtaining a target primary key value according to the product of the first sequence number of the data block corresponding to the resource pool resource primary key list and the preset number of bits and the sum of the second sequence number of the target data bit in the resource pool resource primary key list; The target data is obtained by querying the target cache based on the target primary key value.

5. The cache-based data screening method according to claim 1, characterized in that: When the target data list does not contain the target data position whose value is the second value, the method further comprises the following steps: Incrementing or decrementing the random number, thereby updating the random number; Increasing the number of loops by 1, and returning to the step of obtaining the extracted resource master key list and the resource pool resource master key list of the corresponding data block from the target cache based on the random number, until the value of the target data bit of the resource pool resource master key list is not empty, or the number of loops is greater than a preset loop number threshold; When the number of cycles is greater than the cycle number threshold, resource acquisition is terminated.

6. The cache-based data screening method according to claim 1, characterized in that: After executing the step of querying the target cache to obtain the target data based on the target data bit, the method further comprises the following steps: The value of the target data bit of the extracted resource primary key list is set to the first value.

7. The cache-based data screening method according to claim 1, characterized in that: The target cache stores a plurality of data to be shared, and the primary key value of the resource pool resource primary key list corresponding to each data block is associated with the data to be shared using a hash type data structure and stored in the target cache; the method further includes the following steps: Acquire a second resource acquisition instruction of the target object; the second resource acquisition instruction includes a primary key value of a resource request; Determine whether the primary key value exists in all resource primary key lists of the resource pools in the target cache through a preset request interception filter; When the primary key value does not exist in all resource primary key lists of the resource pools in the target cache, the resource acquisition request of the target object is filtered out.

8. A cache-based data screening device, characterized in that: include: A first module is used to randomly generate a random number within a preset range in response to a first resource acquisition instruction of a target object; The second module is used to obtain the extracted resource primary key list and the resource pool resource primary key list of the corresponding data block from the target cache based on the random number; wherein the target cache includes a preset number of the data blocks; the extracted resource master key list and the resource pool resource master key list both include a preset number of data bits; in the extracted resource master key list, a first data state of the data bit represented by a first value is extracted, and a first data state of the data bit represented by a second value is not extracted; in the resource pool resource master key list, a second data state of the data bit represented by the first value is not present, and a second data state of the data bit represented by the second value is not present; The third module is used to perform an OR operation on the extracted resource primary key list and the resource pool resource primary key list to obtain a target data list; and randomly obtain a target data bit whose value is the second value in the target data list; The fourth module is used to query the target cache to obtain the target data based on the target data position.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images