A retrieval method, device, storage medium and terminal based on an improved MinHash algorithm

Abstract

The present invention relates to a retrieval method, device, storage medium and terminal based on an improved MinHash algorithm. The retrieval method includes: S1. Download m binary component libraries of different types and different versions to obtain a component set S; S2. Extract features for each component in the component set S respectively; S3. Calculate the weight value of the eigenvalue through the feature weight assignment module, and calculate the component hash value through the MinHash algorithm. If the minimum hash value selects the feature f i , while recording its hash value, associate its feature weight value w(f i ); S4. Then calculate the similarity between two components through the similarity calculation formula. The present invention can more accurately identify the similarity of different components and has a wider adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of data security, and particularly to a retrieval method, device, storage medium and terminal based on an improved MinHash algorithm. Background Art

[0002] In the similarity detection of documents, web pages, binary files, etc., it is often necessary to calculate the similarity of sets. In order to calculate the set similarity quickly and efficiently, currently traditional methods all use the MinHash algorithm to obtain the locality-sensitive hash values of the original set objects, use the similarity of the hash values to approximate the similarity of the original objects, and use the b-bit MinHash algorithm to extract only b bits from each hash value and splice them into the final hash value.

[0003] However, in the existing methods, the feature weight values are not distinguished, resulting in the same weights for important feature hash values and ordinary features. In this case, only the number of features can be relied on to calculate the similarity, while ignoring the information contained in the features themselves; for example, in the binary detection scenario, some features only appear in specific binary components and their versions, while some other features may appear in many different components. The two should have different weights. Obviously, the former contains more information and is more discriminative; as Figure 1 shown, the existing methods cannot distinguish the similarity. Assume that the number of features in sets A, B, and C is equal, and the number of overlapping features is also equal, that is, A∪B = B∪C and A∩B = B∩C. From the number of overlapping features, the existing technical solutions cannot distinguish sim(A, B) from sim(B, C), but if the information contained in the overlapping features of sets A and B is lower, different similarities should be given to these two sets of sets; therefore, how to solve the problems existing in the existing methods is what needs to be considered at the present stage.

[0004] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a retrieval method, device, storage medium and terminal based on an improved MinHash algorithm, which solves the deficiencies existing in the existing traditional methods.

[0006] The purpose of the present invention is achieved through the following technical solutions: A retrieval method based on an improved MinHash algorithm, the retrieval method comprising:

[0007] S1. Download m binary component libraries of different types and different versions to obtain a component set S;

[0008] S2. Extract features from each component in the component set S respectively;

[0009] S3. Calculate the weight value of the eigenvalue through the feature weighting module, calculate the component hash value through the MinHash algorithm. If the minimum hash value selects the feature f i , then while recording its hash value, associate its feature weight value w(f i );

[0010] S4. Then calculate the similarity between two components through the similarity calculation formula.

[0011] The retrieval method further includes an indexing step, which is executed between S3 and S4. The indexing step includes: Suppose a total of 6 minimum hash values are generated, and each minimum hash value extracts 4 bits, that is, a total of 24 - bit bits are generated. Then the 24 - bit hash value is divided into 3 parts, and each 8 - bit corresponds to a bucket number. A total of 3 hash tables are generated, and each hash table contains 256 buckets. The hash value with a decimal value of 155 corresponding to the first 8 bits is assigned to bucket numbers 1 to 155 in hash table 1, forming a list. The hash values within the bucket have high similarity, and the hash values between buckets have low similarity. When retrieving, only the candidate values under specific buckets need to be compared, which reduces the calculation amount.

[0012] The specific process of calculating the weight value of the eigenvalue through the feature weighting module includes:

[0013] Suppose a certain feature f i appears k times in a certain component, and n eigenvalues appear a total of t times. Then tf(f i ) = k / t, where tf represents the term frequency. If the feature f i appears in x components, then idf(f i ) = log(m / x), where idf represents the inverse document frequency. The weight value w1(f i ) corresponding to the feature f i ) = k / t * log(m / x);

[0014] Suppose the number of times the feature f i appears in all components is C = {c1, c2, c3,..., cm}. Use the random variable Y to represent the number of times the feature f i appears in the component. Then the information entropy of the introduced random variable Y is expressed as H(Y) = ∑ j P(cj) * log(P(cj)), where P(cj) represents the probability of cj, that is, the number of times the value cj appears in C divided by the total number of C. Suppose the coverage rate acd(f i ) = x / m of the feature f i ), and obtain the weight value w2 = acd(fi) * H(Y);

[0015] Get the final feature weight value w(f i ) = alpha * w1 + beta * w2; where both alpha and beta are empirically set values by humans.

[0016] The calculation of the similarity between two components through the similarity calculation formula specifically includes:

[0017] Let the fingerprint of component A be the set {a1, a2, ···, a m}}, and the fingerprint corresponding to component B be the set {b1, b2, ···, b m}}. The weight value of each feature can be obtained through the feature weighting module. When generating the MinHash fingerprint, the weight of the corresponding feature is retained. Then the feature vectors corresponding to the two components are {w a,1 , w a,2 , ···, w a,m}, {w b,1 , w b,2 , ···, w b,m} respectively. Then the similarity P between the two components is calculated through the similarity calculation formula , where T is the subscript set of the fingerprints that are the same for components A and B.

[0018] A retrieval device based on an improved MinHash algorithm, which includes a component set module, a feature extraction module, a feature weighting module, and a similarity calculation module;

[0019] The component set module: is used to download m binary component libraries of different types and different versions to obtain a component set S;

[0020] The feature extraction module: is used to perform feature extraction on each component in the component set S respectively;

[0021] The feature weighting module: is used to calculate the weight value of the feature value. By calculating the hash value of a component through the MinHash algorithm, if the minimum hash value selects the feature f i , then while recording its hash value, its feature weight value w(f i ) is associated;

[0022] The similarity calculation module: is used to calculate the similarity between two components through the similarity calculation formula.

[0023] The system also includes an index construction module, which is used to assume that a total of 6 minimum hash values are generated, and 4 bits are extracted from each minimum hash value, that is, a total of 24 bits are generated. The 24-bit hash value is divided into 3 parts, and each 8 bits corresponds to a bucket number. A total of 3 hash tables are generated, each hash table contains 256 buckets, and the first 8 bits correspond to a hash value with a decimal value of 155, which is assigned to buckets 1 to 155 of the hash table to form a list. The hash values within a bucket have high similarity, and the hash values between buckets have low similarity. When searching, only the candidate values under a specific bucket need to be compared, which reduces the amount of calculation.

[0024] The feature weighting module includes a first feature weight calculation unit, a second feature weight calculation unit and a feature weight calculation unit;

[0025] The first feature weight calculation unit is used to set a feature f in a component i appears k times, and n eigenvalues appear t times in total, then tf(f i )=k / t, tf represents word frequency, if feature f i appears in x components, then idf(f i )=log(m / x), idf represents inverse document frequency, feature f i The corresponding weight value w1(f i )=k / t*log(m / x);

[0026] The second feature weight calculation unit is used to set the feature f i The number of occurrences in all components is C = {c1, c2, c3, ..., cm}, and the random variable Y is used to represent the feature f i The number of times it appears in the component, the information entropy of the random variable Y is expressed as H(Y)=∑ j P(cj)*log(P(cj)), where P(cj) represents the probability of cj, that is, the number of times the value cj appears in C divided by the total number of C. Let the feature f i The coverage rate acd(f i )=x / m, and get the feature weight value w2=acd(fi)*H(Y);

[0027] The feature weight calculation unit is used to calculate the feature weight by formula w(f i )=alpha*w1+beta*w2 to obtain the final feature weight value, where alpha and beta are both manually set empirical values.

[0028] A computer-readable storage medium stores a computer program, which implements the steps of the retrieval method when executed by a processor.

[0029] A terminal device includes a memory and a processor. A computer program is stored on the memory. When the processor executes the computer program, the steps of the retrieval method are implemented.

[0030] The present invention has the following advantages: A retrieval method, device, storage medium and terminal based on an improved MinHash algorithm. In the feature set extraction stage, a feature weight calculation module is introduced. Considering the coverage rate, frequency and distribution of features, weights are assigned to features, and greater weights are assigned to more important features. When generating MinHash values, the weights of features are assigned to the generated hash values. A new set of hash values corresponds to a set of corresponding feature weights, and these weights will be used to calculate the set similarity, which can more accurately identify the similarity of different components and has a wider adaptability. Description of the Drawings

[0031] Figure 1 A schematic diagram for similarity judgment of the existing method;

[0032] Figure 2 A schematic flowchart of the method of the present invention;

[0033] Figure 3 A schematic diagram of feature extraction of the present invention;

[0034] Figure 4 A schematic diagram of an example of index construction of the present invention;

[0035] Figure 5 A schematic diagram of an example of retrieval of the present invention. Detailed Embodiments

[0036] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application provided below with reference to the accompanying drawings is not intended to limit the protection scope of the present application claimed, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present application. The present invention will be further described below with reference to the accompanying drawings.

[0037] One implementation manner of the present invention relates to a component / set similarity retrieval method based on an improved MinHash algorithm, which can be used in common set search strategies, such as the retrieval of files by a search engine, the retrieval of binary components in SCA, etc.

[0038] As shown in Figure 2 the figure, the specific method includes the following:

[0039] S1. Download m binary component libraries of different types and versions to obtain a component set S;

[0040] S2. Extract features for each component in the component set S; including but not limited to string constants, method call order, class inheritance structure, class names used, etc. As shown in Figure 3 the figure, the component with ID101 extracts n feature values;

[0041] S3. Calculate the weight values of the feature values through the feature weighting module, calculate the component hash values through the MinHash algorithm. If the minimum hash value selects the feature f i , then while recording its hash value, associate its feature weight value w(f i );

[0042] S4. Then calculate the similarity between two components through the similarity calculation formula.

[0043] Furthermore, in order to quickly retrieve according to the hash value, an index needs to be established. Therefore, the retrieval method also includes an index construction step. The index construction step mainly lies in bucketing, evenly distributing all the hash values into multiple buckets as much as possible. The hash values within a bucket have high similarity, and the similarity between buckets is low. During retrieval, according to the needs of the bucket, only the candidate values under a specific bucket need to be compared, which reduces the calculation amount.

[0044] The index construction step includes: As shown in Figure 4 the figure, assume that the minimum hash value is generated 6 times in total, and each minimum hash value extracts 4 bits, that is, a total of 24 bits are generated. Then the 24-bit hash value is divided into 3 parts, and each 8 bits corresponds to a bucket number. A total of 3 hash tables are generated, and each hash table contains 256 buckets. The hash value with a decimal value of 155 corresponding to the first 8 bits is assigned to bucket No. 155 in hash table 1, forming a list. Each table contains all the hash values, that is, the original data needs to be stored additionally by 2 times.

[0045] As shown in Figure 5 the figure, take an example to illustrate the retrieval process. For the component to be retrieved, after feature extraction and construction, the hash value is obtained. Calculate the bucket numbers corresponding to the hash value in different hash tables respectively, return all the candidate values under the matching buckets, calculate the similarity respectively, filter according to the business requirements, and sort to return the top-k candidate values.

[0046] Furthermore, the feature weighting module calculates the weight of the feature value, specifically including: using the TF-IDF algorithm to calculate the importance w1 of the feature value, and counting the number of components in which the feature appears and the frequency ratio of the feature in the component;

[0047] Suppose a feature f in a component i appears k times, and n eigenvalues appear t times in total, then tf(f i )=k / t, tf represents word frequency, if feature f i appears in x components, then idf(f i )=log(m / x), idf represents inverse document frequency, feature f i The corresponding weight value w1(f i )=k / t*log(m / x);

[0048] Let feature f i The number of occurrences in all components is C = {c1, c2, c3, ..., cm}, and the random variable Y is used to represent the feature f i The number of times it appears in the component, the information entropy of the random variable Y is expressed as H(Y)=∑ j P(cj)*log(P(cj)), when the value distribution of the random variable is more different, the information entropy is larger, the value is more concentrated, and the information entropy is smaller. P(cj) represents the probability of cj, that is, the number of times the value cj appears in C divided by the total number of C. Let the feature f i The coverage rate acd(f i )=x / m, and get the feature weight value w2=acd(fi)*H(Y);

[0049] The traditional TF-IDF algorithm has some shortcomings, such as being susceptible to data skew. For example, if a component has many versions and a certain feature appears in all versions, even though the feature's quantity distribution is discriminative, the IDF of that feature will still be underestimated. To correct this effect, this paper introduces information entropy.

[0050] Get the final feature weight value w(f i )=alpha*w1+beta*w2; where alpha and beta are both artificially set empirical values.

[0051] Furthermore, the similarity between two components is calculated using the similarity calculation formula, specifically including:

[0052] Suppose the fingerprint of component A is the set {a1,a2,···,a m}, the fingerprint corresponding to component B is the set {b1,b2,···,b m}, the weight value of each feature can be obtained through the feature weighting module. When generating the MinHash fingerprint, the weights of the corresponding features are retained, and the feature vectors corresponding to the two components are respectively {w a,1 , w a,2 , ···, w a,m}, {w b,1 , w b,2 , ···, w b,m}. Then, the similarity P between the two components is calculated through the similarity calculation formula . Among them, T is the set of subscripts of the same fingerprints of components A and B.

[0053] Another embodiment of the present invention relates to a component / collection similarity retrieval device based on an improved MinHash algorithm, which includes a component collection module, a feature extraction module, a feature weighting module, and a similarity calculation module;

[0054] The component collection module: is used to download m binary component libraries of different types and different versions to obtain a component collection S;

[0055] The feature extraction module: is used to extract features from each component in the component collection S respectively;

[0056] The feature weighting module: is used to calculate the weight value of the feature value. If the minimum hash value selects the feature f i when calculating the hash value of a certain component through the MinHash algorithm, while recording its hash value, the feature weight value w(f i ) is associated;

[0057] The similarity calculation module: is used to calculate the similarity between two components through the similarity calculation formula.

[0058] It further includes an index construction module. The index construction module: is used to generate a total of 6 minimum hash values. Each minimum hash value extracts 4 bits, that is, a total of 24 - bit hash values are generated. Then the 24 - bit hash values are divided into 3 parts, each 8 - bit corresponding to a bucket number, and a total of 3 hash tables are generated. Each hash table contains 256 buckets. The hash value corresponding to the decimal value of 155 for the first 8 bits is assigned to bucket numbers 1 to 155 in hash table 1, forming a list. The hash values within the bucket have high similarity, and the hash values between buckets have low similarity. When retrieving, only the candidate values under specific buckets need to be compared, which reduces the amount of calculation.

[0059] The feature weighting module includes a first feature weight calculation unit, a second feature weight calculation unit, and a feature weight calculation unit;

[0060] The first feature weight calculation unit: is used to set a certain feature f in a certain component iappears k times, and the n eigenvalues appear t times in total, then tf(f i ) = k / t, where tf represents the term frequency. If the feature f i appears in x components, then idf(f i ) = log(m / x), where idf represents the inverse document frequency. The weight value w1(f i ) corresponding to the feature f i ) = k / t * log(m / x);

[0061] The second feature weight calculation unit: is used to set that the number of times the feature f i appears in all components are C = {c1, c2, c3,..., cm}, and the random variable Y is used to represent the number of times the feature f i appears in the component. Then the information entropy of the introduced random variable Y is expressed as H(Y) = ∑ j P(cj) * log(P(cj)), where P(cj) represents the probability of cj, that is, the number of times the value cj appears in C divided by the total number of C. Let the coverage rate acd(f i ) = x / m, and the weight value w2 of the feature is obtained as w2 = acd(fi) * H(Y); i ) = x / m, and the weight value w2 of the feature is obtained as w2 = acd(fi) * H(Y);

[0062] The feature weight calculation unit: is used to obtain the final feature weight value through the formula w(f i ) = alpha * w1 + beta * w2, where both alpha and beta are empirically set values by humans.

[0063] Another embodiment of the present invention relates to a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the retrieval method are implemented.

[0064] Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0065] Another embodiment of the present invention relates to a terminal device, including a memory and a processor. A computer program is stored on the memory, and when the processor executes the computer program, the steps of the retrieval method are implemented.

[0066] The above are only the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein, should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be changed within the scope of the concept described herein through the above teachings or the technology or knowledge in the relevant field. Any changes and modifications made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the protection scope of the appended claims of the present invention.

Claims

1. A retrieval method based on an improved MinHash algorithm, characterized in that: The retrieval method includes: S1. Download m binary component libraries of different types and versions to obtain a component set S; S2. Extract features from each component in the component set S respectively; S3. Calculate the weight value of the eigenvalue through the feature weight assignment module, calculate the component hash value through the MinHash algorithm. If the minimum hash value selects the feature f i , while recording its hash value, associate its feature weight value w(f i ); S4. Calculate the similarity between two components through a similarity calculation formula; The specific process of calculating the weight value of feature values by the feature weight assignment module includes: Let a certain feature f in a certain component i appear k times, and the n eigenvalue appear t times in total, then tf(f i ) = k / t, tf represents the term frequency. If the feature f i appears in x components, then idf(f i ) = log(m / x), idf represents the inverse document frequency. The weight value w1(f i ) corresponding to the feature f i ) = k / t * log(m / x); Let the feature be f i The number of occurrences of f in all components are C = {c1, c2, c3, …, cm} respectively. Let the random variable Y represent the number of occurrences of the feature f i in the component. Then the information entropy of the random variable Y is expressed as H(Y) = ∑ j P(cj) * log(P(cj)), where P(cj) represents the probability of cj, that is, the number of occurrences of the value cj in C divided by the total number of C. Let the feature f i have a coverage rate acd(f i ) = x / m. The weight value of the feature is obtained as w2 = acd(f i ) * H(Y); Obtain the final feature weight value w(f i ) = alpha * w1 + beta * w2; where both alpha and beta are empirically set values by humans; The specific process of calculating the similarity between two components through the similarity calculation formula includes: Let the fingerprint of component A be the set {a1, a2, ···, a m}, and the fingerprint corresponding to component B be the set {b1, b2, ···, b m}. By calculating through the feature weight assignment module, the weight value of each feature can be obtained. When generating the MinHash fingerprint, the weight of the corresponding feature is retained. Then the feature vectors corresponding to the two components are {w a,1 , w a,2 , ···, w a,m}, {w b,1 , w b,2 , ···, w b,m} respectively. Then, the similarity P between the two components is calculated through the similarity calculation formula , where T is the subscript set of the fingerprints that are the same for components A and B.

2. The retrieval method based on the improved MinHash algorithm according to claim 1, wherein: The retrieval method further includes an index construction step, which is executed between S3 and S4. The index construction step includes: Assume that 6 minimum hash values are generated in total, and 4 bits are extracted from each minimum hash value, that is, 24 - bit hash values are generated in total. Then the 24 - bit hash values are divided into 3 parts, and each 8 - bit corresponds to a bucket number. A total of 3 hash tables are generated, and each hash table contains 256 buckets. The hash value corresponding to the decimal value 155 of the first 8 bits is assigned to buckets numbered 1 to 155 in hash table 1, forming a list. The hash values within the bucket have high similarity, and the hash values between buckets have low similarity. During retrieval, only the candidate values under specific buckets need to be compared, which reduces the amount of calculation.

3. A retrieval device based on an improved MinHash algorithm, characterized in that: It includes a component set module, a feature extraction module, a feature weight assignment module, and a similarity calculation module; The component set module: is used to download m binary component libraries of different types and versions to obtain a component set S; The feature extraction module: is used to extract features from each component in the component set S respectively; The feature weight assignment module: It is used to calculate the weight value of the eigenvalue. Calculate the hash value of a component through the MinHash algorithm. If the minimum hash value selects the feature f i , while recording its hash value, associate its feature weight value w(f i ); The similarity calculation module: is used to calculate the similarity between two components through a similarity calculation formula; The feature weight assignment module includes a first feature weight calculation unit, a second feature weight calculation unit, and a feature weight calculation unit; The first feature weight calculation unit: used to set a certain feature f in a certain component i appears k times, and the n eigenvalues appear a total of t times, then tf(f i ) = k / t, tf represents the term frequency. If the feature f i appears in x components, then idf(f i ) = log(m / x), idf represents the inverse document frequency. The feature f i corresponding weight value w1(f i ) = k / t * log(m / x); The second feature weight calculation unit: is used to set the feature f i The number of occurrences in all components is respectively C = {c1, c2, c3, …, cm}, and the random variable Y is used to represent the number of occurrences of the feature f i in the component. Then, the information entropy of the random variable Y is expressed as H(Y) = ∑ j P(cj) * log(P(cj)), where P(cj) represents the probability of cj, that is, the number of occurrences of the value cj in C divided by the total number of C. Set the coverage rate of the feature f i acd(f i ) = x / m, and obtain the weight value of the feature w2 = acd(f i ) * H(Y); The feature weight calculation unit: is used to obtain the final feature weight value through the formula w(f i ) = alpha * w1 + beta * w2, where both alpha and beta are empirically set values by humans.

4. The retrieval device based on the improved MinHash algorithm according to claim 3, characterized in that: It further includes an index construction module. The index construction module: is used to assume that 6 minimum hash values are generated in total, and 4 bits are extracted from each minimum hash value, that is, 24 - bit hash values are generated in total. Then the 24 - bit hash values are divided into 3 parts, and each 8 - bit corresponds to a bucket number. A total of 3 hash tables are generated, and each hash table contains 256 buckets. The hash value corresponding to the decimal value 155 of the first 8 bits is assigned to buckets numbered 1 to 155 in hash table 1, forming a list. The hash values within the bucket have high similarity, and the hash values between buckets have low similarity. During retrieval, only the candidate values under specific buckets need to be compared, which reduces the amount of calculation.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the retrieval method described in claim 1 or 2.

6. A terminal device, comprising a memory and a processor, wherein a computer program is stored on the memory, and characterized in that: When the processor executes the computer program, it implements the steps of the retrieval method described in claim 1 or 2.

Images