Data retrieval method and apparatus, electronic device, and computer-readable storage medium

Abstract

This invention provides a data retrieval method, apparatus, electronic device, and computer-readable storage medium. The method includes: acquiring data items to be retrieved; normalizing the data items to be retrieved based on a preset deep hash function to obtain an initial hash set; performing similarity aggregation processing on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set; performing retrieval processing on the adjusted hash set based on a preset learning filter to obtain initially verified data items; and performing query processing on the initially verified data items based on a preset database to obtain a final data item retrieval result. According to the scheme of this invention, similar data will have more similar and closer representations in hash encoding, thereby maintaining data similarity, facilitating data similarity search and correlation analysis, and improving the query efficiency of similar data items.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data retrieval method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] A common technique for quickly retrieving data items is to use data structures that support set membership checks. This type of data structure reduces search space and avoids unnecessary comparisons or transfers of data items. A Bloom filter is one such data structure; it's a probabilistic data structure that uses multiple ordinary hash functions to map items to a bit array. Bloom filters achieve high space efficiency and fast query times, but they have two significant drawbacks: they cannot support deleting data items and their false positive rate is not zero. To overcome these shortcomings, the Cuckoo Filter was developed. This is another probabilistic data structure that uses a cuckoo hash function to resolve collisions and stores data fingerprints instead of just bits. Cuckoo filters support deleting data items, and their false positives are bounded, which can be adjusted by changing the fingerprint and bucket size. Furthermore, the Cuckoo Filter has a zero false negative rate, meaning that if a data item belongs to the set, it will never be reported as not belonging to the set.

[0003] Currently, during data querying, data sources may contain many nearly identical data points, yet these data points may not be entirely identical. Processing these nearly identical data points is difficult because data items may have different formats. For example, with images, if only one pixel is modified, the visual appearance of the image is almost identical to the original. These nearly identical images should be treated as the same content during processing. However, traditional filtering techniques treat nearly identical images as different images, making it impossible to perform approximate searches and significantly reducing the efficiency of querying nearly identical data items. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0005] To address this, the present invention proposes a data retrieval method that enables similar data to have more similar and closer representations in hash encoding, thereby maintaining data similarity, facilitating data similarity search and correlation analysis, and improving the query efficiency of similar data items.

[0006] The present invention also proposes an apparatus that applies the above-described data retrieval method.

[0007] The present invention also proposes an electronic device that applies the above-described data retrieval method.

[0008] The present invention also proposes a computer-readable storage medium that applies the above-described data retrieval method.

[0009] According to a first aspect of the present invention, a data retrieval method includes:

[0010] Retrieve the data items to be searched;

[0011] The initial hash set is obtained by normalizing the data items to be retrieved based on a preset deep hash function.

[0012] An adjusted hash set is obtained by performing similarity aggregation on the initial hash set based on a preset locality-sensitive hash function;

[0013] The initial verification passed data items are obtained by retrieving the adjusted hash set based on a preset learning filter;

[0014] The initial test is performed based on a preset database to obtain the final data item retrieval results through data item query processing.

[0015] According to some embodiments of the present invention, the deep hash function includes a deep supervised discrete hash function, and the normalization process of the data item to be retrieved based on the preset deep hash function to obtain an initial hash set includes:

[0016] Based on the pre-trained deep supervised discrete hash, feature extraction is performed on each data item in the data item to be retrieved to obtain multiple data features, and the multiple data features are encoded to obtain multiple fixed-length hash codes.

[0017] The multiple fixed-length hash codes are determined as the initial hash set.

[0018] According to some embodiments of the present invention, the step of performing similarity aggregation processing on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set includes:

[0019] Perform a minimum hash operation on all the fixed-length hash codes in the initial hash set to obtain the overall feature matrix;

[0020] The overall feature matrix is ​​segmented into intervals to determine the number of intervals and the number of rows in each interval, and the Jaccard similarity threshold is determined based on the number of intervals, the number of rows in each interval, and a preset precision calculation formula.

[0021] The Jaccard similarity between all the fixed-length hash codes is determined, and if the Jaccard similarity is greater than the Jaccard similarity threshold, the corresponding two fixed-length hash codes are corrected to have the same hash value to obtain the adjusted hash set.

[0022] According to some embodiments of the present invention, determining the Jaccard similarity among all the said fixed-length hash codes includes:

[0023] For each fixed-length hash code in the initial hash set, perform several bit permutations and record the index value of the first bit with a value of 1 in each bit permutation to obtain several minimum hash values ​​for each fixed-length hash code.

[0024] The minimum hash values ​​corresponding to the fixed-length hash code are determined as the feature vectors of the corresponding fixed-length hash code;

[0025] The Jaccard similarity between any two fixed-length hash codes is approximated by comparing the feature vectors corresponding to them.

[0026] According to some embodiments of the present invention, the learning filter includes a learning model and a cuckoo filter, wherein the initial verification passed data items are obtained by retrieving the adjusted hash set based on the preset learning filter, including:

[0027] Based on the pre-trained learning model, the key-value recognition probability is obtained by performing key-value recognition processing on the fixed-length hash codes in the adjusted hash set.

[0028] If the probability of key value judgment is greater than the preset judgment threshold, the data item corresponding to the fixed-length hash code is determined as the initial verification passed data item;

[0029] If the key-value judgment probability is less than or equal to the preset judgment threshold, the fixed-length hash code is identified based on the cuckoo filter to determine whether the data item corresponding to the fixed-length hash code exists in the preset original dataset.

[0030] According to some embodiments of the present invention, the learning model is obtained in the following manner:

[0031] Obtain a set of positive elements and a set of negative elements, wherein the positive elements represent elements that belong to a preset set, and the negative elements represent elements that do not belong to the preset set;

[0032] The positive element and the negative element are input into a preset learning model to obtain a training score, wherein the training score is used to represent the probability of being the positive element;

[0033] The learning model is adjusted based on the training scores and the preset objective function.

[0034] According to some embodiments of the present invention, the process of obtaining the final data item retrieval result by querying data items based on a preset database for the initial test includes:

[0035] The initial test is performed by perceptual hashing the data items to obtain the perceptual hash value;

[0036] The database is queried based on the perceived hash value to obtain the final data item retrieval result.

[0037] According to a second aspect of the present invention, a data retrieval apparatus includes:

[0038] The first processing module is used to obtain the data items to be retrieved;

[0039] The second processing module is used to normalize the data items to be retrieved based on a preset deep hash function to obtain an initial hash set;

[0040] The third processing module is used to perform similarity aggregation processing on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set;

[0041] The fourth processing module is used to perform retrieval processing on the adjusted hash set based on a preset learning filter to obtain the initial verification passed data items;

[0042] The fifth processing module is used to perform query processing on the data items of the initial test based on a preset database to obtain the final data item retrieval result.

[0043] An electronic device according to a third aspect of the present invention includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data retrieval method as described above.

[0044] According to a fourth aspect of the present invention, a computer-readable storage medium stores computer-executable instructions that, when executed by a control processor, implement the data retrieval method described above.

[0045] The data retrieval method according to embodiments of the present invention has at least the following beneficial effects: In the data retrieval process, firstly, the data items to be retrieved are obtained; then, an initial hash set is obtained by normalizing the data items based on a preset deep hash function; next, an adjusted hash set is obtained by similarity aggregation processing of the initial hash set based on a preset locality-sensitive hash function; then, the adjusted hash set is retrieved by searching based on a preset learning filter to obtain the data items that have passed the initial verification; finally, the data items that have passed the initial verification are queried based on a preset database to obtain the final data item retrieval results. Through the above technical solution, similar data will have more similar and closer representations in hash encoding, thereby maintaining data similarity, facilitating data similarity search and correlation analysis, and improving the query efficiency of similar data items.

[0046] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0047] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0048] Figure 1 This is a flowchart of a data retrieval method provided in one embodiment of the present invention;

[0049] Figure 2 This is a flowchart illustrating the normalization process of the data retrieval method provided in one embodiment of the present invention for the data items to be retrieved;

[0050] Figure 3 This is a flowchart illustrating the specific process of performing similarity aggregation on an initial hash set in a data retrieval method provided by an embodiment of the present invention;

[0051] Figure 4 This is a flowchart illustrating the specific process of calculating the Jaccard similarity between any two fixed-length hash codes in the initial hash set using a data retrieval method provided in an embodiment of the present invention.

[0052] Figure 5 This is a flowchart illustrating the specific process of retrieving an adjusted hash set using a data retrieval method provided in one embodiment of the present invention.

[0053] Figure 6 This is a flowchart illustrating the training process of a learning model provided in one embodiment of the present invention.

[0054] Figure 7 This is a flowchart illustrating the specific process of database-based verification and querying in a data retrieval method provided by an embodiment of the present invention.

[0055] Figure 8 This is a schematic diagram illustrating the memory usage of a filter during simulation, provided in one embodiment of the present invention.

[0056] Figure 9 This is a schematic diagram of the single processing time of the filter during the simulation process provided in one embodiment of the present invention;

[0057] Figure 10 This is a schematic diagram of the false positive rate of the selected filter in the first dataset under the first random noise index during the simulation process provided by an embodiment of the present invention;

[0058] Figure 11 This is a schematic diagram of the false positive rate of the filter selected in the first dataset under the second random noise index during the simulation process provided by an embodiment of the present invention;

[0059] Figure 12 This is a schematic diagram of the false positive rate of the selected filter in the second dataset under the first random noise index during the simulation process provided by an embodiment of the present invention;

[0060] Figure 13 This is a schematic diagram of the false positive rate of the selected filter in the second dataset under the second random noise index during the simulation process provided by an embodiment of the present invention;

[0061] Figure 14 This is a schematic diagram of the false positive rate of the filter selected in the third dataset under the first random noise index during the simulation process provided by an embodiment of the present invention;

[0062] Figure 15 This is a schematic diagram of the false positive rate of the filter selected in the third dataset under the second random noise index during the simulation process provided by an embodiment of the present invention;

[0063] Figure 16 This is a schematic diagram of the false positive rate of the selected filter in the fourth dataset under the first random noise index during the simulation process provided by an embodiment of the present invention;

[0064] Figure 17 This is a schematic diagram of the false positive rate of the selected filter in the fourth dataset under the second random noise index during the simulation process provided by an embodiment of the present invention;

[0065] Figure 18 This is a schematic diagram of the structure of a data retrieval device provided in one embodiment of the present invention;

[0066] Figure 19 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0069] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0070] This invention provides a data retrieval method, apparatus, electronic device, and computer-readable storage medium. The method includes: firstly, acquiring the data items to be retrieved during the data retrieval process; then, normalizing the data items to be retrieved based on a preset deep hash function to obtain an initial hash set; next, performing similarity aggregation processing on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set; then, performing retrieval processing on the adjusted hash set based on a preset learning filter to obtain the data items that have passed the initial verification; finally, performing query processing on the data items that have passed the initial verification based on a preset database to obtain the final data item retrieval result. Through the above technical solution, similar data will have more similar and closer representations in hash encoding, thereby maintaining data similarity, facilitating data similarity search and correlation analysis, and improving the query efficiency of similar data items.

[0071] like Figure 1 As shown, Figure 1 This is a flowchart of a data retrieval method provided in an embodiment of the present invention. The method includes, but is not limited to, steps S100, S200, S300, S400, and S500:

[0072] Step S100: Obtain the data item to be retrieved;

[0073] Step S200: Normalize the data items to be retrieved based on the preset deep hash function to obtain the initial hash set;

[0074] Step S300: Perform similarity aggregation on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set;

[0075] Step S400: Based on the preset learning filter, the adjusted hash set is retrieved to obtain the initial data items that pass the test;

[0076] Step S500: Based on the preset database, the initial test data items are queried to obtain the final data item retrieval results.

[0077] It should be noted that the data retrieval process first involves obtaining the data items to be retrieved; then, normalization is performed on the data items using a preset deep hash function to obtain an initial hash set; next, similarity aggregation is performed on the initial hash set using a preset locality-sensitive hash function to obtain an adjusted hash set; then, the adjusted hash set is searched using a preset learning filter to obtain the data items that pass the initial verification; finally, the data items that pass the initial verification are queried using a preset database to obtain the final data item retrieval results. This technical solution ensures that similar data will have more similar and closer representations in hash encoding, thereby maintaining data similarity, facilitating data similarity search and correlation analysis, and improving the efficiency of querying similar data items.

[0078] It's important to note that retrieving data items from distributed data sources using traditional filters encounters the following problems: Data sources may be owned by different parties, and there may be no trusted or third-party server. For example, consider a scenario where several hospitals want to collaborate on a medical research project by sharing patient records; each hospital has its own sensitive and private patient record database; these hospitals do not want to share raw data with each other or with third-party servers, but want to retrieve relevant records based on certain criteria. Traditional filtering techniques cannot guarantee data privacy and security. Data sources may be heterogeneous and not independent yet homo-distributed, meaning they may have different distributions, formats, and data qualities. For example, hospitals may use different data systems to store and manage patient records, resulting in different data formats, attributes, and values; different formats mean that a single traditional filter cannot solve all problems. Data sources may be massive and dynamic, meaning they may contain many data elements that change over time. For example, hospitals may continuously update patient records with new information or delete outdated information; large datasets require higher query accuracy and lower memory usage, which traditional filtering techniques, especially in terms of memory consumption, are insufficient to support. Data sources may contain a lot of nearly identical data, but these data may not be completely identical. It is difficult to process this nearly identical data because the data items may be in different formats. For example, if an image is modified by only one pixel, its visual appearance is almost the same as before. These nearly identical images should be treated as the same content when processing, but traditional filtering techniques will treat nearly identical images as different images, making it impossible to perform approximate searches and significantly reducing the query efficiency for nearly identical data items.

[0079] It is worth noting that deep hashing functions preprocess data, transforming it into hash values ​​containing the original data characteristics, thus avoiding the impact of data heterogeneity. Deep hashing functions can convert heterogeneous data into a unified fixed-length binary code, called a hash code. This unified representation enables comparison and clustering of different types and forms of data, thereby promoting data analysis and mining. Based on deep learning models, deep hashing functions can learn effective data representations and perform feature extraction. By training a deep hashing model, useful features in the data can be automatically learned and encoded into hash codes. This reduces redundant information in the data and improves its expressive power. Deep hashing functions consider data similarity when generating hash codes; similar data will have more similar and closer representations in hash encoding, thus maintaining data similarity and facilitating data similarity searches and correlation analysis.

[0080] It is worth noting that the learning filter in this embodiment is a learning cuckoo filter. By using a learning filter, memory usage is significantly reduced and query accuracy is improved to address the challenges of constantly updating large-scale data queries. The learning cuckoo filter effectively reduces memory usage by learning the distribution and characteristics of the data. Traditional cuckoo filters use a fixed-size bit array to represent the presence or absence of data, while the learning cuckoo filter dynamically adjusts the size of the bit array based on the characteristics and distribution of the data, thereby reducing memory usage. The learning cuckoo filter improves the accuracy of the filter by learning the distribution and characteristics of the data, thus reducing the false positive rate. Traditional cuckoo filters, in the presence of a large number of hash collisions, lead to an increased false positive rate, while the learning cuckoo filter can reduce the occurrence of hash collisions by learning the distribution characteristics of the data, thereby improving the accuracy of the filter (i.e., reducing the false positive rate).

[0081] It's worth noting that using hash aggregation techniques to handle the problem of nearly identical data items reduces the interference of a large number of similar data items in the data source on data retrieval. A common hash aggregation technique is Locality Sensitive Hashing (LSH), which is a technique for handling nearly identical data. Its function is to map similar data into the same hash bucket, thereby accelerating the search and processing of similar data. When handling nearly identical data, traditional hash functions often fail to meet the requirements because even small changes in the data can lead to large changes in the hash value. LSH, however, designs special hash functions that increase the probability of similar data being mapped into the same hash bucket. The main advantage of LSH is its ability to quickly find similar data points in a high-dimensional data space. By mapping data to hash buckets, fast searches can be performed within the hash buckets of similar data, greatly reducing computational and storage overhead and thus avoiding the problem of nearly identical data items interfering with data item retrieval.

[0082] It is worth noting that, in the embodiments of the present invention, since the false positive rate of the learning filter is not zero, an exact query must be performed to determine the true result. Although the exact query of the database is very time-consuming, the query efficiency is also greatly improved because the previous fast query steps greatly reduce the number of data items that need to be queried directly.

[0083] This invention can normalize different heterogeneous data through deep hashing algorithms, making it easier to process. This allows for unified processing in subsequent hash aggregation steps instead of using multiple models to adapt to different data source formats. Learning-based filters can significantly reduce memory usage and improve query accuracy in data retrieval, avoiding errors caused by false positives. Furthermore, this invention can solve the problem of interference from nearly identical data in retrieval, accelerating query efficiency and ensuring that nearly identical data is processed uniformly, preventing malicious data attacks that modify data to bypass filters during retrieval.

[0084] It is understandable that the data item to be retrieved is the relevant data item that needs to be retrieved and determined to exist in the current database.

[0085] Additionally, in one embodiment, such as Figure 2 As shown, the deep hash function includes deep supervised discrete hashing, and the above step S200 may include, but is not limited to, steps S210 and S220.

[0086] Step S210: Based on pre-trained deep supervised discrete hashing, feature extraction is performed on each data item in the data item to be retrieved to obtain multiple data features, and the multiple data features are encoded to obtain multiple fixed-length hash codes.

[0087] Step S220: Determine multiple fixed-length hash codes as the initial hash set.

[0088] It should be noted that, in the process of normalizing the data items to be retrieved based on the preset deep hash function, firstly, based on the pre-trained deep supervised discrete hash, features are extracted from each data item in the data items to be retrieved, thereby obtaining multiple data features. Then, the multiple data features are encoded to obtain multiple fixed-length hash codes. Finally, the multiple fixed-length hash codes can be determined as the initial hash set.

[0089] It is worth noting that the data items to be retrieved are processed by deep supervised discrete hashing into fixed-length hash codes containing the original data features. In this step, the data items to be retrieved are normalized, enabling data items with different formats and attributes to be processed uniformly. Preprocessing the data using a deep hash function transforms it into hash values ​​containing the original data features, thus avoiding the impact of data heterogeneity. Deep hash functions can convert heterogeneous data into unified fixed-length binary codes, called hash codes. This unified representation allows for comparison and clustering of data of different types and forms, thereby promoting data analysis and mining. Deep hash functions are based on deep learning models and can learn effective data representations and perform feature extraction. By training a deep hash model, useful features in the data can be automatically learned and encoded into hash codes. This reduces redundant information in the data and improves the data's expressive power. Deep hash functions consider data similarity when generating hash codes. Similar data will have more similar and closer representations in hash encoding, thus maintaining data similarity and facilitating data similarity searches and correlation analysis.

[0090] It's worth noting that deep hashing calculates similar hash values ​​for similar images (or files in other formats). This compact binary code can be efficiently stored and compared. The goal of deep hashing is to preserve the semantic similarity of the original images in the hash space, so that images that are close in the feature space are also close in the Hamming space (and other spaces). Deep hashing has many applications in image retrieval, indexing, clustering, and compression. One of the challenges of deep hashing is designing a hash function that maps high-dimensional images to low-dimensional binary code while minimizing quantization errors and semantic loss. These methods can be broadly categorized into supervised, unsupervised, and semi-supervised methods, depending on whether they use label information. Deep hashing is actually a special type of perceptual hashing function based on deep learning. Perceptual hashing functions aim to overcome the similarity problem by mimicking human perception and focus on the features that drive human senses. For example, a perceptual hashing function for an image can consider the color, shape, texture, and contrast of an image region while ignoring noise, compression artifacts, cropping, scaling, or rotation. Similarly, perceptual hash functions for sound can consider the frequency, amplitude, pitch, and timbre of sound segments while ignoring background noise, volume variations, or speed variations. In this way, perceptual hash functions can generate hash values ​​that are close to each other (measured by some distance) for perceptually similar inputs, and generate hash values ​​that are far apart for perceptually dissimilar inputs. The performance of perceptual hash functions can be evaluated by measuring their accuracy (the probability of correctly identifying similar or dissimilar inputs), robustness (the ability to resist various input distortions or attacks), efficiency (the time and space complexity of computing and comparing hashes), and security (the difficulty of forging or tampering with hashes). The deep hash function method used in this invention is Deep Supervised Discrete Hash (DSDH). This method is used because it creates more discrete hash values, thereby reducing the error rate when using hash aggregation techniques. This means that hash values ​​from similar original data can be aggregated together without generating as many hash collision errors. After deep hash function processing, the set of hash values ​​enters the hash aggregation processing stage.

[0091] Additionally, in one embodiment, such as Figure 3 As shown, step S300 may include, but is not limited to, steps S310, S320 and S330.

[0092] Step S310: Perform a minimum hash operation on all fixed-length hash codes in the initial hash set to obtain the overall feature matrix;

[0093] Step S320: Perform interval segmentation on the overall feature matrix to determine the number of intervals and the number of rows in each interval, and determine the Jaccard similarity threshold based on the number of intervals, the number of rows in each interval, and the preset precision calculation formula;

[0094] Step S330: Determine the Jaccard similarity between all fixed-length hash codes, and if the Jaccard similarity is greater than the Jaccard similarity threshold, correct the corresponding two fixed-length hash codes to have the same hash value to obtain an adjusted hash set.

[0095] It should be noted that in the process of similarity aggregation of the initial hash set, firstly, a minimum hash operation is performed on all fixed-length hash codes in the initial hash set to obtain the overall feature matrix; then, the overall feature matrix is ​​divided into intervals to determine the number of intervals and the number of rows in each interval, and the Jaccard similarity threshold is determined based on the number of intervals, the number of rows in each interval, and a preset precision calculation formula; finally, the Jaccard similarity between all fixed-length hash codes is determined, and if the Jaccard similarity is greater than the Jaccard similarity threshold, the corresponding two fixed-length hash codes are corrected to have the same hash value to obtain an adjusted hash set, thereby realizing the aggregation of similar data items, which facilitates subsequent similarity search and correlation analysis of similar data items and improves the query efficiency of similar data items.

[0096] In some embodiments of this application, a MinHash operation is performed on all fixed-length hash codes in the initial hash set, aggregating the m-dimensional feature vectors of all fixed-length hash codes into an m×k overall feature matrix. Here, m is the number of hash functions used in the MinHash operation, and k is the total number of fixed-length hash codes. Then, the m×k overall feature matrix is ​​divided into b intervals along the vertical axis, with each interval having r rows (m = b×r). The precision calculation formula is P = 1 - (1 - s) r ) b s is calculated (s is the Jaccard similarity threshold that can determine whether the original contents of two fixed-length hash codes are similar). Here, P is the probability that the original contents of two fixed-length hash codes are similar, and controlling the value of P can control the precision. Finally, the Jaccard similarity between all fixed-length hash codes is compared. If the Jaccard similarity is greater than the threshold s, the corresponding two fixed-length hash codes are corrected to have the same hash value to obtain the adjusted hash set.

[0097] It is worth noting that deep hash functions are a type of perceptual hash function based on deep learning. Therefore, if two images (and videos) are preprocessed to calculate the same hash value for similar and identical images (and videos), filters can be used to quickly retrieve data items. However, current deep hash function methods cannot calculate the same hash value for similar or identical images to achieve more accurate data retrieval; instead, they calculate slightly similar hash values ​​(differences not exceeding a set threshold). This method is suitable for improving accuracy but significantly reduces retrieval speed. Especially when the data items to be retrieved are already determined (e.g., data deduplication and similarity checks), such high accuracy is not required, and when it is necessary to treat approximately identical data items as the same content, high performance and the ability to solve the approximation problem are needed. Hash aggregation is required on the set of hash values ​​obtained from deep hash functions to merge similar hash values ​​into the same hash value. In this invention, the hash aggregation method used is the location-sensitive hashing method. Locality-sensitive hashing (LSH) is an algorithm that can solve approximate or exact nearest neighbor searches in high-dimensional space; it is a widely used method for finding approximate nearest neighbors in high-dimensional space. Its greatest advantage is that it can complete the search in a shorter time than linear search, and its accuracy is highly reliable, theoretically guaranteed. LSH works by hashing the input items, thereby assigning similar items to the same bucket with a high probability. To this end, a smaller number of hash buckets are used, rather than attempting to cover all possible input items in the search space. This idea makes it possible to solve high-dimensional data problems in sublinear time. The LSH algorithm has been widely used in various fields, including but not limited to image and speech recognition, and recommender systems. The idea behind ordinary hash functions is to extract features from the processed data, so it tries to ensure that different data (even very similar ones) fall into different buckets (i.e., produce different hash values) and needs to avoid hash collisions. However, LSH takes the opposite approach, typically aiming to maximize the probability of collisions. This idea is based on the fact that for a well-designed hash function, the probability of a hash collision when two original data items are similar or identical is much greater than the probability of two original data items being different. Since in applications such as the main target data retrieval problem of this invention, a certain probability of misclassification is allowed (i.e., data that is not in the set is judged to be in the set), LSH can be used for images (and videos). In this way, similar or identical images (and videos) can be processed to obtain the same hash value, which facilitates fast processing using filters.

[0098] Additionally, in one embodiment, such as Figure 4 As shown, step S330 may include, but is not limited to, steps S331, S332 and S333.

[0099] Step S331: Perform several bit permutations on each fixed-length hash code in the initial hash set, and record the index value of the first bit with a value of 1 in each bit permutation, to obtain several minimum hash values ​​for each fixed-length hash code.

[0100] Step S332: Determine several minimum hash values ​​corresponding to the fixed-length hash code as the feature vector of the corresponding fixed-length hash code;

[0101] Step S333: Compare the feature vectors corresponding to any two fixed-length hash codes to approximate the Jaccard similarity between the fixed-length hash codes.

[0102] It should be noted that in the process of calculating the Jaccard similarity between any two fixed-length hash codes in the initial hash set, each fixed-length hash code in the initial hash set is first subjected to several bit permutations, and the index value of the first bit with a value of 1 in each bit permutation is recorded to obtain several minimum hash values ​​for each fixed-length hash code. Then, the several minimum hash values ​​corresponding to the fixed-length hash code are determined as the feature vector of the corresponding fixed-length hash code. Finally, by comparing the feature vectors corresponding to any two fixed-length hash codes, the Jaccard similarity between the fixed-length hash codes can be approximately obtained, which prepares for subsequent data item similarity judgment.

[0103] In some embodiments of this invention, the MinHash method from the LSH algorithm is used to process the set of hash values. This method is more efficient and has a higher fault tolerance than the Bit Sampling method. The MinHash method is a Locality Sensitive Hash (LSH) technique that utilizes Jaccard distance. Jaccard distance is a metric that measures the difference between two sets of data. It is based on Jaccard similarity, which quantifies the similarity between two sets of data. To calculate Jaccard similarity, the intersection size of the two sets of data is divided by their combination size. Simply put, it is the proportion of all elements in the two sets of data. The calculation formula is as follows:

[0104]

[0105]

[0106] Where J represents Jaccard similarity, DJ represents Jaccard distance, and A and B are two sets; the Jaccard distance ranges from 0 to 1. The closer to 1, the less similar the two sets of data are. The specific steps are as follows:

[0107] (1) Randomly arrange the bits of each vector and record the index of the first bit that is 1. Repeat this process for k different arrangements, and then obtain k MinHash values ​​for each vector.

[0108] (2) These k values ​​form the eigenvector representing the vector. The Jaccard similarity between two vectors is equal to the probability that their MinHash values ​​are equal under random permutation. Therefore, by comparing the eigenvectors of two vectors, their Jaccard similarity can be estimated. This method is more effective than another method, therefore this invention will use this method as the implementation of LSH.

[0109]

[0110] Where A is a binary vector, π is the bit arrangement of A, h(A) is the new MinHash value of A, and S is the hash set.

[0111] In this step, the present invention transforms the problem of similar identical data items into the problem of identical data items, which is more suitable for filter data processing in subsequent steps and more suitable for real-world scenarios where data item queries are mainly based on images and videos.

[0112] Additionally, in one embodiment, such as Figure 5 As shown, step S400 may include, but is not limited to, steps S410 and S430.

[0113] Step S410: Based on the pre-trained learning model, perform key-value recognition processing on the fixed-length hash codes in the adjusted hash set to obtain the key-value judgment probability;

[0114] Step S420: If the key value judgment probability is greater than the preset judgment threshold, the data item corresponding to the corresponding fixed-length hash code is determined as the initial verification passed data item;

[0115] Step S430: If the key value judgment probability is less than or equal to the preset judgment threshold, the fixed-length hash code is identified based on the cuckoo filter to determine whether the data item corresponding to the fixed-length hash code exists in the preset original dataset.

[0116] It should be noted that in the process of retrieving the adjusted hash set based on the learning filter, the key-value recognition process is first performed on the fixed-length hash codes in the adjusted hash set based on the pre-trained learning model to obtain the key-value judgment probability. If the key-value judgment probability is greater than the preset judgment threshold, the data item corresponding to the fixed-length hash code is determined as the initial data item that passes the test. If the key-value judgment probability is less than or equal to the preset judgment threshold, the fixed-length hash code is identified based on the cuckoo filter to determine whether the data item corresponding to the fixed-length hash code exists in the preset original dataset.

[0117] It's worth noting that the Cuckoo Filter is based on the Cuckoo Hash algorithm, a technique for resolving hash collisions in a hash table by using two or more hash functions and relocating existing items when necessary. The Cuckoo Filter only stores the data fingerprint of an item, i.e., a short string obtained by hashing the data item. The Cuckoo Filter consists of an array of buckets, each holding one or more fingerprints. An item can be mapped to two candidate buckets using two hash functions. The Cuckoo Filter is insufficient for handling large-scale data queries due to its low accuracy and memory efficiency. Therefore, it can be compensated for by using a learning data structure approach, such as a learning filter. Learning data structures suggest using learning models such as neural networks instead of traditional indexing structures like B-trees and hash tables. These models can predict the location or existence of records based on key values. The main idea is that learning models can leverage patterns and correlations in the data distribution, achieving better performance and efficiency than fixed data structures. To transform a Cuckoo Filter into a learning Cuckoo Filter, a learning model that can predict the best hash bucket for each data fingerprint is used instead of the hash function that maps fingerprints to hash buckets. Learning models can be easily created using fingerprints and their corresponding fingerprint buckets. These models are trained using efficient loss functions to minimize any shifts or collisions. Furthermore, the model can be effortlessly updated online whenever a new data fingerprint is added or removed from the filter. The advantage of using a learning model instead of a hash function is that the learning model can capture the unevenness and dependencies in the distribution of data fingerprints and distribute fingerprints to hash buckets more evenly and intelligently. This reduces the number of hash shifts and collisions, thereby improving insertion and query performance and reducing false positive rates. The disadvantage is that the learning model may incur additional computational and memory overhead, putting performance pressure on the system. Therefore, the learning model is combined with the cuckoo filter to form a learned cuckoo filter. However, learning cuckoo filters alone are far from sufficient to handle complex data item retrieval problems, and real-world scenarios are often less than ideal. To ensure privacy and security, data is often distributed across different servers or even different companies, making traditional single-machine deep learning ineffective. Therefore, federated learning is used to improve this situation. Federated learning is a method that allows multiple parties to collaboratively train machine learning models without sharing data with each other or any third-party servers. Each party retains its own data locally, sharing model parameters or updates only with other parties. This approach helps protect the confidentiality and privacy of data while reducing costs associated with centralized learning, such as communication and computation costs.

[0118] Furthermore, the learning-based cuckoo filter does not rely solely on hash functions, but primarily employs a learning model that accurately predicts the presence of keys in the key set. This model is trained on positive and negative key samples and uses a threshold to classify them with high confidence. If the model predicts with high confidence that a key is in the set, it returns true without checking the bit array. If the model predicts with high confidence that a key is not in the set, it returns false without querying the bit array. If the model is uncertain about a key, it returns the bit array and uses the hash function as usual. The learning-based cuckoo filter adds a learning filter model before the standard cuckoo filter. It still uses the standard cuckoo filter as a backup mechanism to handle situations where the learning model is uncertain about keys. The model may need help distinguishing between positive and negative keys, especially when the key distribution is complex or ambiguous. The cuckoo filter probabilistically guarantees no false negatives and further reduces the false positive rate by checking the bits in the array. Therefore, the learning cuckoo filter combines the advantages of the learning model and the cuckoo filter, and its performance and efficiency are better than using either one alone.

[0119] Additionally, in one embodiment, such as Figure 6 As shown, the learning model can be obtained through, but is not limited to, steps S411 to S413:

[0120] Step S411: Obtain a set of positive elements and a set of negative elements, wherein positive elements represent elements that belong to a preset set, and negative elements represent elements that do not belong to the preset set.

[0121] Step S412: Input positive and negative elements into a preset learning model to obtain training scores, wherein the training scores are used to represent the probability of being a positive element.

[0122] Step S413: Adjust the parameters of the learning model based on the training score and the preset objective function.

[0123] It should be noted that during the training of the learning model, a set of positive elements and a set of negative elements are first obtained. Positive elements represent elements that belong to a preset set, and negative elements represent elements that do not belong to the preset set. Then, the positive and negative elements are input into the preset learning model to obtain training scores, which are used to represent the probability of being a positive element. Finally, the parameters of the learning model are adjusted according to the obtained training scores and the preset objective function to achieve the training process of the learning model.

[0124] It is worth noting that the learning cuckoo filter is a data structure that uses a learning model and a traditional cuckoo filter to efficiently store a set of items and accurately answer set membership queries with a predetermined error level. Federated learning will be used to address the model training problem under a distributed data source. To train the learning cuckoo filter, the following steps are required: (1) Collect a set of positive elements (elements belonging to the set) and a set of negative elements (elements not belonging to the set); (2) Use binary classification as the objective function to train a learning model (such as a neural network) on the positive and negative elements. The model should output a score between 0 and 1 for each element, representing the probability that it is a positive element; (3) Choose a threshold τ between 0 and 1 so that the model can correctly classify most positive elements (making the score higher than τ) and misclassify most negative elements (making the score lower than τ); (4) Create a traditional cuckoo filter, slightly smaller than usual, and only insert positive elements with scores lower than τ. These are elements that the model cannot determine to be classified as positive; (5) To query element x, first use the model to calculate its score s(x). If s(x) ≥ τ, then x is identified as a positive element without referencing the cuckoo filter. If s(x) < τ, x is checked against the cuckoo filter to determine if it exists in the original dataset. A key aspect of training the model is determining the threshold. The threshold τ for learning the model is a hyperparameter that determines the level of confidence required for the model to make predictions without referencing the cuckoo filter. A higher threshold means the model will only return "true" for keys it is very certain of, while referring to the cuckoo filter for many more keys. A lower threshold means the model will directly return "true" for more keywords, using the cuckoo filter less. The ideal threshold depends on a balance between memory usage and the false positive rate. A higher threshold can save memory by using a smaller cuckoo filter, but may allow more false positives to pass through the model, increasing the false positive rate. A lower threshold can filter out more false positives, reducing the false positive rate, but may require more memory to use a larger cuckoo filter. A suitable threshold can be chosen by testing different values ​​on the validation set and measuring memory consumption and false positive rates.

[0125] It's worth noting that federated learning addresses data security and privacy issues related to sharing data between distributed data sources during model training. In federated learning, each participant trains their own local model. The raw data remains on each participant's local device or data center, avoiding transmission to a central server to prevent data leakage. Model parameters are then shared with other participants. By training the model locally and sharing parameters, sharing raw data with other participants is avoided, thus protecting data privacy. In addition, differential privacy techniques can be used. This method protects data privacy by adding noise to the data to hide personally identifiable information. In federated learning, participants can process data locally to achieve differential privacy, and then share the processed data for model training, thereby protecting data privacy.

[0126] Additionally, in one embodiment, such as Figure 7 As shown, step S500 may also include, but is not limited to, steps S510 and S520.

[0127] Step S510: The initial test data items are processed by a perceptual hash operation to obtain a perceptual hash value;

[0128] Step S520: Query the database based on the perceptual hash value to obtain the final data item retrieval result.

[0129] It should be noted that in the process of querying the data items of the initial test, the initial test data items are first hashed to obtain the perceptual hash value; then the database is queried based on the perceptual hash value to obtain the final data item retrieval result.

[0130] It's important to note that because the false positive rate of the filter is not zero—meaning there are data items that don't belong to the original dataset but are still identified as present by the filter—data items tested by the learned filter must undergo a final exact query in the database to completely confirm their existence in the original data. Data is often stored in a MySQL database. For image or video files, the perceptual hashing algorithm can be used again, pre-calculating the perceptual hash value for each file as an index, which can significantly improve the efficiency of exact queries.

[0131] To demonstrate the superiority of the learning cuckoo filter used in the embodiments of this application, the following experiments were conducted.

[0132] The datasets used in the tests are described below: ISCX-URL2016 (text): This is a URL dataset containing 1.7 million URLs labeled as benign or malicious. It is used for research purposes in the field of network security. IPNetwork Traffic Flows (text): This dataset has 87 feature fields, providing detailed information about network device IP traffic, including source and destination IP addresses, ports, arrival times, and the Layer 7 protocol used for traffic categories. The IP portion of the dataset is a primary focus. COCO (images): This dataset (Common Objects in Context) is a comprehensive resource for computer vision research. It includes over 330,000 images and 2.5 million labeled object instances across 80 categories, making it ideal for object detection, segmentation, and captioning research. Kinetics-600 (video): This is a comprehensive video dataset primarily used for action recognition research in computer vision. It includes 600 action categories, with at least 600 video clips for each category.

[0133] Reference Figures 8-17As shown, CF represents the traditional cuckoo filter; LCF represents the learned cuckoo filter used in this embodiment; the content in parentheses after LCF represents different models used; the training ratio of the dataset will be gradually increased during testing (i.e., 0.1, 0.5, and 1); noise will be gradually added to the dataset during testing (i.e., changing the random noise index). As the noise index increases, the accuracy of the filter also decreases significantly. This is because binary classifiers are very sensitive to noisy data, which greatly reduces classification efficiency. Especially for text data, i.e., URLs and IPs, since deep hashing is not required, their similarity detection is completely different from that of images. Once this data is retrained with noise, the classification efficiency of the filter drops sharply. This shows that data retrieval problems have high requirements for low-noise data, especially for filter-based methods. For image and video datasets, the impact of noise is not as obvious as that of text datasets. This is because deep hashing is used, so that noisy data items can still produce the same or similar hash values. Comparing the results of the standard cuckoo filter and several learned cuckoo filters, the standard version is faster (because it doesn't require a time-consuming model), but it lags far behind the learned cuckoo filter in terms of false positive rate and memory consumption. This is especially true when the dataset gradually expands, as the filter function relies on a large array storing the fingerprints of data items. Assuming this large array gradually fills up, hash collisions will occur frequently, significantly reducing accuracy and drastically increasing the false positive rate, which is unacceptable in practical applications. Therefore, using a learned cuckoo filter instead of the standard one is a better choice. Comparing different learning models, the RF model (Random Forest model) performs best in terms of false positive rate. However, its memory consumption is significantly larger than the SVM model (Support Vector Machine model). However, the SVM model performs very poorly in non-linear cases (i.e., images and videos), which is reflected in its very high false positive rate in the experimental results. Furthermore, although the FFNN model (Feedforward Neural Network model) is comparable to the RF model in terms of false positive rate, its inference speed is an order of magnitude slower than the RF model. Therefore, after weighing speed, false positive rate, and memory consumption, the best choice is to use the RF model.

[0134] Furthermore, to illustrate the specific application scenarios and potential limitations of this application, the following analysis is provided. For the problem of rapid data retrieval by multiple participants, this invention has advantages. It can train a learned cuckoo filter capable of rapidly retrieving data while protecting the privacy of each participant, thereby significantly reducing the number of direct database queries, greatly alleviating database performance pressure, and accelerating data retrieval efficiency. Regarding potential limitations, because the learned cuckoo filter used in this invention includes a traditional cuckoo filter as a backup (to ensure a false negative rate of 0), the traditional cuckoo filter cannot be updated when federated learning requires model updates. Therefore, it has limitations in scenarios requiring continuous model updates. However, in reality, scenarios requiring continuous real-time updates are rare. Requiring real-time updates might compromise the privacy of the participants, and continuous updates of a model consume significant computational resources. In reality, the model can often be completely retrained at fixed intervals, which better protects the privacy of multiple participants and reduces computational consumption. Therefore, the embodiments of this application can be considered to have minimal limitations in this scenario and do not need to be deliberately considered.

[0135] In some embodiments of the present invention, such as Figure 18 As shown, one embodiment of the present invention also provides a data retrieval device 10, which includes:

[0136] The first processing module 100 is used to obtain the data items to be retrieved;

[0137] The second processing module 200 is used to normalize the data items to be retrieved based on a preset deep hash function to obtain an initial hash set;

[0138] The third processing module 300 is used to perform similarity aggregation processing on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set;

[0139] The fourth processing module 400 is used to perform retrieval processing on the adjusted hash set based on a preset learning filter to obtain the initial verification passed data items;

[0140] The fifth processing module 500 is used to query and process the initial test data items based on a preset database to obtain the final data item retrieval results.

[0141] The specific implementation of the data retrieval device 10 is basically the same as the specific embodiment of the data retrieval method described above, and will not be repeated here.

[0142] In some embodiments of the present invention, such as Figure 19As shown, one embodiment of the present invention also provides an electronic device 700, including: a memory 720, a processor 710, and a computer program stored in the memory 720 and executable on the processor 710. When the processor 710 executes the computer program, it implements the data retrieval method described above, for example, performing the above-described... Figure 1 Method steps S100 to S500 Figure 2 Method steps S210 to S220, Figure 3 Method steps S310 to S330 in the text Figure 4 Method steps S331 to S333 in the text Figure 5 Method steps S410 to S430 in the text Figure 6 Method steps S411 to S413 and Figure 7 Method steps S510 to S520.

[0143] In some embodiments of the present invention, one embodiment further provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described device embodiments, causing the processor to perform the data retrieval method described above, for example, performing the above-described... Figure 1 Method steps S100 to S500 Figure 2 Method steps S210 to S220, Figure 3 Method steps S310 to S330 in the text Figure 4 Method steps S331 to S333 in the text Figure 5 Method steps S410 to S430 in the text Figure 6 Method steps S411 to S413 and Figure 7 Method steps S510 to S520.

[0144] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0145] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A data retrieval method, characterized in that, The method includes: Retrieve the data items to be searched; The initial hash set is obtained by normalizing the data items to be retrieved based on a preset deep hash function. An adjusted hash set is obtained by performing similarity aggregation on the initial hash set based on a preset locality-sensitive hash function; The initial verification passed data items are obtained by retrieving the adjusted hash set based on a preset learning filter; The initial test is performed based on a preset database to obtain the final data item retrieval results through data item querying. The deep hash function includes a deep supervised discrete hash function. The process of normalizing the data item to be retrieved based on the preset deep hash function to obtain an initial hash set includes: Based on the pre-trained deep supervised discrete hash, feature extraction is performed on each data item in the data item to be retrieved to obtain multiple data features, and the multiple data features are encoded to obtain multiple fixed-length hash codes. The multiple fixed-length hash codes are determined as the initial hash set; The process of performing similarity aggregation on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set includes: Perform a minimum hash operation on all the fixed-length hash codes in the initial hash set to obtain the overall feature matrix; The overall feature matrix is ​​segmented into intervals to determine the number of intervals and the number of rows in each interval, and the Jaccard similarity threshold is determined based on the number of intervals, the number of rows in each interval, and a preset precision calculation formula. The Jaccard similarity between all the fixed-length hash codes is determined, and if the Jaccard similarity is greater than the Jaccard similarity threshold, the corresponding two fixed-length hash codes are corrected to have the same hash value to obtain the adjusted hash set.

2. The data retrieval method according to claim 1, characterized in that, Determining the Jaccard similarity between all the fixed-length hash codes includes: For each fixed-length hash code in the initial hash set, perform several bit permutations and record the index value of the first bit with a value of 1 in each bit permutation to obtain several minimum hash values ​​for each fixed-length hash code. The minimum hash values ​​corresponding to the fixed-length hash code are determined as the feature vectors of the corresponding fixed-length hash code; The Jaccard similarity between any two fixed-length hash codes is obtained by comparing the feature vectors corresponding to them.

3. The data retrieval method according to claim 1, characterized in that, The learning filter includes a learning model and a cuckoo filter. The initial verification-passing data items are obtained by retrieving the adjusted hash set based on the preset learning filter, including: Based on the pre-trained learning model, the key-value recognition probability is obtained by performing key-value recognition processing on the fixed-length hash codes in the adjusted hash set. If the probability of key value judgment is greater than the preset judgment threshold, the data item corresponding to the fixed-length hash code is determined as the initial verification passed data item; If the key-value judgment probability is less than or equal to the preset judgment threshold, the fixed-length hash code is identified based on the cuckoo filter to determine whether the data item corresponding to the fixed-length hash code exists in the preset original dataset.

4. The data retrieval method according to claim 3, characterized in that, The learning model is obtained in the following way: Obtain a set of positive elements and a set of negative elements, wherein the positive elements represent elements that belong to a preset set, and the negative elements represent elements that do not belong to the preset set; The positive element and the negative element are input into a preset learning model to obtain a training score, wherein the training score is used to represent the probability of being the positive element; The learning model is adjusted based on the training scores and the preset objective function.

5. The data retrieval method according to claim 1, characterized in that, The initial test based on a preset database is processed by querying data items to obtain the final data item retrieval results, including: The initial test is performed by perceptual hashing the data items to obtain the perceptual hash value; The database is queried based on the perceived hash value to obtain the final data item retrieval result.

6. A data retrieval device, characterized in that, include: The first processing module is used to obtain the data items to be retrieved; The second processing module is used to normalize the data items to be retrieved based on a preset deep hash function to obtain an initial hash set; The third processing module is used to perform similarity aggregation processing on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set; The fourth processing module is used to perform retrieval processing on the adjusted hash set based on a preset learning filter to obtain the initial verification passed data items; The fifth processing module is used to perform query processing on the initial test data items based on a preset database to obtain the final data item retrieval result; The deep hash function includes a deep supervised discrete hash function. The process of normalizing the data item to be retrieved based on the preset deep hash function to obtain an initial hash set includes: Based on the pre-trained deep supervised discrete hash, feature extraction is performed on each data item in the data item to be retrieved to obtain multiple data features, and the multiple data features are encoded to obtain multiple fixed-length hash codes. The multiple fixed-length hash codes are determined as the initial hash set; The process of performing similarity aggregation on the initial hash set based on a preset locality-sensitive hash function to obtain an adjusted hash set includes: Perform a minimum hash operation on all the fixed-length hash codes in the initial hash set to obtain the overall feature matrix; The overall feature matrix is ​​segmented into intervals to determine the number of intervals and the number of rows in each interval, and the Jaccard similarity threshold is determined based on the number of intervals, the number of rows in each interval, and a preset precision calculation formula. The Jaccard similarity between all the fixed-length hash codes is determined, and if the Jaccard similarity is greater than the Jaccard similarity threshold, the corresponding two fixed-length hash codes are corrected to have the same hash value to obtain the adjusted hash set.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the data retrieval method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the control processor, they implement the data retrieval method as described in any one of claims 1 to 5.

Images