Method and device for data retrieval, electronic device and storage medium

Abstract

This invention provides a data retrieval method, apparatus, electronic device, and storage medium, relating to the field of data processing technology. It preprocesses the data to be queried and generates a binary identifier fingerprint of fixed length, transforming unstructured data content into a numerical representation. During the retrieval process, similar data groups are divided based on the identifier fingerprints of candidate data, enabling the identification and classification of duplicate and nearly duplicate data, reducing computational complexity while ensuring deduplication accuracy. Simultaneously, by combining the data query command and the attribute information of each candidate data, target data is selected from similar data groups, and multi-dimensional scoring and ranking are performed based on the multi-dimensional data information of the target data. Through diversified data analysis, the ranking results accurately reflect the actual importance of the data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data retrieval method and apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of the internet and the modernization of governance systems, the amount of information and data in fields such as energy technology has surged. The sources and channels are diverse and the data is frequently reprinted, resulting in a large amount of completely duplicated or highly similar redundant content in the collected and aggregated information data. This increases information processing costs and reduces retrieval efficiency.

[0003] Among existing deduplication techniques for data retrieval, string matching-based methods have low accuracy and cannot identify text with similar content; text fingerprint-based methods have high computational and storage overhead; and while deep learning semantic vector-based methods have high accuracy, their high computational complexity makes them unsuitable for real-time processing of large-scale information data. Furthermore, existing data retrieval ranking techniques often rely on preset rules or are based solely on text relevance, lacking diversified analysis of the information data. This results in ranking results that fail to fully reflect the actual importance of the information, impacting the efficiency of obtaining high-value information.

[0004] Therefore, how to balance efficiency and accuracy in the deduplication process of data retrieval, and how to accurately sort the information data based on deduplication to improve the quality of data retrieval, is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a data retrieval method, apparatus, electronic device, and storage medium. Its main objective is to address how to balance efficiency and accuracy in the deduplication process of data retrieval, and to accurately sort the information data based on deduplication.

[0006] According to a first aspect of the present invention, a data retrieval method is provided, comprising:

[0007] The acquired multiple data to be queried are preprocessed to obtain the feature sequences corresponding to each of the multiple data to be queried, and an identifier fingerprint is generated based on the feature sequences. The feature sequences are used to characterize the data content of the data to be queried, and the identifier fingerprint is a binary fingerprint with a preset fixed length. In response to a data query command, multiple candidate data corresponding to the data query command are determined from multiple data to be queried, and the multiple candidate data are divided into multiple similar data groups according to the candidate identifier fingerprints corresponding to each of the multiple candidate data. Based on the data query command and the attribute information of each of the multiple candidate data, the corresponding target data is selected from multiple similar data groups respectively; Multiple target data are scored based on their respective multidimensional data information, and then sorted according to their corresponding data scores to obtain the target retrieval results.

[0008] According to a second aspect of the present invention, a data retrieval apparatus is provided, comprising: The processing unit is used to preprocess the acquired multiple data to be queried to obtain the feature sequences corresponding to each of the multiple data to be queried. The generation unit is used to generate an identifier fingerprint based on a feature sequence, wherein the feature sequence is used to characterize the data content of the data to be queried, and the identifier fingerprint is a binary fingerprint with a preset fixed length. The determining unit is used to determine multiple candidate data corresponding to the data query instruction from multiple data to be queried in response to the data query instruction; The partitioning unit is used to divide multiple candidate data into multiple similar data groups based on the candidate identifier fingerprints corresponding to each candidate data. The filtering unit is used to filter out the corresponding target data from multiple similar data groups according to the data query command and the attribute information of each of the multiple candidate data. The sorting unit is used to score multiple target data based on their respective multidimensional data information, and sort the multiple target data according to their respective data scores to obtain the target retrieval results.

[0009] According to a third aspect of the present invention, an electronic device is provided, comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform the method described in the first aspect above.

[0010] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the method of the first aspect described above.

[0011] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method as described in the first aspect above.

[0012] The data retrieval method, apparatus, electronic device, and storage medium provided by this invention preprocess the data to be queried and generate binary identifier fingerprints of fixed length, transforming unstructured data content into numerical representations. During the retrieval process, similar data groups are divided based on the identifier fingerprints of candidate data, enabling the identification and classification of duplicate and nearly duplicate data, reducing computational complexity while ensuring deduplication accuracy. Simultaneously, by combining data query instructions and attribute information of each candidate data, target data is selected from similar data groups, and multi-dimensional scoring and ranking are performed based on the multi-dimensional data information of the target data. Through diversified data analysis, the ranking results accurately reflect the actual importance of the data. Therefore, this invention can balance deduplication efficiency and accuracy during data retrieval, and performs accurate data ranking based on deduplication, improving the quality of data retrieval.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of the invention. Wherein: Figure 1 This is a flowchart illustrating a data retrieval method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a data deduplication process provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a data sorting process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a data retrieval device provided in an embodiment of the present invention; Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0015] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] The following description, with reference to the accompanying drawings, outlines a data retrieval method, apparatus, electronic device, and storage medium according to embodiments of the present invention.

[0017] Figure 1 This is a flowchart illustrating a data retrieval method provided in an embodiment of the present invention.

[0018] like Figure 1 As shown, the method includes the following steps: Step 101: Preprocess the multiple data to be queried to obtain the feature sequences corresponding to each of the multiple data to be queried, and generate an identifier fingerprint based on the feature sequences. The feature sequences are used to characterize the data content of the data to be queried, and the identifier fingerprint is a binary fingerprint with a preset fixed length.

[0019] In embodiments of this invention, the data to be queried refers to various types of data collected through different acquisition channels, such as policy data and industry analysis reports related to the energy sector. Preprocessing involves performing standardized cleaning on the data to be queried to obtain a processing procedure that reflects the core content of the data. The preprocessing process includes, but is not limited to, data cleaning, word segmentation of the cleaned data, and filtering of the segmented data. After preprocessing, the resulting serialized set composed of the core words of the data to be queried is the feature sequence, which is used to semantically represent the data content of the data to be queried.

[0020] The fingerprint identifier is a binary fingerprint with a preset fixed length generated for each piece of data to be queried based on the feature sequence.

[0021] Step 102: In response to the data query command, determine multiple candidate data corresponding to the data query command from multiple data to be queried, and divide the multiple candidate data into multiple similar data groups according to the candidate identifier fingerprints corresponding to each of the multiple candidate data.

[0022] In embodiments of the present invention, the data query instruction at least includes the user's query intent, such as query keywords, statements, or other forms of query expressions. A preliminary retrieval is performed from a database storing multiple pieces of data to be queried, based on the data query instruction, to obtain multiple candidate data items related to the data content of the data query instruction.

[0023] The identifier fingerprint corresponding to each candidate data is called the candidate identifier fingerprint. Then, for all candidate data, a pairwise similarity comparison is performed based on their corresponding candidate identifier fingerprints to determine whether each candidate data is duplicated or similar to other candidate data. This comparison process can be implemented, but is not limited to, calculating the Hamming distance between two candidate identifier fingerprints.

[0024] When dividing similar data groups, a preset similarity threshold can be set. If the similarity between two candidate fingerprints is less than or equal to this threshold, the two candidate data corresponding to these two candidate fingerprints are determined to be similar or duplicates. Then, clustering is performed to group all similar candidate data into the same group, thus obtaining multiple similar data groups.

[0025] Step 103: Based on the data query instruction and the attribute information of each of the multiple candidate data, select the corresponding target data from multiple similar data groups respectively.

[0026] In an embodiment of the present invention, for each similar data group, a data filtering operation is performed according to the data query instruction and the attribute information of each candidate data, that is, the most representative target data is selected from each similar data group.

[0027] Attribute information refers to various metadata recorded when candidate data is stored in the database, such as the data's source, publication time, content completeness, and authority tags. When filtering data, the attribute information of candidate data within a similar data group can be matched and compared based on the emphasis of the data query (e.g., emphasizing timeliness or authority). The candidate data that best meets the requirements is selected as the target data for this similar data group, while other data within the similar data group is discarded.

[0028] This filtering process can eliminate redundant data while ensuring the comprehensiveness of information, thereby achieving high-precision deduplication of query results.

[0029] Step 104: Calculate data scores for multiple target data based on their respective multidimensional data information, and sort the multiple target data according to their corresponding data scores to obtain the target retrieval results.

[0030] In an embodiment of the present invention, after deduplicating the retrieved data, it is necessary to sort the multiple target data that are retained after deduplication.

[0031] The ranking criteria, in addition to the relevance to the data query command, also comprehensively analyze the multidimensional data information carried by each target data. Multidimensional data information refers to a set of attributes that characterize the value of data from different dimensions, which may include, but are not limited to: keyword relevance scores that represent the degree of relevance between data and query commands, media weight that measures the authority of the data source organization, data channel weight that reflects the quality of the data release channel, and popularity values ​​that combine indicators such as data release time and user interaction volume.

[0032] Then, a preset weighted algorithm is used to score each target data point to calculate the data score.

[0033] The data is sorted according to its corresponding scores (e.g., ascending or descending) to form an ordered data sequence, which is the target retrieval result. The target retrieval result not only ensures the relevance of the returned content but also eliminates redundant information through deduplication and further prioritizes high-quality data based on multi-dimensional comprehensive scores, thereby improving the efficiency of information retrieval and user experience.

[0034] This invention transforms the data to be queried into a fixed-length identifier fingerprint, performs efficient data grouping and filtering based on the identifier fingerprint, and then combines multi-dimensional data information for comprehensive scoring and sorting to obtain non-redundant target retrieval results. This reduces the computational and storage overhead of data retrieval while improving the quality of retrieval results.

[0035] In one possible implementation of this invention, when generating an identifier fingerprint based on a feature sequence, the following methods can be used, but are not limited to: calculating the hash value corresponding to each feature item in the feature sequence to obtain a string corresponding to each feature item, wherein the length of the string is a preset fixed length; converting the string into a first feature vector, wherein each character position of the string corresponds to a vector component of the first feature vector, and the vector value of the vector component is determined according to the value of the corresponding character position; weighting the first feature vector according to the feature weights corresponding to the feature items to obtain a weighted vector, wherein the feature sequence contains multiple feature items and the feature weights of each feature item; accumulating the weighted vectors corresponding to each feature item into a floating-point vector, wherein the floating-point vector is a zero-based floating-point vector of a preset fixed length; and constructing an identifier fingerprint based on each first vector value of the accumulated floating-point vector.

[0036] In embodiments of the present invention, the feature items are the individual words, lexical units, etc., that constitute the feature sequence, which are keywords that carry the semantics of the data to be queried after preprocessing. For each feature item, a hash value is calculated using a preset hash function to obtain a fixed-length and randomly distributed binary string. The length of this string is equal to the preset fixed length of the fingerprint identifier, for example, 64 bits. The string consists of "0" and "1".

[0037] Then, the string corresponding to each feature term is converted into a first feature vector. The first feature vector is a mathematical vector of the same length as the string, and each vector component of the first feature vector corresponds one-to-one with each character position in the string.

[0038] The feature sequence includes not only multiple feature terms but also corresponding feature weights for each feature term. Feature weights are numerical values ​​used to quantify the importance of a word in the query data or domain. They can be calculated based on Term Frequency-Inverse Document Frequency (TF-IDF) values ​​or assigned using custom rules. The feature weights of each feature term are used as coefficients to multiply the first feature vector generated from that feature term to obtain a weighted vector.

[0039] A floating-point vector is a variable floating-point vector with the same preset fixed length as the fingerprint identifier, and all component values ​​are initially set to zero. The weighted vector corresponding to each feature term in the feature sequence is added to the corresponding component of the floating-point vector, one by one. This process continues until all weighted vectors corresponding to all feature terms in the feature sequence are added to the floating-point vector.

[0040] The final identifier fingerprint is constructed based on each first vector value of the accumulated floating-point vector (i.e., the value of the accumulated floating-point vector at each component).

[0041] In one possible implementation of this invention, when constructing an identifier fingerprint based on each first vector value of the accumulated floating-point vector, the following methods can be used, but are not limited to: traversing each first vector value of the accumulated floating-point vector and determining the index position of each first vector value in the accumulated floating-point vector; in response to a target first vector value being greater than a preset threshold, setting the index position corresponding to the target index position in the identifier fingerprint to a first value, wherein the target first vector value is any vector value of the accumulated floating-point vector, and the target index position is the index position of the target first vector value in the accumulated floating-point vector; in response to a target first vector value not being greater than a preset threshold, setting the index position corresponding to the target index position in the identifier fingerprint to a second value.

[0042] In an embodiment of the present invention, when traversing the accumulated floating-point vector, the position of the first vector value within the accumulated floating-point vector is determined; this position information is the index bit. The index bit has a one-to-one correspondence with the bits in the fingerprint identifier.

[0043] The preset threshold is a fixed value set in advance, such as 0. The purpose of this setting is to determine whether the value of the accumulated result is biased towards positive or negative.

[0044] When constructing the identifier fingerprint, the following methods can be used, but are not limited to: traversing each index bit of the accumulated floating-point vector and determining the magnitude of the first vector value at that index bit. If the first vector value is greater than zero, the bit corresponding to the index bit in the identifier fingerprint is set to the first value (e.g., 1); if the first vector value is less than or equal to zero, the bit corresponding to the index bit in the identifier fingerprint is set to the second value (e.g., 0). This then generates a binary identifier fingerprint consisting of a sequence of "0"s and "1"s.

[0045] In one possible implementation of this invention, when converting a string into a first feature vector, the following methods can be used, but are not limited to: in response to the value of the target character position being a third value, setting the target vector component in the first feature vector corresponding to the target character position as a first vector value, wherein the target character position is any character position in the string; in response to the value of the target character position being a fourth value, setting the target vector component in the first feature vector corresponding to the target character position as a second vector value.

[0046] In embodiments of the present invention, each position in the string is a character position. Simultaneously, an initial first feature vector of the same length as the string is pre-created. In this initial first feature vector, each vector component corresponds precisely to a character position. For example, the first character position of the string corresponds to the first vector component of the first feature vector, the second character position corresponds to the second vector component, and so on.

[0047] During the conversion, the value of the target character position is determined. When the value of the target character position is determined to be the third value (e.g., 1), the component value of the target vector component corresponding to the target character position in the first feature vector is set as the first vector value. The first vector value is a signed value, such as +1.

[0048] When the value of the target character position is determined to be the fourth value (e.g., 0), the component value of the target vector component corresponding to the target character position in the first feature vector is set as the second vector value. The second vector value is also a signed value, for example, -1.

[0049] In one possible implementation of this invention, to facilitate understanding of the construction of the identification fingerprint, this invention also provides a description of a method for constructing the identification fingerprint. Specifically, it may be implemented in ways including but not limited to the following: This method maps text features (i.e., feature sequences) to an f-bit (usually f=64) binary fingerprint (i.e., identifier fingerprint), ensuring that similar texts have similar fingerprints. The mathematical model is as follows: (1) Input: Information document (data to be queried) D1, after preprocessing, n feature words and their weight sequences (i.e., feature sequences) are obtained: (t1, w1), (t2, w2), ..., (t n , w n ).

[0050] (2) Output: The 64-bit SimHash fingerprint (identifier fingerprint) F of document D1 (can be stored as a 64-bit signed integer, i.e., BIGINT).

[0051] (3) Algorithm steps: Step 1: Initialize a 64-dimensional floating-point vector V, with all component values ​​set to 0; Step 2: For each feature word t i ; a. Calculate the standard 64-bit hash value h(t) of the word. i This will result in a 64-bit binary string (e.g., 110010...101). b. Convert the binary string into a feature vector H. i Iterate through each bit; if the bit value is 1, then H i The corresponding component is +1; if it is 0, the corresponding component is -1. c. The weight w i Multiply by the eigenvector H i The weighted vector W is obtained. i = w i H i ; d. Weight the vector W i Accumulated onto the overall vector V: V = V + W i .

[0052] Step 3: Generate the final fingerprint by iterating through each j of the 64-dimensional vector V: a. If V[j] > 0, then the j-th bit of the final fingerprint F is set to 1; b. If V[j]≤0, then the j-th bit of the final fingerprint F is set to 0.

[0053] Weight selection: Feature weight w i You can choose TF-IDF values, or custom weights that combine document characteristics (e.g., words appearing in keywords have higher weights).

[0054] In one possible implementation of this invention, when dividing multiple candidate data into multiple similar data groups based on the candidate identifier fingerprints corresponding to each of the multiple candidate data, the following methods can be used, but are not limited to: performing bitwise operations based on the candidate identifier fingerprints corresponding to each of the multiple candidate data to obtain the similarity between the multiple candidate data; dividing the candidate data with similarity greater than a preset similarity threshold into the same similar data group to obtain multiple similar data groups.

[0055] In embodiments of the present invention, similarity calculation mainly involves performing bitwise operations. Bitwise operations refer to performing logical operations bit by bit on two candidate fingerprints. For example, a bitwise XOR operation is used to compare each corresponding bit of the two candidate fingerprints. If the values ​​of the two bits are the same (both are "0" or both are "1"), the result of the operation on that bit is set to "0"; if the values ​​of the two bits are different (one is "0" and the other is "1"), the result of the operation on that bit is set to "1". After performing the bitwise XOR operation on all bits of the entire candidate fingerprint, a result string consisting of "0" and "1" is obtained. The number of "1"s in this result string is the Hamming distance between the two candidate fingerprints. The magnitude of this Hamming distance directly characterizes the degree of difference between the two candidate fingerprints. The smaller the Hamming distance, the more consistent the two candidate fingerprints are on more bits, indicating that the content of the two candidate data is more similar; conversely, the larger the Hamming distance, the more significant the difference.

[0056] After obtaining the similarity between multiple candidate data through bitwise operations, they are grouped according to a pre-set similarity threshold. The pre-set similarity threshold is a custom value. For example, the pre-set similarity threshold is set to a Hamming distance of 3.

[0057] The similarity between any two candidate data points is compared to a preset similarity threshold. When the similarity between two candidate data points is greater than the preset similarity threshold, it indicates that the difference in the identifier fingerprints of the two candidate data points is small, and at this point, the two candidate data points can be determined to be similar. Multiple candidate data points that are determined to be similar will be merged into the same similar data group.

[0058] In one possible embodiment, the division of the aforementioned similar data groups can also be achieved in the following way: Hamming distance is a metric that measures the difference between two strings of equal length (in this case, binary fingerprints, i.e., candidate identifier fingerprints). It is defined as the number of different characters at corresponding positions in the two strings. Given two 64-bit SimHash fingerprints F1 and F2, their Hamming distance d(F1, F2) is calculated as: d(F1, F2) = popcount(F1 XOR F2). Here, XOR is the bitwise XOR operation, where identical bits are returned as 0 and different bits as 1. popcount is a function that counts the number of 1s in the operands (bit counting).

[0059] Similarity determination: Set a threshold K (usually 3). If d(F1, F2)≥K, then the two data are considered similar; otherwise, they are not similar.

[0060] In one possible implementation of this invention, when selecting target data from multiple similar data groups based on data query instructions and the attribute information of each of the multiple candidate data, the following methods can be used, but are not limited to: determining the retrieval similarity of each of the multiple target candidate data in the target similar data group, wherein the target similar data group is any similar data group, and the retrieval similarity is determined by the word similarity between the query terms carried in the data query instructions and the data content of the target candidate data; and filtering data in the target similar data group based on the first attribute information corresponding to each of the multiple target candidate data and the retrieval similarity to obtain the target data, wherein the first attribute information includes at least the publication time, data source, and data integrity of the target candidate data.

[0061] In embodiments of the present invention, retrieval similarity is an indicator used to quantify the degree of data content matching between target candidate data and query terms. Retrieval similarity is calculated using the query terms carried in the data query instruction and the data content of the target candidate data. Specifically, lexical similarity can be used as the calculation basis. Lexical similarity can be obtained by analyzing the degree of overlap between the data content of the query terms and candidate data at the lexical unit level using similarity algorithms, such as TF-IDF, Best Matching 25 (BM25) algorithms, etc.

[0062] First attribute information refers to a variety of information that can reflect the inherent characteristics and quality of candidate data.

[0063] After obtaining the retrieval similarity and primary attribute information for each target candidate data, a comprehensive data filtering process is performed within the target similarity data group. This data filtering can integrate multiple factors from the retrieval similarity and primary attribute information to filter multiple target candidate data within the group. For example, it prioritizes data with the latest publication time, the most authoritative data source, the most complete content, and a high degree of relevance to the query intent.

[0064] In one possible implementation of this invention, to facilitate understanding of the deduplication process in the data retrieval process described above, this invention provides a schematic diagram of the data deduplication process, as follows: Figure 2 As shown, it includes the following: A[New Article Input] --> B{Text Preprocessing} B --> C [Word segmentation / stop word filtering] C --> D [SimHash fingerprint calculation] D --> E [Fingerprint storage, BIGINT column storage] E --> F [User Query] F --> G [Elasticsearch (ES) Recall Result Set] G --> H [Real-time Hamming distance comparison] H --> I [Similar Article Grouping] I --> J [Retain the top 1 document for each group] J --> K [Returns the deduplicated results] Among them, A [New Article Input] has the core objective of incorporating new policy texts to be processed into the system's data processing chain, providing the original data source for the subsequent deduplication process. The system receives newly added text data from external sources and transmits the original text data to the text preprocessing module to initiate deduplication preparations; B[Text Preprocessing] has the core objective of standardizing and cleaning the original text, eliminating irrelevant and interfering information, and providing a high-quality text foundation for subsequent word segmentation and fingerprint calculation. Before performing preprocessing, it is necessary to determine whether the original text meets the basic conditions for preprocessing, such as plain text and valid characters, and then perform the preprocessing operation. C[Word Segmentation / Stop Word Filtering] aims to break down preprocessed continuous text into its smallest semantic units (words), remove redundant words with no real semantic meaning, and focus on the core features of the text. It employs a Chinese word segmentation tool, combined with a domain dictionary (energy policy dictionary), to segment the text into word sequences. The core objective of D[SimHash fingerprint calculation] is to transform the filtered core word sequence into a fixed-length binary fingerprint, achieving a compressed mapping from text to numerical values ​​while ensuring that similar texts have similar fingerprints. It follows the SimHash algorithm steps, including word weighting, hash mapping, weighted summation, and fingerprint generation. E[Fingerprint Storage, BIGINT Column Storage] has the core goal of storing the generated SimHash fingerprint in an efficient and queryable manner, so that users can quickly retrieve and compare it when querying it later. It includes fingerprint format conversion, structured storage and index optimization. The core objective of F[User Query] is to receive users' search requests and transform them into query instructions that the system can process, triggering the subsequent deduplication process. First, the system receives the user's input query keywords or complete statements. Through semantic analysis, it clarifies the text range the user needs to search, converts the parsed keywords into query statements that the database or search engine can recognize, and performs deduplication to ensure that the retrieved results enter the deduplication process. G[ES Recall Result Set] has the core objective of quickly recalling a set of candidate texts related to the query requirements from a massive amount of policy texts based on user query commands, so as to narrow down the scope for subsequent deduplication and comparison, including retrieval matching and result set filtering. H[Real-time Hamming Distance Comparison] has the core objective of calculating the similarity of the SimHash fingerprints of each text in the candidate result set, and determining whether the texts are duplicates or highly similar. It includes Hamming distance calculation and similarity determination calculation. The core objective of I[Similar Article Grouping] is to group texts that are determined to be nearly identical after real-time Hamming distance comparison into the same group, forming duplicate text groups, in preparation for retaining the best text later. Using a Hamming distance less than or equal to a threshold as the core rule, a clustering algorithm is employed to group all texts that meet the fingerprint similarity threshold with a given text into one group, assigning a unique identifier to each duplicate group, and recording the metadata of each text within the group for subsequent selection of the best text. J[Retain Top 1 Document in Each Group] has the core objective of selecting the highest quality and most relevant document to the user's needs from each group of duplicate texts and retaining it as a representative document, while removing the rest of the duplicate or highly similar texts. K[Return Deduplication Results] has the core objective of sorting the retained documents after deduplication according to their relevance to the user's query and providing clear feedback to the user, thus completing the entire deduplication process.

[0065] Furthermore, during the deduplication process, the algorithms and parameters involved will be optimized, specifically including: SimHash algorithm debugging: The word segmentation results were tested to assess the accuracy of different word segmentation tools and custom dictionaries in segmenting patent-related terms, ensuring that core features were effectively extracted.

[0066] Weighting strategies are compared to assess the impact of different weighting schemes, such as TF-IDF, on deduplication effectiveness. The optimal strategy can be determined by manually evaluating its performance on small sample datasets.

[0067] Optimization of Hamming distance threshold K: Construct a labeled test set containing positive samples (true duplicate pairs) and negative samples (non-duplicate pairs). Iterate through different K values ​​(e.g., 1-5) and calculate the precision and recall for each K value. Plot the PR curve or calculate the F1-Score, and select the optimal K value based on business requirements (whether to prioritize precision or recall), for example, K=3 for 64-bit fingerprints.

[0068] In one possible implementation of this invention, when scoring multiple target data based on their respective multidimensional data information, the following methods can be used, but are not limited to: matching multiple types of first information in the target multidimensional data information with their respective corresponding preset score tables to obtain information scores corresponding to each type of first information, wherein the target multidimensional data information is the multidimensional data information of any target data, and the multiple types of first information at least include the data source and data integrity of the target data; calculating the target popularity value based on the second information in the target multidimensional data information, wherein the second information at least includes the publication time and interaction data of the target data; and weighting and summing the retrieval similarity between the target data and the query terms carried by the data query instruction according to preset weight coefficients, the multiple information scores, and the target popularity value to obtain the data score of the target data.

[0069] In embodiments of the present invention, before sorting multiple target data, each target data needs to be comprehensively scored. For any target data to be scored, a score is given based on its multidimensional data information, i.e., target multidimensional data information. Target multidimensional data information can be divided into multiple categories of first information and second information based on different attributes and uses of the information.

[0070] Multi-category primary information refers to attribute information with clear classification labels that can be mapped to tables using preset standards. Specifically, multi-category primary information includes at least the data source and data integrity of the target data.

[0071] The second piece of information includes at least the publication time of the target data and interaction data. The publication time records the moment when the target data was generated or published; the interaction data records quantitative indicators of the interaction between the target data and users during the dissemination process, such as: the number of views, likes, shares, or other statistical data reflecting user engagement.

[0072] The preset weighting coefficients are custom-defined coefficients, including weight parameters corresponding to retrieval similarity, various information scores, and target popularity value. The retrieval similarity between the target data and the query term, the multiple information scores matched from multiple types of first information, and the target popularity value calculated from the second information are each multiplied by their respective preset weighting coefficients, and then summed to obtain the data score of the target data.

[0073] In one possible implementation of this invention, the calculation process of the above-mentioned data scoring can also be implemented in, but is not limited to, the following ways: Given target data D and a query term Q, the calculation model for its data score FinalScore (D, Q) is as follows: FinalScore(D, Q) = w1 keywordScore(D, Q) + w2 mediaWeight(D) + w3 sourceWeight(D) + w4 The hotScore(D) model is explained as follows: (1) keywordScore(D, Q): The relevance score between the query term and the target data, calculated by the underlying search engine (such as Elasticsearch).

[0074] (2) mediaWeight(D): The authority score of the target data source media.

[0075] (3) sourceWeight(D): The score of the target data release channel type.

[0076] (4) hotScore(D): The popularity score of the target data, which can be calculated by combining the release time, click rate, interaction volume, etc.

[0077] (5) w1, w2, w3, w4: are the weight coefficients of the corresponding dimensions, and satisfy w1+w2+w3+w4=1.

[0078] Based on business needs, the weights can be freely allocated. For example, keyword relevance accounts for 50% (w1=0.5), media weight accounts for 25% (w2=0.25), data channel weight accounts for 15% (w3=0.15), and popularity value accounts for 10% (w4=0.10). This weight can be dynamically adjusted according to the actual results.

[0079] (1) Relevance score (keywordScore): This score is calculated in real time by relevance algorithms such as BM25 in Elasticsearch, and the score range is usually between [0, 1). To eliminate the difference in the units of the scores between different queries, the returned raw score needs to be normalized to the minimum-maximum value (Min-Max Normalization) to transform it to the range of [0, 1]. During the query, the contribution of the title field to the query terms is increased by using the should clause and boost parameter in Elasticsearch. For example, the weight boost factor (boost) of the title field is set to 2, and the body field is set to 1, thereby realizing the business rule of prioritizing title hits.

[0080] (2) Media Weight Based on the authority and influence of media organizations, a media weight dictionary is pre-established, with weight values ​​defined in the range of [0, 1]. Table 1 below provides an example of the media dictionary: Table 1

[0081] (3) Data channel information score (sourceWeight) Based on the content quality, completeness, and reliability of different content channels, a channel weight dictionary is pre-established, with the weight value range defined as [0, 1]. The channel weight dictionary can be illustrated by the example in Table 2 below.

[0082] Table 2

[0083] (4) Hot Score Hotness is a metric that combines time decay and user interaction behavior. A commonly used formula is as follows: hotScore = log(1 + interaction) timeDecay. Wherein: Interaction data, such as the number of reads, likes, and comments, can be represented by any one of these metrics or a weighted sum. The log function is used to smooth the data and prevent individual highly interactive articles from receiving excessively high scores.

[0084] timeDecay, the time decay factor, such as: λ is the time elapsed since the article's publication, where Δt is the time elapsed since the article's publication, and λ is the decay coefficient. This value is close to 1 for newly published articles and close to 0 for older articles.

[0085] Normalization: The calculated original heat value needs to be normalized and mapped to the [0, 1] interval.

[0086] In one possible implementation of this invention, when sorting multiple target data according to their respective data scores to obtain target retrieval results, the following methods can be used, but are not limited to: sorting multiple target data in descending or ascending order according to their respective data scores to obtain target retrieval results.

[0087] In embodiments of the present invention, data scoring can measure the value of target data and user query intent in multiple dimensions such as content relevance, source authority, timeliness, and popularity.

[0088] To obtain the final search results, multiple target data points need to be processed in an ordered manner. This involves sorting the target data points according to their respective data score values. The sorting strategy can be pre-configured as descending or ascending based on business needs. Descending order is typically used.

[0089] In one possible implementation of this invention, to facilitate understanding of the data sorting process in this embodiment, a schematic diagram of the data sorting process is provided, as shown below. Figure 3 As shown, it includes: Query Recall: After a user initiates a query, Elasticsearch (ES) is first invoked to recall a preliminary candidate document set C (i.e., multiple candidate data) based on keyword matching, and the original relevance score (retrieval similarity) calculated by ES for each document is obtained.

[0090] Data Acquisition: Retrieve metadata information for each document in the candidate set C in batches from the database or cache, including: media type, publishing channel, publishing time, and interaction data (reads, likes, etc.).

[0091] Overall score calculation: Normalize the ES score to obtain the keywordScore; look up the mediaWeight and sourceWeight in the table according to the media type and channel type; Based on interaction data and posting time, the hotScore is calculated in real time and normalized; then substituted into the comprehensive scoring formula, for example: FinalScore=w1×keywordScore+w2×mediaWeight+w3×sourceWeight+w4×hotScore.

[0092] Sorting and Returning: Sort the documents in the candidate set C in descending order based on the calculated final score (i.e., data score), and return the sorted list to the user front end.

[0093] Furthermore, during the sorting process, the algorithms and parameters involved will be optimized, specifically including: Weight optimization: The weight coefficients (w1, w2, w3, w4) in the formula are the core of optimization and can be achieved using grid search or A / B testing. Prepare a well-labeled query-document set (labeling documents considered high quality by the user), calculate the ranking results under different weight combinations (e.g., Normalized Discounted Cumulative Gain (NDCG), Mean Reciprocal Rank (MRR), etc.), and select the weight combination with the optimal metrics.

[0094] Normalization method optimization: The data distribution differs across dimensions, requiring analysis of distribution plots such as score and hot score to determine whether to use Min-Max normalization, Z-Score standardization, or quantile transformation, in order to avoid certain dimensions being dominated by a few extreme values.

[0095] Popularity formula optimization: The combination of the decay coefficient λ and the interaction data needs to be adjusted. A suitable decay coefficient can be fitted by analyzing the click-through rate curve of historical data over time.

[0096] In one possible implementation of this invention, when preprocessing the multiple data to be queried, the following methods can be used, but are not limited to: cleaning the multiple data to be queried to obtain multiple cleaned data; performing word segmentation on the multiple cleaned data to obtain word sequences corresponding to each of the multiple cleaned data; and filtering the multiple word sequences to obtain feature sequences corresponding to each of the multiple data to be queried.

[0097] In embodiments of the present invention, the data to be queried is standardized and cleaned to eliminate irrelevant and interfering information, providing a high-quality text foundation for subsequent word segmentation and fingerprint calculation. Before preprocessing, it is necessary to determine whether the data to be queried meets the basic conditions for preprocessing, such as plain text and valid characters, before performing the preprocessing operation. The preprocessed continuous text is broken down into its smallest semantic units (words, lexical units), and redundant words without actual semantic meaning are removed, focusing on the core features of the text. For example, a Chinese word segmentation tool is used in conjunction with a domain dictionary to break the text down into word sequences.

[0098] Corresponding to the data retrieval method described above, the present invention also proposes a data retrieval apparatus. Since the apparatus embodiments of the present invention correspond to the method embodiments described above, details not disclosed in the apparatus embodiments can be referred to the method embodiments described above, and will not be repeated here.

[0099] Figure 4 This is a schematic diagram of the structure of a data retrieval device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, it includes: The processing unit 41 is used to preprocess the acquired multiple data to be queried to obtain the feature sequences corresponding to each of the multiple data to be queried. The generation unit 42 is used to generate an identifier fingerprint based on a feature sequence, wherein the feature sequence is used to characterize the data content of the data to be queried, and the identifier fingerprint is a binary fingerprint with a preset fixed length. The determining unit 43 is used to determine multiple candidate data corresponding to the data query instruction from multiple data to be queried in response to the data query instruction; The partitioning unit 44 is used to divide multiple candidate data into multiple similar data groups based on the candidate identifier fingerprints corresponding to each candidate data. The filtering unit 45 is used to filter out the corresponding target data from multiple similar data groups according to the data query instruction and the attribute information of each of the multiple candidate data. The sorting unit 46 is used to score multiple target data according to their respective multidimensional data information, and sort the multiple target data according to their respective data scores to obtain target retrieval results.

[0100] Furthermore, in one possible implementation of this embodiment of the invention, the generating unit 42 is specifically used for: Calculate the hash value corresponding to each feature item in the feature sequence to obtain the string corresponding to each feature item. The length of the string is a preset fixed length. The string is converted into a first feature vector, where each character position of the string corresponds to a vector component of the first feature vector, and the vector value of the vector component is determined according to the value of the corresponding character position. The first feature vector is weighted according to the feature weights corresponding to the feature terms to obtain a weighted vector. The feature sequence contains multiple feature terms and their respective feature weights. The weighted vectors corresponding to each feature term are summed into a floating-point vector, where the floating-point vector is a zero-type floating-point vector of a preset fixed length; Construct an identifier fingerprint based on each first vector value of the accumulated floating-point vector.

[0101] Furthermore, in one possible implementation of this invention, the generating unit 42 is specifically used for: Iterate through each first vector value of the accumulated floating-point vector and determine the index of each first vector value in the accumulated floating-point vector; In response to the target first vector value being greater than a preset threshold, the index bit in the identifier fingerprint corresponding to the target index bit is set to the first value, wherein the target first vector value is any vector value of the accumulated floating-point vector, and the target index bit is the index bit of the target first vector value in the accumulated floating-point vector; In response to the first target vector value not being greater than a preset threshold, the index bit in the identifier fingerprint corresponding to the target index bit is set to the second value.

[0102] Furthermore, in one possible implementation of this invention, the generating unit 42 is specifically used for: In response to the value of the target character position being the third value, the target vector component in the first feature vector corresponding to the target character position is set as the first vector value, where the target character position is any character position in the string; In response to the value of the target character position being the fourth value, the target vector component in the first feature vector corresponding to the target character position is set as the second vector value.

[0103] Furthermore, in one possible implementation of this invention, the dividing unit 44 is specifically used for: Perform bitwise operations on the candidate identifier fingerprints corresponding to each of the multiple candidate data to obtain the similarity between the multiple candidate data; Candidate data with a similarity greater than a preset similarity threshold are grouped into the same similar data group, resulting in multiple similar data groups.

[0104] Furthermore, in one possible implementation of this embodiment of the invention, the filtering unit 45 is specifically used for: Determine the retrieval similarity of each of the multiple target candidate data in the target similar data group. The target similar data group is any similar data group. The retrieval similarity is determined by the word similarity between the query terms carried in the data query instruction and the data content of the target candidate data. Based on the first attribute information and retrieval similarity of multiple target candidate data, data is filtered in the target similar data group to obtain target data. The first attribute information includes at least the publication time, data source and data completeness of the target candidate data.

[0105] Furthermore, in one possible implementation of this invention, the sorting unit 46 is specifically used for: The first information of each category in the target multidimensional data information is matched with its corresponding preset score table to obtain the information score corresponding to each category of the first information. The target multidimensional data information is the multidimensional data information of any target data, and the first information of each category includes at least the data source and data integrity of the target data. The target popularity value is calculated based on the second information in the target's multidimensional data information, wherein the second information includes at least the release time of the target data and the interaction data; The retrieval similarity between the target data and the query terms carried in the data query command; Based on preset weighting coefficients, the retrieval similarity between the target data and the query terms carried by the data query command, multiple information scores, and target popularity value are weighted and summed to obtain the data score of the target data.

[0106] Furthermore, in one possible implementation of this invention, the sorting unit 46 is specifically used for: Multiple target data are sorted in descending or ascending order according to their respective data scores to obtain target retrieval results.

[0107] Furthermore, in one possible implementation of this invention, the processing unit 41 is specifically used for: Data cleaning is performed on multiple sets of data to be queried, resulting in multiple cleaned sets of data. Each of the cleaned data sets was segmented into words to obtain the word sequence corresponding to each of the cleaned data sets. Lexical filtering is performed on multiple lexical sequences to obtain the feature sequences corresponding to each of the multiple query data.

[0108] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of the present invention, and the principle is the same. Therefore, the present invention will not limit the scope of the apparatus.

[0109] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0110] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0111] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 502 or loaded from storage unit 508 into RAM (Random Access Memory) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. I / O (Input / Output) interface 505 is also connected to bus 504.

[0112] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0113] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as data retrieval methods. For example, in some embodiments, the data retrieval method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the aforementioned data retrieval method by any other suitable means (e.g., by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fibers, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0119] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0120] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data retrieval method, characterized in that, include: The acquired multiple data to be queried are preprocessed to obtain the feature sequences corresponding to each of the multiple data to be queried, and an identification fingerprint is generated based on the feature sequences. The feature sequences are used to characterize the data content of the data to be queried, and the identification fingerprint is a binary fingerprint with a preset fixed length. In response to a data query command, multiple candidate data corresponding to the data query command are determined from multiple data to be queried, and the multiple candidate data are divided into multiple similar data groups according to the candidate identifier fingerprints corresponding to each of the multiple candidate data. Based on the data query instruction and the attribute information of each of the multiple candidate data, the corresponding target data is selected from each of the multiple similar data groups; The multiple target data are scored according to their respective multidimensional data information, and then sorted according to their respective data scores to obtain the target retrieval results.

2. The data retrieval method according to claim 1, characterized in that, The step of generating an identifier fingerprint based on the feature sequence includes: Calculate the hash value corresponding to each feature item in the feature sequence to obtain the string corresponding to each feature item, wherein the length of the string is the preset fixed length; The string is converted into a first feature vector, wherein each character position of the string corresponds to a vector component of the first feature vector, and the vector value of the vector component is determined according to the value of the corresponding character position. The first feature vector is weighted according to the feature weights corresponding to the feature terms to obtain a weighted vector, wherein the feature sequence includes multiple feature terms and the feature weights of each of the multiple feature terms; The weighted vectors corresponding to each feature item are summed into a floating-point vector, wherein the floating-point vector is a zero-type floating-point vector of a preset fixed length; The identifier fingerprint is constructed based on each first vector value of the accumulated floating-point vector.

3. The data retrieval method according to claim 2, characterized in that, The step of constructing the identifier fingerprint based on each first vector value of the accumulated floating-point vector includes: Iterate through each first vector value of the accumulated floating-point vector and determine the index of each first vector value in the accumulated floating-point vector; In response to the target first vector value being greater than a preset threshold, the index bit corresponding to the target index bit in the identification fingerprint is set to a first value, wherein the target first vector value is any vector value of the accumulated floating-point vector, and the target index bit is the index bit of the target first vector value in the accumulated floating-point vector; In response to the first target vector value not being greater than the preset threshold, the index bit in the identifier fingerprint corresponding to the target index bit is set to the second value.

4. The data retrieval method according to claim 2, characterized in that, The step of converting the string into a first feature vector includes: In response to the value of the target character position being a third value, the target vector component in the first feature vector corresponding to the target character position is set as the first vector value, wherein the target character position is any character position in the string; In response to the value of the target character position being a fourth value, the target vector component in the first feature vector corresponding to the target character position is set as the second vector value.

5. The data retrieval method according to claim 1, characterized in that, The step of dividing the multiple candidate data into multiple similar data groups based on the candidate identifier fingerprints corresponding to each of the multiple candidate data includes: Perform bitwise operations based on the candidate identifier fingerprints corresponding to each of the multiple candidate data to obtain the similarity between the multiple candidate data; Candidate data with a similarity greater than a preset similarity threshold are divided into the same similar data group to obtain the multiple similar data groups.

6. The data retrieval method according to claim 1, characterized in that, The step of filtering out the corresponding target data from the multiple similar data groups according to the data query instruction and the attribute information of each of the multiple candidate data includes: Determine the retrieval similarity of each of the multiple target candidate data in the target similar data group, wherein the target similar data group is any similar data group, and the retrieval similarity is determined by the word similarity between the query term carried by the data query instruction and the data content of the target candidate data; Based on the first attribute information corresponding to each of the multiple target candidate data and the retrieval similarity, data is filtered in the target similar data group to obtain the target data. The first attribute information includes at least the publication time, data source and data integrity of the target candidate data.

7. The data retrieval method according to claim 1, characterized in that, The step of scoring multiple target data based on their respective multidimensional data information includes: The first information of the target multidimensional data information is matched with its corresponding preset score table to obtain the information score corresponding to each of the first information. The target multidimensional data information is the multidimensional data information of any target data, and the first information includes at least the data source and data integrity of the target data. The target popularity value is calculated based on the second information in the target multidimensional data information, wherein the second information includes at least the publication time and interaction data of the target data; The retrieval similarity between the target data and the query terms carried by the data query instruction; Based on preset weighting coefficients, the retrieval similarity between the target data and the query terms carried by the data query instruction, multiple information scores, and the target popularity value are weighted and summed to obtain the data score of the target data.

8. The data retrieval method according to claim 1, characterized in that, The step of sorting the multiple target data according to their respective data scores to obtain target retrieval results includes: The target data are sorted in descending or ascending order according to their respective data scores to obtain the target retrieval results.

9. The data retrieval method according to claim 1, characterized in that, The step of preprocessing the acquired multiple data sets to obtain the feature sequences corresponding to each of the multiple data sets to be queried includes: The multiple sets of data to be queried are cleaned to obtain multiple cleaned data sets. Each of the multiple cleaned data sets is segmented into words to obtain the word sequence corresponding to each of the multiple cleaned data sets. Each of the multiple word sequences is filtered to obtain the feature sequence corresponding to each of the multiple query data.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

Images