Search result filtering method and apparatus, and electronic device

Abstract

The application discloses a search result filtering method and device and electronic equipment, and belongs to the technical field of computers. The method comprises the following steps: obtaining a to-be-filtered search result; performing word segmentation processing on the to-be-filtered search result to obtain a to-be-matched word segmentation set; obtaining a hidden keyword set associated with a search target matched with the to-be-filtered search result; and filtering the to-be-filtered search result according to the hidden keyword set and the to-be-matched word segmentation set to obtain the search result matched with the search target. The hidden keyword set obtained by prederivation is used for automatically filtering the search result, the data hidden features concerned by the search target are fully utilized, the filtering efficiency of the search result is improved, and the accuracy of the search result filtering is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to search result filtering methods, apparatus, electronic devices, and computer-readable storage media. Background Technology

[0002] In data mining and information gathering tasks, it is often necessary to use specific keywords for searching and matching. However, due to the limitations of current search engines and data warehouse algorithms, some information obtained through keyword searches is irrelevant to the target task. Existing technologies for filtering search results mainly include keyword filtering and manual filtering. Filtering search results through keyword matching has too low an accuracy rate, while manual filtering is time-consuming and labor-intensive.

[0003] In conclusion, an accurate and efficient method for filtering search results is needed. Summary of the Invention

[0004] This application provides a search result filtering method, apparatus, and electronic device that can solve the problems of high labor costs, low efficiency, and low accuracy in filtering irrelevant search results.

[0005] In a first aspect, embodiments of this application disclose a search result filtering method, including:

[0006] Get the search results to be filtered;

[0007] The search results to be filtered are segmented to obtain a set of segmented words to be matched;

[0008] Obtain the set of hidden keywords associated with the search target that matches the search results to be filtered;

[0009] Based on the set of hidden keywords and the set of words to be matched, the search results to be filtered are obtained to obtain the search results that match the search target.

[0010] Optionally, the set of hidden keywords is derived in advance using the following method:

[0011] Obtain the set of positive samples and the set of negative samples corresponding to the search target;

[0012] The samples in the positive sample set and the negative sample set are processed by word segmentation to obtain a sample word segmentation set;

[0013] Using the sample word segmentation set as the set of search keywords, a hidden keyword set is obtained by reverse derivation through a preset Bayesian model, which is included in the positive sample set but not in the negative sample set.

[0014] Optionally, the set of sample word segments is used as the set of search keywords. A preset Bayesian model is used for reverse derivation to obtain hidden keywords included in the positive sample set but not in the negative sample set, forming a hidden keyword set, including:

[0015] The sample word segmentation set is used as the set of search keywords;

[0016] The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the positive sample set is derived by back-inferring the preset Bayesian model, and is used as the first candidate hidden keyword.

[0017] The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the negative sample set is derived by reverse deduction using the preset Bayesian model, and is used as the second candidate hidden keyword.

[0018] The first candidate hidden keywords located outside the intersection of the first candidate hidden keywords and the second candidate hidden keywords are used as hidden keywords to form a set of hidden keywords.

[0019] Optionally, the step of back-deriving the candidate keyword with the highest probability of appearing in the positive sample set under the condition that the search keyword appears in the positive sample set through a preset Bayesian model, and using it as the first candidate hidden keyword, includes:

[0020] Obtain the first probability of a candidate keyword appearing in a sample set, wherein the sample set includes: the positive sample set and the negative sample set;

[0021] Given that the candidate keyword appears in the positive sample set, obtain a second probability of the search keyword appearing;

[0022] The first logarithmic probability corresponding to the candidate keyword is obtained by summing the logarithmic values ​​of the first probability and the second probability.

[0023] The candidate keyword that maximizes the first logarithmic probability is selected as the first candidate hidden keyword.

[0024] Optionally, the step of back-deriving the candidate keyword with the highest probability of appearing in the negative sample set under the condition that the search keyword appears in the preset Bayesian model, as the second candidate hidden keyword, includes:

[0025] Obtain the third probability of a candidate keyword appearing in a sample set, wherein the sample set includes: the positive sample set and the negative sample set;

[0026] Given that the candidate keyword appears in the negative sample set, obtain the fourth probability of the search keyword appearing;

[0027] The second logarithmic probability corresponding to the candidate keyword is obtained by summing the logarithmic values ​​of the third probability and the fourth probability.

[0028] The candidate keyword that maximizes the second logarithmic probability will be selected as the second candidate hidden keyword.

[0029] Optionally, the step of performing word segmentation on the samples in the positive sample set and the negative sample set to obtain a sample word segmentation set includes:

[0030] The samples in the positive sample set and the negative sample set are segmented into words to obtain several candidate sample word segments;

[0031] Based on the positive sample set and the negative sample set, the inverse text word frequency of each candidate sample word segment is calculated;

[0032] Candidate samples with word frequencies greater than a preset word frequency threshold are selected for word segmentation to form a sample word segmentation set.

[0033] Optionally, the step of filtering the search results to be filtered based on the hidden keyword set and the word segmentation set to be matched, to obtain the search results matching the search target, includes:

[0034] Obtain the intersection of the hidden keyword set and the word segmentation set to be matched, and use it as the intersection keyword;

[0035] Based on the proportion of the intersection keywords in the set of hidden keywords, obtain the predicted matching degree between the search results to be filtered and the search target;

[0036] The search results to be filtered are filtered based on the predicted matching degree to obtain the search results that match the search target.

[0037] Secondly, embodiments of this application disclose a search result filtering device, including:

[0038] The module for obtaining search results to be filtered is used to obtain search results to be filtered.

[0039] The module for obtaining the word segmentation set to be matched is used to perform word segmentation processing on the search results to be filtered and obtain the word segmentation set to be matched.

[0040] The hidden keyword set acquisition module is used to acquire the set of hidden keywords associated with the search target that matches the search results to be filtered;

[0041] The search result filtering module is used to filter the search results to be filtered based on the hidden keyword set and the word segmentation set to be matched, so as to obtain the search results that match the search target.

[0042] Optionally, the device further includes: a hidden keyword set derivation module, which is used to pre-derive the hidden keyword set using the following method:

[0043] Obtain the set of positive samples and the set of negative samples corresponding to the search target;

[0044] The samples in the positive sample set and the negative sample set are processed by word segmentation to obtain a sample word segmentation set;

[0045] Using the sample word segmentation set as the set of search keywords, a hidden keyword set is obtained by reverse derivation through a preset Bayesian model, which is included in the positive sample set but not in the negative sample set.

[0046] Optionally, the set of sample word segments is used as the set of search keywords. A preset Bayesian model is used for reverse derivation to obtain hidden keywords included in the positive sample set but not in the negative sample set, forming a hidden keyword set, including:

[0047] The sample word segmentation set is used as the set of search keywords;

[0048] The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the positive sample set is derived by back-inferring the preset Bayesian model, and is used as the first candidate hidden keyword.

[0049] The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the negative sample set is derived by reverse deduction using the preset Bayesian model, and is used as the second candidate hidden keyword.

[0050] The first candidate hidden keywords located outside the intersection of the first candidate hidden keywords and the second candidate hidden keywords are used as hidden keywords to form a set of hidden keywords.

[0051] Optionally, the step of back-deriving the candidate keyword with the highest probability of appearing in the positive sample set under the condition that the search keyword appears in the positive sample set through a preset Bayesian model, and using it as the first candidate hidden keyword, includes:

[0052] Obtain the first probability of a candidate keyword appearing in a sample set, wherein the sample set includes: the positive sample set and the negative sample set;

[0053] Given that the candidate keyword appears in the positive sample set, obtain a second probability of the search keyword appearing;

[0054] The first logarithmic probability corresponding to the candidate keyword is obtained by summing the logarithmic values ​​of the first probability and the second probability.

[0055] The candidate keyword that maximizes the first logarithmic probability is selected as the first candidate hidden keyword.

[0056] Optionally, the step of back-deriving the candidate keyword with the highest probability of appearing in the negative sample set under the condition that the search keyword appears in the preset Bayesian model, as the second candidate hidden keyword, includes:

[0057] Obtain the third probability of a candidate keyword appearing in a sample set, wherein the sample set includes: the positive sample set and the negative sample set;

[0058] Given that the candidate keyword appears in the negative sample set, obtain the fourth probability of the search keyword appearing;

[0059] The second logarithmic probability corresponding to the candidate keyword is obtained by summing the logarithmic values ​​of the third probability and the fourth probability.

[0060] The candidate keyword that maximizes the second logarithmic probability will be selected as the second candidate hidden keyword.

[0061] Optionally, the step of performing word segmentation on the samples in the positive sample set and the negative sample set to obtain a sample word segmentation set includes:

[0062] The samples in the positive sample set and the negative sample set are segmented into words to obtain several candidate sample word segments;

[0063] Based on the positive sample set and the negative sample set, the inverse text word frequency of each candidate sample word segment is calculated;

[0064] Candidate samples with word frequencies greater than a preset word frequency threshold are selected for word segmentation to form a sample word segmentation set.

[0065] Optionally, the search result filtering module is further configured to:

[0066] Obtain the intersection of the hidden keyword set and the word segmentation set to be matched, and use it as the intersection keyword;

[0067] Based on the proportion of the intersection keywords in the set of hidden keywords, obtain the predicted matching degree between the search results to be filtered and the search target;

[0068] The search results to be filtered are filtered based on the predicted matching degree to obtain the search results that match the search target.

[0069] Thirdly, embodiments of this application also disclose an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the search result filtering method described in embodiments of this application.

[0070] Fourthly, embodiments of this application disclose a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, represents the steps of the search result filtering method disclosed in embodiments of this application.

[0071] The search result filtering method disclosed in this application involves: obtaining search results to be filtered; performing word segmentation on the search results to obtain a set of words to be matched; obtaining a set of hidden keywords associated with the search target that matches the search results to be filtered; and filtering the search results to be filtered based on the set of hidden keywords and the set of words to be matched to obtain the search results that match the search target. By using a pre-derived set of hidden keywords to automatically further filter the search results, this method fully utilizes the hidden data features that the search target is interested in, thereby improving not only the filtering efficiency of search results but also the accuracy of the filtering.

[0072] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0073] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0074] Figure 1 This is a flowchart of the search result filtering method disclosed in the embodiments of this application;

[0075] Figure 2 This is another flowchart of the search result filtering method disclosed in the embodiments of this application;

[0076] Figure 3 This is a schematic diagram illustrating the process of deriving the hidden keyword set in the search result filtering method disclosed in this application embodiment;

[0077] Figure 4 This is a schematic diagram illustrating an application scenario of the search result filtering method disclosed in the embodiments of this application;

[0078] Figure 5 This is one of the schematic diagrams of the search result filtering device disclosed in the embodiments of this application;

[0079] Figure 6 This is the second schematic diagram of the structure of the search result filtering device disclosed in the embodiments of this application;

[0080] Figure 7 A block diagram schematically illustrates an electronic device for performing the method according to this application; and

[0081] Figure 8 A storage unit for holding or carrying program code implementing the method according to this application is illustrated schematically. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0083] The following describes specific implementations of the search result filtering method disclosed in the embodiments of this application by way of example.

[0084] like Figure 1 As shown in the embodiment of this application, a search result filtering method includes steps 110 to 140.

[0085] Step 110: Obtain the search results to be filtered.

[0086] The search results to be filtered can be search results retrieved based on search keywords, or search results obtained through other means.

[0087] For example, in one application scenario, the search results to be filtered are the search results retrieved by the search engine after the user enters search keywords. As another example, when filtering content for a specific section on a website, the search results are documents in that section crawled using web crawlers or other website content scraping technologies.

[0088] Typically, the search results to be filtered are unstructured documents.

[0089] Step 120: Perform word segmentation on the search results to be filtered to obtain a set of words to be matched.

[0090] Next, to facilitate document comparison, the search results to be filtered are first segmented into words to obtain the words that make up the search results. Typically, segmenting the search results yields several words. In this embodiment, the words obtained from segmenting the search results are used to form a set of words to be matched.

[0091] Optionally, the specific implementation of segmenting the search results to be filtered into words can be found in the prior art, and will not be repeated in the embodiments of this application.

[0092] Step 130: Obtain the set of hidden keywords associated with the search target that matches the search results to be filtered.

[0093] Optionally, the search target is the final desired search result. The hidden keywords included in the hidden keyword set are keywords other than the search keywords entered by the user, and the hidden keywords have a strong feature representation ability for the search target.

[0094] In embodiments of this application, in order to further determine whether the search results to be filtered are the target search results of interest based on the word segments included in the search results to be filtered, it is necessary to pre-derive a set of hidden keywords. Then, in the application phase, a set of hidden keywords associated with the search target that matches the search results to be filtered can be obtained and used to match the set of word segments to be matched.

[0095] In some embodiments of this application, the set of hidden keywords is obtained by reverse inference analysis based on sample data.

[0096] The method for deriving the set of hidden keywords is described below and will not be repeated here.

[0097] Step 140: Filter the search results to be filtered based on the hidden keyword set and the word segmentation set to be matched, to obtain the search results that match the search target.

[0098] Next, the matching degree between the search results and the search target needs to be determined based on the matching degree of the words in the word segmentation set to be matched in the hidden keyword set, so as to filter the search results.

[0099] Optionally, the step of filtering the search results to be filtered based on the hidden keyword set and the word segmentation set to be matched to obtain the search results matching the search target includes: obtaining the intersection of the hidden keyword set and the word segmentation set to be matched as the intersection keywords; obtaining the predicted matching degree between the search results to be filtered and the search target based on the proportion of the intersection keywords in the hidden keyword set; and filtering the search results to be filtered based on the predicted matching degree to obtain the search results matching the search target.

[0100] Taking the derived set of hidden keywords as denoted by the symbol "S" and the set of words to be matched as denoted by the symbol "T" as an example, the predicted matching degree between the search results to be filtered and the search target can be calculated using the formula: predict = (T∩S) / S. From the calculation method of the predicted matching degree value predict, it can be seen that the larger the predicted matching degree value predict, the more likely the search results to be filtered are the data of interest.

[0101] For a specific example, if the hidden keyword set S contains M words, and the word segmentation set T obtained from the segmentation of the search result doc1 to be filtered contains N words, where n words in the word segmentation set T are elements of the hidden keyword set S, and n < M, then the predicted match degree of the search result doc1 to the search target is n / M. That is, the probability that the search result doc1 is the search target can be expressed as n / M, where n is a natural number, and M and N are positive integers.

[0102] In practical applications, if multiple search results to be filtered are obtained (for example, a search engine recalls multiple search results for a single user-input keyword), for each search result to be filtered, a set of words to be matched, T, can be obtained. By matching this set with the hidden keyword set S, a predicted matching degree value, predict, can be obtained for each search result to be filtered. Then, the search results can be filtered based on the predicted matching degree value, predict, for each search result to be filtered.

[0103] In some embodiments of this application, search results can be filtered by pre-setting a matching degree prediction threshold. For example, search results with a matching degree prediction value greater than or equal to the matching degree prediction threshold are selected as the search results that match the search target.

[0104] In some other embodiments of this application, a preset number of search results to be filtered may be selected as the search results that match the search target, based on the highest predicted matching value.

[0105] This application does not limit the specific implementation of filtering the search results to be filtered based on the predicted matching degree to obtain the search results that match the search target.

[0106] To make the search result filtering method disclosed in the embodiments of this application clearer, the derivation method of the hidden keyword set will be further explained below.

[0107] like Figure 2 As shown, the method further includes step 100.

[0108] Step 100: Derive the set of hidden keywords associated with the specified search target.

[0109] In the step of deriving the set of hidden keywords associated with a specified search target, the set of hidden keywords is derived in advance by the following method: obtaining a set of positive samples and a set of negative samples corresponding to the search target; performing word segmentation on the samples in the set of positive samples and the set of negative samples to obtain a set of sample word segments; using the set of sample word segments as the set of search keywords, performing reverse derivation through a preset Bayesian model to obtain hidden keywords that are included in the set of positive samples but not included in the set of negative samples, thus forming the set of hidden keywords.

[0110] The following is combined Figure 3 The diagram shown illustrates the derivation process of the hidden keyword set, and provides an example of how to derive the hidden keyword set.

[0111] Taking the search target as documents containing the word "telecom" and describing the topology of a telecommunications network, this paper illustrates the method for obtaining a set of hidden keywords.

[0112] First, obtain the set of positive samples and the set of negative samples corresponding to the search target.

[0113] For example, we need to collect data containing the keyword "telecom" to obtain the raw dataset. Optionally, we can use specific web crawlers to collect data from sources such as search engines, code hosting platforms, and document management platforms. Next, we preprocess the collected data. For example, we can use algorithms such as similarity hashing to compare the similarity between the collected data, filter out duplicates, and obtain a preprocessed dataset. Then, we label each document in the preprocessed dataset to determine the positive and negative samples, resulting in sets of positive and negative samples respectively.

[0114] For example, documents in the preprocessed dataset that contain network topology information (such as words like "telecommunications network topology" or "telecommunications topology map") can be manually classified as positive samples, while documents that do not contain such network topology information can be classified as negative samples.

[0115] Then, the samples in the positive sample set and the negative sample set are processed by word segmentation to obtain the sample word segmentation set.

[0116] In the embodiments of this application, the step of performing word segmentation on the samples in the positive sample set and the negative sample set to obtain a sample word segmentation set includes: performing word segmentation on the samples in the positive sample set and the negative sample set to obtain a number of candidate sample word segments; calculating the inverse text word frequency of each candidate sample word segment based on the positive sample set and the negative sample set; and selecting the candidate sample word segments whose inverse text word frequency is greater than a preset word frequency threshold to form a sample word segmentation set.

[0117] Optionally, existing word segmentation techniques can be used to segment the documents in the positive sample set and the documents in the negative sample set to obtain the words included in each positive sample and each negative sample, which can then be used as candidate sample word segments.

[0118] In order to reduce the amount of computation and reduce word noise in the document, in the embodiments of this application, it is necessary to further filter the candidate sample word segmentation to obtain sample word segmentation.

[0119] In the embodiments of this application, candidate sample word segments can be filtered based on the inverse text word frequency of the candidate sample word segments to obtain sample word segments.

[0120] Taking |D| as an example, the inverse text frequency (tf-idf) of the i-th word can be calculated using the following formula: tf-idf = tf·idf, where, n i，j ∑n is the total number of times the i-th word appears in the document. k，j It is the total number of times all words appear in the document; |D| represents the total number of documents, |{j:{t t∈ d j}| indicates that the word t is included. t The number of documents (i.e., n) k，j (Number of documents ≠ 0). If a word is not in either the positive or negative sample set, the denominator will be zero. Therefore, 1+|{j:{t t∈ d j}| indicates that the word t is included. t The number of documents.

[0121] Using the aforementioned inverse text term frequency (ITF) calculation method, the ITF-IDF of each candidate word segment contained in the positive and negative sample sets can be obtained. A higher ITF-IDF indicates a greater probability that the word set appears in a document. In the embodiments of this application, an ITF feature algorithm is used to calculate the term frequency feature scores of all words, and then all words with scores exceeding a specified threshold are selected as sample words. For example, candidate word segments whose ITF-IDF is greater than or equal to a preset term frequency threshold can be used as sample words, thereby obtaining a sample word set.

[0122] After obtaining the sample word segmentation set, the next step is to use the sample word segmentation set as the set of search keywords, and then use a preset Bayesian model to perform reverse derivation to obtain hidden keywords that are included in the positive sample set but not in the negative sample set, thus forming a hidden keyword set.

[0123] The essence of filtering irrelevant search results is to extract the hidden features of all truly relevant results from all search results, features that are not included in irrelevant results. The relationships between various feature information of search results are as follows:

[0124] W = W H ∪W s ∪W U

[0125] S = S T ∪S F

[0126] Where W represents all feature information, W H W represents hidden related feature information. S W represents the surface feature information. U Let S represent irrelevant feature information, and let S represent the set of all search results. T S represents the set of search results that are of genuine interest. F This represents a set of irrelevant search results.

[0127] During the calculation process, the symbol "W" can be used to represent the feature of sample words in the sample word segmentation set. H "W" represents the characteristics of hidden keywords in a set of hidden keywords, denoted by the symbol "W". S "" represents the features of the sample word segmentation in the positive sample set (which can be understood as the features of the search keywords). It can be understood that the target search result Y is limited by the features W of the hidden keywords. H The features W of sample word segmentation in the positive sample set S .

[0128] In the application process, if feature W is known S and feature W H This allows us to determine the search target. Therefore, in this embodiment, the features W of the hidden keywords are first derived based on sample data. H Features of hidden keywords W H There is a one-to-one correspondence between the hidden keywords and the word vectors. During the computer calculation process, similarity calculation is performed based on the word vectors. Therefore, in this embodiment, the data used for calculation in the model is the word vectors. The method for obtaining the word vectors is existing technology, and this embodiment does not limit the specific implementation method of the word vectors.

[0129] In the embodiments of this application, the hidden keyword set W is calculated by reverse inference statistical method based on the sample word segmentation set W that has been calculated and analyzed. H .

[0130] Optionally, the sample word segmentation set is used as the set of search keywords, and a hidden keyword set is formed by reverse derivation through a preset Bayesian model, which includes the hidden keywords contained in the positive sample set but not in the negative sample set. This includes sub-steps S1 to S4.

[0131] Sub-step S1: Use the sample word segmentation set as the set of search keywords.

[0132] In the following text, the set of search keywords is represented by the symbol "W". S "express.

[0133] Sub-step S2 involves using a preset Bayesian model to reverse-engineer the candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the positive sample set, and using it as the first candidate hidden keyword.

[0134] First, a pre-defined Bayesian model is used to reverse-engineer hidden features that have stronger expressive power for positive samples, i.e., the first candidate hidden keywords. That is, given the set of positive samples Y as the target, the probability of its appearance is limited by W. H and WS Among them, W S Given the known search keywords, the ultimate goal is to find suitable W. H To maximize the probability of Y occurring.

[0135] Optionally, the step of back-deriving the candidate keyword with the highest probability of occurrence under the condition that the search keyword appears in the positive sample set through a preset Bayesian model, and using it as the first candidate hidden keyword, includes: obtaining a first probability of the candidate keyword appearing in the sample set, wherein the sample set includes: the positive sample set and the negative sample set; obtaining a second probability of the search keyword appearing under the condition that the candidate keyword appears in the positive sample set; obtaining a first logarithmic probability corresponding to the candidate keyword based on the sum of the logarithmic values ​​of the first probability and the second probability; and using the candidate keyword that maximizes the first logarithmic probability as the first candidate hidden keyword.

[0136] Alternatively, the Bayesian model can be represented as:

[0137]

[0138]

[0139]

[0140] The ultimate goal of the Bayesian model described above is to maximize the probability value of the objective function by iterating through and finding a suitable set of X.

[0141] In the Bayesian model above, Y represents the set of positive samples, X represents the set of features (i.e., candidate keywords, used to deduce the probability of being part of the hidden keyword set) that maximizes the value obtained from the calculation of the entire formula, and P(W H P(Y, W) represents the first probability of a candidate keyword appearing in the sample set. S |W H P(Y, W) represents the second probability of the search keyword appearing given that the candidate keyword appears in the positive sample set. S P(Y, W) represents the probability that the search keyword appears in the set of positive samples. For the same batch of sample data, P(Y, W) represents the probability that the search keyword appears in the set of positive samples. S The value of ) is constant, therefore, in the process of model derivation, it can be treated as a non-zero constant and its influence on the derivation process can be ignored.

[0142] In the Bayesian model described above, the probability of a known probability can be obtained by calculating the probability value of the statistical level by calculating the proportion of each item in the sample set. For example: P(Y, W) SThe proportion of the total number of samples that intersect with the set of samples containing the search keyword can be calculated based on the number of positive samples.

[0143] Since the probability value is between 0 and 1, the product of numbers less than 1 will get closer and closer to 0, resulting in a loss of precision in the computer. Therefore, in the embodiments of this application, the logarithm of the calculated probability value is taken, which can maintain the monotonicity of the function and ensure the accuracy of the calculation.

[0144] The value of the candidate keyword X can be determined using a dynamic programming algorithm.

[0145] By continuously trying various values ​​of candidate keyword X until the objective function probability value is maximized, candidate keyword X is taken as the hidden keyword corresponding to the positive sample, i.e., the first candidate hidden keyword.

[0146] Sub-step S3 involves using the preset Bayesian model to reverse-engineer the candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the negative sample set, and using it as the second candidate hidden keyword.

[0147] Similarly, using the known set of negative samples as the target, the method in sub-step S2 is adopted to determine the second candidate hidden keyword.

[0148] Optionally, the candidate keyword with the highest probability of appearing in the negative sample set under the condition that the search keyword appears in the negative sample set is derived by back-deriving the preset Bayesian model, and used as the second candidate hidden keyword, includes: obtaining the third probability of the candidate keyword appearing in the sample set, wherein the sample set includes: the positive sample set and the negative sample set; obtaining the fourth probability of the search keyword appearing under the condition that the candidate keyword appears in the negative sample set; obtaining the second logarithmic probability corresponding to the candidate keyword according to the sum of the logarithmic values ​​of the third probability and the fourth probability; and using the candidate keyword that maximizes the second logarithmic probability as the second candidate hidden keyword.

[0149] The method for obtaining candidate keywords is described above.

[0150] Each known probability quantity can be used to calculate the probability value of its statistical level by finding the proportion of each quantity in the sample set. For example: P(Y, W) S The proportion of the total number of samples that intersect with the set of samples containing the search keywords can be calculated based on the number of such samples.

[0151] Sub-step S4: The first candidate hidden keywords located outside the intersection of the first candidate hidden keywords and the second candidate hidden keywords are taken as hidden keywords to form a set of hidden keywords.

[0152] After obtaining the first candidate hidden keywords corresponding to the positive samples (e.g., denoted as set S), T ), and the second candidate hidden keywords corresponding to the negative samples (e.g., denoted as set S). F After that, the intersection of the first candidate hidden keyword and the second candidate hidden keyword is further determined, that is, the intersection S' = S T ∩S F The candidate hidden keywords in the intersection S' can represent both positive and negative samples, and therefore do not distinguish between search results. Thus, these candidate hidden keywords need to be removed from the first candidate hidden keywords corresponding to positive samples.

[0153] Then, the first candidate hidden keywords remaining after removing the candidate hidden keywords in the intersection S' from the first candidate hidden keywords are used as hidden keywords to express positive samples (i.e., search targets), forming a set of hidden keywords.

[0154] This completes the derivation of the hidden keyword set. The output hidden keyword set will be used to filter the search results.

[0155] To facilitate readers' understanding of this application, the implementation process of the search result filtering method disclosed in the embodiments of this application will be further explained below with reference to a search application scenario of a certain topic shown in Figure 4.

[0156] After a user obtains search results through various search engines or data sources, the search result filtering method disclosed in this application is executed to retrieve the search results. Then, the search results undergo Chinese word segmentation, data cleaning, feature extraction using a vector space model, and inverse text word frequency feature weight calculation to obtain a vector-form sample word segmentation set. Using this sample word segmentation set as input to a back-derived statistical model, a probability graph is used for classification to obtain hidden features (i.e., a set of hidden keywords) related to the data the user is interested in. Finally, based on the set of hidden keywords, further classification is performed to obtain the information the user is interested in, and this information is output to the user, thereby filtering out irrelevant content. In subsequent search application scenarios targeting this topic, the previously determined set of hidden keywords can be directly applied to filter the current search results.

[0157] The search result filtering method disclosed in this application involves: obtaining search results to be filtered; performing word segmentation on the search results to obtain a set of words to be matched; obtaining a set of hidden keywords associated with the search target that matches the search results to be filtered; and filtering the search results to be filtered based on the set of hidden keywords and the set of words to be matched to obtain the search results that match the search target. By using a pre-derived set of hidden keywords to automatically further filter the search results, this method fully utilizes the hidden data features that the search target is interested in, thereby improving not only the filtering efficiency of search results but also the accuracy of the filtering.

[0158] The search result filtering method disclosed in this application only requires manual labeling of positive and negative data samples when deriving the hidden keyword set. The hidden keyword set is trained end-to-end using the self-learning ability of statistical machine learning, which greatly reduces the manual cost of manual screening.

[0159] By comparing the keyword matching method with the search result filtering method disclosed in the embodiments of this application, the accuracy comparison results under different data orders are obtained, as shown in the table below.

[0160] Order of magnitude Keyword matching methods This method 2000 62.38％ 91.07％ 5000 63.48％ 90.51％ 10000 65.58％ 93.02％ 15000 61.43％ 95.69％ 20000 60.29％ 95.02％ 35000 60.74％ 92.04％ 50000 63.42％ 92.24％

[0161] As can be seen from the accuracy comparison results in the table above, the search result filtering method disclosed in the embodiments of this application has a significant improvement in the accuracy of search result filtering.

[0162] Furthermore, in the process of reverse-engineering the hidden keyword set (i.e., the hidden features of the data), an inverse text term frequency feature extraction algorithm is used to quickly analyze and detect key feature information points in the data. Moreover, by employing a reverse-engineering statistical method to derive the hidden keyword set, and combining the characteristics of inverse text term frequency features, irrelevant search results can be effectively filtered out with high accuracy. Even with limited data, inference analysis can be performed, and the inference results become increasingly accurate as the amount of data increases, making it applicable to a wider range of scenarios.

[0163] Furthermore, as the dataset changes (e.g., the amount of data increases over time, or the search target is adjusted), the derivation process of the hidden keyword set can be re-executed to react instantly to filtering needs and automatically upgrade the inference.

[0164] The search result filtering method disclosed in this application can not only be applied to search applications, but also proactively and preemptively detect unauthorized uploads on code-sharing platforms, technical forums, WeChat official accounts, and document repositories through proactive and preemptive IT technologies. This reduces security risks caused by information leakage and builds the capability for automatic monitoring and identification of information leakage.

[0165] Accordingly, embodiments of this application also disclose a search result filtering device, such as... Figure 5 As shown, the device includes:

[0166] The module 510 for obtaining search results to be filtered is used to obtain search results to be filtered.

[0167] The module 520 for obtaining the word segmentation set to be matched is used to perform word segmentation processing on the search results to be filtered and obtain the word segmentation set to be matched.

[0168] The hidden keyword set acquisition module 530 is used to acquire the set of hidden keywords associated with the search target that matches the search results to be filtered;

[0169] The search result filtering module 540 is used to filter the search results to be filtered based on the hidden keyword set and the word segmentation set to be matched, so as to obtain the search results that match the search target.

[0170] In some embodiments of this application, such as Figure 6 As shown, the device further includes a hidden keyword set derivation module 500, which is used to pre-derive the hidden keyword set using the following method:

[0171] Obtain the set of positive samples and the set of negative samples corresponding to the search target;

[0172] The samples in the positive sample set and the negative sample set are processed by word segmentation to obtain a sample word segmentation set;

[0173] Using the sample word segmentation set as the set of search keywords, a hidden keyword set is obtained by reverse derivation through a preset Bayesian model, which is included in the positive sample set but not in the negative sample set.

[0174] Optionally, the set of sample word segments is used as the set of search keywords. A preset Bayesian model is used for reverse derivation to obtain hidden keywords included in the positive sample set but not in the negative sample set, forming a hidden keyword set, including:

[0175] The sample word segmentation set is used as the set of search keywords;

[0176] The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the positive sample set is derived by back-inferring the preset Bayesian model, and is used as the first candidate hidden keyword.

[0177] The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the negative sample set is derived by reverse deduction using the preset Bayesian model, and is used as the second candidate hidden keyword.

[0178] The first candidate hidden keywords located outside the intersection of the first candidate hidden keywords and the second candidate hidden keywords are used as hidden keywords to form a set of hidden keywords.

[0179] Optionally, the step of back-deriving the candidate keyword with the highest probability of appearing in the positive sample set under the condition that the search keyword appears in the positive sample set through a preset Bayesian model, and using it as the first candidate hidden keyword, includes:

[0180] Obtain the first probability of a candidate keyword appearing in a sample set, wherein the sample set includes: the positive sample set and the negative sample set;

[0181] Given that the candidate keyword appears in the positive sample set, obtain a second probability of the search keyword appearing;

[0182] The first logarithmic probability corresponding to the candidate keyword is obtained by summing the logarithmic values ​​of the first probability and the second probability.

[0183] The candidate keyword that maximizes the first logarithmic probability is selected as the first candidate hidden keyword.

[0184] Optionally, the step of back-deriving the candidate keyword with the highest probability of appearing in the negative sample set under the condition that the search keyword appears in the preset Bayesian model, as the second candidate hidden keyword, includes:

[0185] Obtain the third probability of a candidate keyword appearing in a sample set, wherein the sample set includes: the positive sample set and the negative sample set;

[0186] Given that the candidate keyword appears in the negative sample set, obtain the fourth probability of the search keyword appearing;

[0187] The second logarithmic probability corresponding to the candidate keyword is obtained by summing the logarithmic values ​​of the third probability and the fourth probability.

[0188] The candidate keyword that maximizes the second logarithmic probability will be selected as the second candidate hidden keyword.

[0189] Optionally, the step of performing word segmentation on the samples in the positive sample set and the negative sample set to obtain a sample word segmentation set includes:

[0190] The samples in the positive sample set and the negative sample set are segmented into words to obtain several candidate sample word segments;

[0191] Based on the positive sample set and the negative sample set, the inverse text word frequency of each candidate sample word segment is calculated;

[0192] Candidate samples with word frequencies greater than a preset word frequency threshold are selected for word segmentation to form a sample word segmentation set.

[0193] Optionally, the search result filtering module 540 is further configured to:

[0194] Obtain the intersection of the hidden keyword set and the word segmentation set to be matched, and use it as the intersection keyword;

[0195] Based on the proportion of the intersection keywords in the set of hidden keywords, obtain the predicted matching degree between the search results to be filtered and the search target;

[0196] The search results to be filtered are filtered based on the predicted matching degree to obtain the search results that match the search target.

[0197] The search result filtering device disclosed in this application is used to implement the search result filtering method described in this application. The specific implementation methods of each module of the device will not be repeated here, but can be found in the specific implementation methods of the corresponding steps in the method embodiments.

[0198] The search result filtering device disclosed in this application obtains search results to be filtered; performs word segmentation on the search results to be filtered to obtain a set of word segments to be matched; obtains a set of hidden keywords associated with the search target that matches the search results to be filtered; and filters the search results to be filtered based on the set of hidden keywords and the set of word segments to be matched to obtain the search results that match the search target. By using the pre-derived set of hidden keywords to automatically further filter the search results, the device fully utilizes the data hiding features that the search target is interested in, thereby improving not only the filtering efficiency of the search results but also the accuracy of the filtering of the search results.

[0199] The search result filtering method disclosed in this application only requires manual labeling of positive and negative data samples when deriving the hidden keyword set. The hidden keyword set is trained end-to-end using the self-learning ability of statistical machine learning, which greatly reduces the manual cost of manual screening.

[0200] Furthermore, in the process of reverse-engineering the hidden keyword set (i.e., the hidden features of the data), an inverse text term frequency feature extraction algorithm is used to quickly analyze and detect key feature information points in the data. Moreover, by employing a reverse-engineering statistical method to derive the hidden keyword set, and combining the characteristics of inverse text term frequency features, irrelevant search results can be effectively filtered out with high accuracy. Even with limited data, inference analysis can be performed, and the inference results become increasingly accurate as the amount of data increases, making it applicable to a wider range of scenarios.

[0201] Furthermore, in the event of changes in the dataset (such as an increase in data volume over time or an adjustment in the search target), the derivation process of the hidden keyword set can be re-executed to react instantly to filtering needs and automatically upgrade the inference.

[0202] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus embodiments, since they are fundamentally similar to the method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0203] The above provides a detailed description of a search result filtering method and apparatus provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method of this application and its core idea. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the idea of ​​this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0205] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such a program implementing this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0206] For example, Figure 7 An electronic device is shown that can implement the methods according to this application. The electronic device may be a PC, mobile terminal, personal digital assistant, tablet computer, etc. The electronic device conventionally includes a processor 710 and a memory 720, and program code 730 stored in the memory 720 and executable on the processor 710. When the processor 710 executes the program code 730, it implements the methods described in the above embodiments. The memory 720 may be a computer program product or a computer-readable medium. The memory 720 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 720 has a storage space 7201 for the program code 730 of a computer program for performing any of the method steps described above. For example, the storage space 7201 for the program code 730 may include various computer programs for implementing the various steps in the above methods. The program code 730 is computer-readable code. These computer programs can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The computer program includes computer-readable code that, when executed on an electronic device, causes the electronic device to perform the method according to the above embodiments.

[0207] This application also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the search result filtering method as described in this application.

[0208] Such a computer program product can be a computer-readable storage medium, which can have the same characteristics as... Figure 7The memory 720 in the illustrated electronic device is similarly arranged with storage segments, storage spaces, etc. Program code can be stored, for example, in a compressed form on the computer-readable storage medium. The computer-readable storage medium is typically as shown in the reference... Figure 8 The portable or fixed storage unit is described above. Typically, the storage unit includes computer-readable code 730', which is code read by a processor and, when executed by the processor, implements the various steps of the method described above.

[0209] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this application. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0210] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0211] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A search result filtering method, characterized in that, include: Get the search results to be filtered; The search results to be filtered are segmented to obtain a set of segmented words to be matched; Obtain the set of hidden keywords associated with the search target that matches the search results to be filtered; Obtain the intersection of the hidden keyword set and the word segmentation set to be matched, and use it as the intersection keyword; Based on the proportion of the intersection keywords in the set of hidden keywords, obtain the predicted matching degree between the search results to be filtered and the search target; The search results to be filtered are filtered based on the predicted matching degree to obtain the search results that match the search target; The set of hidden keywords is derived in advance using the following method: Obtain the set of positive samples and the set of negative samples corresponding to the search target; The samples in the positive sample set and the negative sample set are processed by word segmentation to obtain a sample word segmentation set; The sample word segmentation set is used as the set of search keywords; The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the positive sample set is derived by back-inferring the preset Bayesian model, and is used as the first candidate hidden keyword. The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the negative sample set is derived by reverse deduction using the preset Bayesian model, and is used as the second candidate hidden keyword. The first candidate hidden keywords located outside the intersection of the first candidate hidden keywords and the second candidate hidden keywords are used as hidden keywords to form a set of hidden keywords.

2. The method according to claim 1, characterized in that, The process of back-deriving the candidate keyword with the highest probability of appearing in the positive sample set under the condition that the search keyword appears, through a preset Bayesian model, and using it as the first candidate hidden keyword, includes: Obtain the first probability of a candidate keyword appearing in a sample set, wherein the sample set includes: the positive sample set and the negative sample set; Given that the candidate keyword appears in the positive sample set, obtain a second probability of the search keyword appearing; The first logarithmic probability corresponding to the candidate keyword is obtained by summing the logarithmic values ​​of the first probability and the second probability. The candidate keyword that maximizes the first logarithmic probability is selected as the first candidate hidden keyword.

3. The method according to claim 1, characterized in that, The step of back-deriving the candidate keyword with the highest probability of appearing in the negative sample set under the condition that the search keyword appears in the preset Bayesian model, and using it as the second candidate hidden keyword, includes: Obtain the third probability of a candidate keyword appearing in a sample set, wherein the sample set includes: the positive sample set and the negative sample set; Given that the candidate keyword appears in the negative sample set, obtain the fourth probability of the search keyword appearing; The second logarithmic probability corresponding to the candidate keyword is obtained by summing the logarithmic values ​​of the third probability and the fourth probability. The candidate keyword that maximizes the second logarithmic probability will be selected as the second candidate hidden keyword.

4. The method according to claim 1, characterized in that, The step of performing word segmentation on samples in the positive sample set and the negative sample set to obtain a sample word segmentation set includes: The samples in the positive sample set and the negative sample set are segmented into words to obtain several candidate sample word segments; Based on the positive sample set and the negative sample set, the inverse text word frequency of each candidate sample word segment is calculated; Candidate samples with word frequencies greater than a preset word frequency threshold are selected for word segmentation to form a sample word segmentation set.

5. A search result filtering device, characterized in that, The device includes: The module for obtaining search results to be filtered is used to obtain search results to be filtered. The module for obtaining the word segmentation set to be matched is used to perform word segmentation processing on the search results to be filtered and obtain the word segmentation set to be matched. The hidden keyword set acquisition module is used to acquire the set of hidden keywords associated with the search target that matches the search results to be filtered; The search result filtering module is used to obtain the intersection of the hidden keyword set and the word segmentation set to be matched, as the intersection keyword; obtain the matching degree prediction value between the search result to be filtered and the search target based on the proportion of the intersection keyword in the hidden keyword set; filter the search result to be filtered based on the matching degree prediction value to obtain the search result that matches the search target. The hidden keyword set derivation module is used to pre-derive the hidden keyword set using the following methods: Obtain the set of positive samples and the set of negative samples corresponding to the search target; The samples in the positive sample set and the negative sample set are processed by word segmentation to obtain a sample word segmentation set; The sample word segmentation set is used as the set of search keywords; The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the positive sample set is derived by back-inferring the preset Bayesian model, and is used as the first candidate hidden keyword. The candidate keyword with the highest probability of appearing under the condition that the search keyword appears in the negative sample set is derived by reverse deduction using the preset Bayesian model, and is used as the second candidate hidden keyword. The first candidate hidden keywords located outside the intersection of the first candidate hidden keywords and the second candidate hidden keywords are used as hidden keywords to form a set of hidden keywords.

6. An electronic device, comprising a memory, a processor, and program code stored in the memory and executable on the processor, characterized in that, When the processor executes the program code, it implements the search result filtering method according to any one of claims 1 to 4.

7. A computer-readable storage medium having program code stored thereon, characterized in that, When the program code is executed by the processor, it implements the steps of the search result filtering method according to any one of claims 1 to 4.

Images