Refining search requests to content providers

Abstract

A computer-implemented method, computer program product, and computer system for refining an initial search request to a content provider. The computer-implemented method includes receiving an initial search request from a user, transmitting the initial search request to a content provider, receiving from the content provider a set of sample documents and a source identifier for each one of the sample documents, applying a topic model to the set of sample documents to obtain topic representations, presenting the topic representations to the user, receiving from the user a topic relevance score for each one of the topics, and sorting the data sources according to the topic relevance scores to obtain a source relevance score for each one of the data sources. The computer-implemented method further includes determining a refined search request with increased selectivity for documents covering one of the highest-scoring topics based on the source relevance scores.

Description

[Technical field]

[0001] The present invention relates to computer-based retrieval of textual information, and more particularly to automatically generating suggestions for refined search requests based on an initial search request. [Background technology]

[0002] Online media analysis tools retrieve documents about specific data sources from content providers such as Twitter, Facebook, SocialGist, etc. Content provider APIs typically support at least two access mechanisms that can often be combined: keyword-based retrieval, where the user specifies a set of keywords, optionally combined with AND, OR, or NOT, and the data provider returns documents that contain this content, such as websites, text documents, forum posts, blog entries, etc.; site-based retrieval, where the user specifies a list of data sources, such as websites, website parts, channels, feeds, etc., and the content provider returns documents specifically from these sources.

[0003] In response to entering a search request, the user is typically presented with a sample of documents. Before performing a full analysis, the user can continue searching for a set of more relevant documents that better support the objectives of the analysis. Especially when searching for user-generated content, which can be found in abundance in social media data sources, keyword searches are prone to ambiguous search results. For example, a search on "F-50" may return content about sports cars, soccer shoes, turboprop aircraft, and coffee machines. To this end, the user can refine the query by adding keywords or sites, or both. Summary of the Invention

[0004] In one aspect, a computer-implemented method for refining an initial search request to a content provider is disclosed. The computer-implemented method includes receiving an initial search request from a user and sending the initial search request to the content provider. The computer-implemented method further includes receiving from the content provider a set of sample documents and a source identifier for each one of the sample documents, where the source identifier identifies a respective one of the data sources associated with each one of the sample documents. The computer-implemented method further includes applying a topic model to the set of sample documents to obtain a topic representation, where the topic representation is a description of a topic covered by each one of the sample documents. The computer-implemented method further includes presenting the topic representation to a user and receiving a topic relevance score from the user for each one of the topics. The computer-implemented method further includes sorting the data sources according to the topic relevance scores to obtain a source relevance score for each one of the data sources. The computer-implemented method further includes determining a refined search request based on the source relevance scores, the refined search request having increased selectivity for documents covering the topics having the highest scores among the topics.

[0005] In another aspect, a computer program product for refining an initial search request to a content provider is disclosed. The computer program product comprises a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by one or more processors. The program instructions are executable to receive an initial search request from a user. The program instructions are further executable to send the initial search request to a content provider. The program instructions are further executable to receive from the content provider a set of sample documents and a source identifier for each one of the sample documents, where the source identifier identifies a respective one of the data sources associated with each one of the sample documents. The program instructions are further executable to apply a topic model to the set of sample documents to obtain a topic representation, where the topic representation is a description of a topic covered by each one of the sample documents. The program instructions are further executable to present the topic representation to a user. The program instructions are further executable to receive from the user a topic relevance score for each one of the topics. The program instructions are further executable to sort the data sources according to the topic relevance scores to obtain a source relevance score for each one of the data sources. The program instructions are further executable to determine a refined search request based on the source relevance scores, with increased selectivity for documents covering the topics having the highest scores among the topics.

[0006] In yet another aspect, a computer system for refining search requests to content providers is disclosed, the computer system comprising one or more processors, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more computer-readable tangible storage devices for execution by at least one of the one or more processors. The program instructions are executable to receive an initial search request from a user, send the initial search request to a content provider, receive from the content provider a set of sample documents and a source identifier for each one of the sample documents, the source identifier identifying a respective one of the data sources associated with each one of the sample documents, apply a topic model to the set of sample documents to obtain a topic representation that is a description of the topic covered by each one of the sample documents, present the topic representation to the user, receive from the user a topic relevance score for each one of the topics, sort the data sources according to the topic relevance score to obtain a source relevance score for each one of the data sources, and determine a refined search request based on the source relevance scores, with increased selectivity for documents covering the topics with the highest scores among the topics. [Brief description of the drawings]

[0007] [Figure 1] 1 illustrates an exemplary computing environment adapted to obtain sample documents in response to a search request, according to one embodiment of the present invention. [Diagram 2] 1 illustrates an exemplary topic representation according to one embodiment of the present invention. [Diagram 3] FIG. 2 illustrates the handling of relevance scores according to one embodiment of the present invention. [Figure 4]2 is a flow diagram illustrating method steps for refining an initial search request to a content provider according to one embodiment of the present invention. [Diagram 5] 1 illustrates components of an exemplary computing device, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Due to the ambiguous nature of text-based search queries, users typically iteratively refine their search queries to be sufficiently sure that all relevant content is retrieved, but without spending money to retrieve irrelevant content (content provider APIs typically employ a pay-per-volume model). This may involve coming up with logical terms that include relevant keywords and data sources, exclude irrelevant keywords and data sources, and / or take into account alternative keywords or data sources or both. Doing this in a productive manner may require expert knowledge. Thus, a simplified approach for iterative search queries is desirable.

[0009] A method for refining an initial search request to a content provider typically includes the steps of receiving an initial search request from a user and transmitting the initial search request to a content provider. The initial search request may be received from the user in a direct manner (e.g., using an input device) or indirectly (e.g., transferred over a network from a computing device to a computing device performing the method). The content provider will not necessarily be the same as one of the data sources.

[0010] In response to the transmission, the computing device receives a set of sample documents from the content provider. The set of sample documents is accompanied by a source identifier (e.g., a URL) that assigns each sample document to the data source from which it originates. Within the scope of this disclosure, a document shall be understood as any computer-readable data object that conveys human-readable information to be embodied by an output device. Without limitation, such human-readable information may be text, images, audio, video, or a combination thereof.

[0011] A data source, sometimes referred to herein as a "source" for simplicity, may be any computing device connected to a network and accessible by calling and routing a source identifier using the respective network. However, a data source may also be any computing device that is not connected to a network and can be accessed by, for example, using a local interface of the respective computing device, where the local interface, any other component of the respective computing device, or a variable stored by the respective computing device is identified by a source identifier. Furthermore, a data source may be an entity that is not a computing device, including, but not limited to, an analog information storage, where the source identifier may indicate, for example, a non-digital or non-electronic source, such as a book, from which a document that can be processed by a computing device implementing the method is obtained by a digitizing device, such as a scanner, used to reproduce, by digitization, the information contained in the document in the non-digital or non-electronic source, or a location, such as an archive or library, where the respective non-computing source can be found, or both.

[0012] Without limitation, the data sources and documents that may be obtained from them, respectively, may include forums, where the documents may be a particular forum, a sub-forum within a larger forum site, or a single page or a portion of multiple pages from a respective forum or sub-forum; web blogs or blogs, where the documents may be a particular blog, blog portion, or a single page or a portion of multiple pages from a respective blog or blog portion; microblogging services, where the documents may be a particular microblog associated with a particular user account, or a page containing an excerpt thereof; audio or video download or streaming services, where the documents may be a page containing a particular audio or video channel, or a portion thereof; review providers, where the documents may be reviews or discussions that are optional for the provider or that are tied to a particular performance category or performance; virtual social networks, where the documents may be a particular public or private user page, fan page, etc.; or news providers, where the documents may be a page representing a particular news outlet, channel, or news item on a news outlet, or a portion thereof.

[0013] The amount of sample documents received in response to sending an initial search request can be preset by the content provider or specified by the user. In one example, the content provider has a rule that limits the set of sample documents to 1000 hits if the total number of search results (e.g., 25 billion) for the initial search request exceeds 1000 documents. Generally, algorithms incorporating this method augment the set of sample documents by sending initial search requests to multiple content providers, thus receiving multiple portions of sample documents that are added to the total set of sample documents.

[0014] The computing device applies the topic model to the set of sample documents to obtain a topic representation. A topic representation is understood herein as a data structure in which keywords from the sample documents are organized in a manner (e.g., topologically, sequentially, tabular, etc.) that highlights similarities or differences or both between the keywords. Such keywords may be obtained by statistical analysis provided in the topic model, and thus the topic representation may further include weights assigned to the keywords such that each keyword may be used (e.g., processed or displayed) according to the weight assigned to it. A group of keywords that are represented as similar may be understood as a topic described or covered by the sample documents, and keywords (including identical keywords) that are represented in different contexts may be understood as belonging to different topics. Non-exhaustive examples of topic representations include topic maps, tag or word clouds, tree structures, lists or tables of topics, keywords, etc., or combinations thereof, etc.

[0015] The topic representations are presented to a user, for example, using an output device of a computing device. In response to the presentation, the user inputs one or more topic relevance scores for the topics included in the topic representations. A score, as used herein, can be a continuous scale, such as a numeric or alphanumeric value, or a level or category within a set of discrete levels or categories, or both.

[0016] A computing device implementing the method classifies data sources according to the topic relevance scores obtained from the user. As a result, one or more of the data sources are assigned a source relevance score inferred from the topic relevance score value(s) entered by the user. It should be noted that there are no general limitations on the methods that may be used to infer source relevance score(s) from topic relevance score(s), and that the examples described herein are merely illustrative that may be useful in certain application scenarios, while those skilled in the art will know or can conceive of various other rules of inference that may prove more useful or suitable for other applications.

[0017] Topic relevance scores and source relevance scores may be different scales that need to be appropriately mapped. It seems preferable to use the same scale or categories for topics and data sources so that mapping may not be necessary. However, if topic relevance scores are to be associated with source relevance scores with weightings that are not 1:1, it may be necessary to define a mapping for the same relevance scale as well. In the general case, it would be possible to define a mapping that, for example, assigns a given number of topic relevance levels to a larger number of source relevance levels, maps a given interval of numeric topic relevance scores to a shifted numeric interval of source relevance scores that is larger and / or smaller, converts category-based topic relevance scores to numeric source relevance scores, or converts numeric topic relevance scores to category-based source relevance scores. Topic and source relevance scores can be designed with the understanding that relevance can be expressed in a one-dimensional scale that allows for relative scaling of relevance (e.g., "Document A / Source B is more / less relevant than Document X / Source Y") and, preferably, can also be expressed in an absolute description of relevance (e.g., "Most / Least Relevant Documents / Data Sources", "Top 5 / Bottom 5 Relevant Documents / Data Sources", "Most / Least Relevant 10% Sample Documents / Data Sources", etc.).

[0018] Based on the source relevance scores, a computing device implementing the method determines a refined search request with increased selectivity for documents covering the highest scoring topics of the topics. The refined search request can be suggested to a user to use the refined search request without starting a new search. An algorithm implementing the method can determine one or more refined search requests that can be suggested to a user for selecting the most appropriate suggestion.

[0019] Due to the unidimensional nature of the relevance score(s) above, one or more of the data sources can be identified as having the highest score, for example using a predefined threshold criterion, where the highest score can be understood as a higher score compared to further data sources, or compared to a predefined threshold for distinguishing high relevance sources from low relevance sources, or both. The increase in selectivity can be assessed by sending a refined search request to the content provider and receiving a second set of sample documents in response, measuring the number of sample documents from the second set covering one or more of the topics having the highest score values ​​previously identified, and comparing the number with a corresponding number of sample documents from the first set covering one or more of the topics having the highest score, where the number determined from the second set should be greater than the number determined from the first set.

[0020] Without limitation, increased selectivity may be achieved by adding one or more additional criteria to the initial search request that can be reasonably expected to limit the topics covered by the potential second set of sample documents to those with the highest scores previously identified and / or limit the data sources included in the potential second search to those with the highest scores in terms of the previously determined source relevance score(s). Without limitation, this may mean including one or more keywords of the highest scoring topic or one or more data sources of the highest scoring source or both in the refined search request as a limiting or focusing criterion and / or including one or more keywords of the correspondingly determined lowest scoring topic or one or more data sources of the correspondingly determined lowest scoring source or both in the refined search request as an exclusion criterion.

[0021] A refined search request may include keywords and / or data sources to be included or excluded from prospective searches based on the refined search request. Such keywords or data sources may be explicitly selected by the user from a presentation of topics (which may require the inclusion of a source identifier in the presentation) or may be automatically derived from topic relevance score(s) and / or source relevance score(s) assigned by the user, for example, by including the most frequent keyword(s) from each topic marked as relevant or highly relevant by the user, excluding the most frequent keyword(s) from each topic marked as not relevant or less relevant by the user, by including data source(s) marked as relevant or highly relevant by the user, and / or by excluding data source(s) marked as not relevant or less relevant by the user.

[0022] After determining the refined search request, a user or an algorithm incorporating the method can resume the method by using the refined search request of a completed iteration of the method as the initial search request for the next iteration.

[0023] The method may further include, prior to sending any search request described herein, querying a content provider regarding the total number of hits that would be found if the search request were submitted, and, accordingly, modifying the search request prior to sending by limiting the number of documents retrieved to less than the total number of hits.

[0024] Furthermore, it may be advantageous to define rules for excluding a data source from applying the topic model, presenting the topic representation, classifying or determining the refined search request, or both, if the number of sample documents retrieved from the data source is less than a predefined minimum number of documents. This can ensure that the classification of data sources with high or low relevance is statistically significant. For example, if only one sample document is received from a particular data source, the data source may be classified as having a high source relevance score (the proportion of relevant documents in the sample is 100%) by maximizing its topic relevance, while other documents from the same data source that match the initial search request but are not included in the set of sample documents may have a non-maximal topic relevance score, and the classification may assign the same data source a source relevance score representing low relevance if it is based on the highly relevant sample documents and also on less relevant documents. In this example, erroneous relevance classifications could be avoided by defining a minimum number of documents per source (e.g., three documents) that must be included in the set of sample documents for the data source to be considered important enough to yield a reliable classification. Similarly, it may be undesirable to exclude a data source due to an erroneous low relevance classification.

[0025] An embodiment of the method may have the advantage that, when a second topic representation is determined based on a second set of sample documents to be retrieved from a content provider in response to submitting a refined search request for another search, the refined search request obtained during one iteration of the method is highly selective for documents covering topics to which the user will assign high relevance. In comparison, manual construction of a refined search request may require multiple iterations of trial and error until the user can identify the appropriate combination of keywords and / or data sources to include and / or exclude from the search. Thus, an embodiment of the invention may reduce the number of search iterations required to obtain search results with desired relevance. This may reduce the workload of the content provider to search, accumulate, and deliver hits (sample documents and / or larger packages of documents) to the requesting user, increase the efficiency of the requesting user, and reduce the skill level of the user required to construct a complex search request that also obtains documents with desired relevance and excludes irrelevant search results, but does not accidentally exclude relevant search results.

[0026] According to one embodiment, the refined search request includes a conjunction of the initial search request and the source criteria, where the source criteria reduces the number of data sources covered by the refined search request compared to the number of data sources covered by the initial search request. In other words, the initial search request is extended by a statement, i.e., an "AND" relationship, connecting the initial search request to the source criteria, which results in the selection of the data sources reached by the initial search request in preference to other data sources that are also reached by the initial search request but are not selected for future searches based on the refined search request. This allows future searches to be focused on data sources of higher relevance and to exclude data sources of lower relevance. The second set of sample documents that may be received during the future search may thus include a larger portion of sample documents that have a higher probability of covering high relevance topics than sample documents from the first set of sample documents (obtained in response to the initial search request) that are different from the sample documents received from sources excluded in the refined search request.

[0027] According to one embodiment, the source criteria includes either a focus statement that selects one or more of the data sources based on the source relevance score, or an exclusion statement that deselects one or more of the data sources based on the source relevance score. The focus statement allows for the explicit inclusion of documents from data sources with a high source relevance score, while the exclusion statement allows for the explicit exclusion of documents from data sources with a low source relevance score. This can form a relatively easy way to appropriately reduce the number of covered data sources compared to source criteria that implicitly or indirectly reduce the number of covered data sources, and can allow for the targeting of data sources to be excluded or included, such that less relevant sources are not inadvertently covered by the indirect source criteria.

[0028] In one embodiment, the classification includes selecting a source relevance score from a predetermined set of discrete source relevance levels, and the method further includes comparing, for each source relevance level used to determine the source criteria, determining a total number of hits found for the initial search request in each data source having the respective source relevance level, and identifying N data sources having the greatest total number of hits from the data sources having the respective source relevance levels, where N is a predetermined upper limit, and determining the refined search request includes, for each source relevance level used to determine the source criteria, populating the source criteria with the N identified data sources.

[0029] The upper limit N allows to take into account that many content providers have limitations on the length of a query, i.e., they only allow up to a maximum number of statements that can be included in a single search query. By choosing N appropriately, refined search queries can be designed to ensure that a sufficient number of restrictions can be imposed on the data sources to be covered or excluded, and at the same time ensure that sufficient overhead is maintained without relying on data source specifications to reserve space for statements defining keywords to be included or excluded. For example, if a search provider allows up to 40 statements per query, one can choose N=30 to explicitly include or exclude up to 30 data sources in a refined search request, while still reserving 10 statements for keywords.

[0030] In one embodiment, the classification includes selecting a source relevance score from a predetermined set of discrete source relevance levels, and determining the refined search request may include selecting one or more of the source relevance levels to improve selectivity, or a topic relevance score is selected from a predetermined set of discrete topic relevance levels, or both.

[0031] This can result in discretization of the source relevance scores or topic relevance scores or both. Discretization of topic relevance scores can simplify a user's decision regarding the assignment of a particular relevance score to a topic, since it may be the case that the user must assign a numerical score that may appear more abstract than deciding among a number of discrete options. On the other hand, a discretized source relevance score may be beneficial by avoiding the need to predefine a threshold for determining when a data score has high relevance or low relevance, respectively. Such a threshold may ignore the variability in the statistical distribution of numerical relevance between search requests that yield a high percentage of relevant sample documents and other search requests that yield a low percentage. In a non-exhaustive example, the set of topic relevance levels includes a high relevance level that allows a user to identify topics that should be included in subsequent search queries and a low relevance level that allows a user to identify topics that should be excluded from subsequent search queries.

[0032] According to one embodiment, the source relevance scores and the topic relevance scores are selected from a predetermined set of discrete common relevance levels, and determining the refined search request includes selecting one or more of the common relevance levels to improve selectivity. This may simplify the mapping of topic relevance levels to source relevance levels because it may not be necessary to make assumptions about a numerical or quantitative correspondence between topic relevance levels and separately defined source relevance levels.

[0033] According to one embodiment, the set of common relevance levels includes a high relevance level, the classification includes assigning a given data source to the high relevance level if a relative frequency of sample documents associated with the given data source and covering a topic assigned to the high relevance level equals or exceeds a predefined high relevance threshold, and determining the refined search request includes limiting the refined search request to one or more of the data sources assigned to the high relevance level compared to the initial search request when the high relevance level is used to determine the refined search request. According to one embodiment, the high relevance threshold is 100 percent.

[0034] The high relevance level may be considered to represent the highest relevant documents and data sources, respectively, compared to any other relevance levels in the set of common relevance levels. Specifically, but not necessarily, the high relevance threshold may be set to 100%, in which case data sources providing only documents covering (highly) relevant topics may be classified as high relevance. Otherwise, data sources providing documents below a certain percentage of documents covering less relevant topics may also be included in the high relevance level. Such relaxation of the high relevance filtering may increase the total number of documents considered to be highly relevant documents and may result in a greater diversity of subjects and / or document types available for subsequent analysis.

[0035] According to one embodiment, the set of common relevance levels further includes a low relevance level, and the classification includes assigning the given data source to the low relevance level if a relative frequency of sample documents associated with the given data source and covering a topic assigned to the low relevance level is equal to or exceeds a first predetermined low relevance threshold, or if a relative frequency of sample documents associated with the given data source and covering a topic assigned to the high relevance level is equal to or less than a second predetermined low relevance threshold, or both, and the determination of the refined search request includes excluding from the refined search request one or more of the data sources assigned to the low relevance level compared to the initial search request when the low relevance level is used for determining the refined search request. According to one embodiment, the first low relevance threshold is 100 percent and / or the second low relevance threshold is zero percent.

[0036] The low relevance level may be considered to represent documents and data sources, respectively, with the lowest relevance compared to any other relevance level in the set of common relevance levels. In particular, although not necessarily, the first low relevance threshold may be set to 100%, in which case data sources providing only documents covering topics with low relevance may be classified in the low relevance level. Additionally or alternatively, although also not necessarily, the second low relevance threshold may be set to 0%, in which case data sources providing no documents covering topics with high relevance may be classified in the low relevance level. Otherwise, relaxing these conditions to a value below 100% for the first low relevance threshold or above 0% for the second low relevance threshold, respectively, may result in data sources providing documents with less than a certain percentage of documents covering topics with higher relevance, or data sources providing documents with less than a certain percentage of documents covering topics with higher relevance, or both, being included in the low relevance level. Such relaxation of low relevance filtering may increase the total number of documents considered to be low relevance documents and may result in more comprehensive elimination of data sources that return fewer highly relevant documents.

[0037] The set of common relevance levels may further include additional relevance levels, such as intermediate or partial relevance levels, to which topics or data sources, or both, that cannot be assigned to either the high or low relevance levels.

[0038] According to one embodiment, the method further includes obtaining a precision level, and the refined search request is further determined based on the precision level, the precision level being obtained from a predetermined set of discrete precision levels, and the increase in selectivity is based on the precision level. The precision level can reflect the need to create a search query that balances precision (ensuring that computing resources and money are not spent on retrieving irrelevant content) and recall (ensuring that all relevant content is retrieved). Incorporating the precision level allows the determination of the source relevance level for use in determining the refined search request to be tied to user preferences or statistical characteristics of a set of sample documents, or both, thus facilitating the determination of the refined search request. Without limitation, the precision level can be received as an input by a user (e.g., using a slider or radio button field implemented in software) or can be determined by a computing device performing the method by analyzing the received set of sample documents.

[0039] According to one embodiment, the precision level is obtained as an input from the user, which may give the user additional freedom to influence the results of the method according to the precision requirements of the individual case that may be specific to the user's higher level task, including refining the search request, and that may run counter to any statistical findings, such as keyword frequencies that may be found in the set of sample documents received by the computing device performing the method.

[0040] In one embodiment, the set of precision levels includes a high precision level, a balanced precision level, and a high recall level, the source relevance scores are selected from the group consisting of a high relevance level, an intermediate relevance level, and a low relevance level, and the classification includes assigning the given data source to the high relevance level if a relative frequency of sample documents associated with the given data source and covering topics assigned to the high relevance level is equal to or exceeds a predetermined high relevance threshold, the classification further includes assigning the given data source to the low relevance level if a relative frequency of sample documents associated with the given data source and covering topics assigned to the low relevance level is equal to or exceeds a predetermined first low relevance threshold, or if a relative frequency of sample documents associated with the given data source and covering topics assigned to the high relevance level is equal to or less than a predetermined second low relevance threshold, or both, and the classification further includes assigning the given data source to the intermediate relevance level if the given data source cannot be assigned to either the high relevance level or the low relevance level. If the resulting precision level is a high precision level, the refined search request includes a logical conjunction of the initial search request with a focusing statement that selects one or more of the data sources having a high relevance level as a source relevance score. If the resulting precision level is a balanced precision level, the refined search request includes a logical conjunction of the initial search request with a focusing statement that includes a logical conjunction of one or more of the data sources having a high relevance level as a source relevance score and one or more of the data sources having an intermediate relevance level as a source relevance score. If the resulting precision level is a high recall level, the refined search request includes a logical conjunction of the initial search request with a focusing statement that includes a logical negation of one or more of the data sources having a low relevance level as a source relevance score.

[0041] This allows one to construct an advantageous correspondence between a set of precision levels and source relevance scores that can simplify the selection of which source relevance level to use for determining a refined search request. Let K be an initial search request, and S R Let be the set of relevant data sources, i.e., the set of data sources classified as having a high relevance level, and let S P Let be a partial relevance data set, i.e., a set of data sources classified as having an intermediate relevance level, and let S I Let K AND S be the set of irrelevant data sources, i.e., the set of data sources classified as having a low relevance level. Furthermore, the above three cases for determining the refined search request are not necessarily related to the high precision level, i.e., the refined search request by limiting the initial search request to highly relevant data sources. R " and "K AND (S R OR S P )” for determining high recall levels, i.e., refined search requests by excluding irrelevant data sources from the initial search request, and “K AND (NOT S I )" can be written simply as

[0042] It will be seen that the refined search request determined for the high precision level may result in the strictest focus on highly relevant documents, while the balanced precision level may include more relevant documents at the cost of including an additional portion of documents that may be outside the high relevance level, i.e. documents that may cover less relevant topics. Furthermore, it should be noted that both the high precision level and the balanced precision level correspond to a strategy of focus refinement, i.e. the spectrum of data sources is explicitly limited to sources that have a high probability of delivering highly relevant documents, while the high recall level corresponds to a strategy of exclusion refinement, i.e. only the least relevant data sources are explicitly excluded, so that in the case of high recall a diversity of highly relevant and intermediately relevant topics can be expected. Thus, a high recall level may be recommended in cases where it may be expected that the number of highly relevant search results will be low compared to the number of irrelevant documents, maximizing the number of relevant search results by excluding less relevant data sources. On the other hand, a high precision level may be recommended when the number of relevant search results is large compared to the number of irrelevant documents, or when the initial search request is very vague and high selectivity may be desired to filter out all less relevant topics or keywords, or both. Additionally, a balanced precision level may be advantageous as a suggested choice to users as a standard setting that may be useful when neither a high precision level nor a high recall level seems clearly advantageous.

[0043] According to one embodiment, the topic relevance score is selected from a topic relevance group, the topic relevance group including a high relevance level representing the highest relevance compared to all other relevance levels therein, and the method further comprises: selecting a number D of sample documents covering the topic having the high relevance level as the topic relevance score; RThe method includes determining D for the total number of sample documents in the set of sample documents D. R Ratio of D R If / D is less than or equal to a predetermined low precision threshold, the high precision level is set as the precision level. R If / D is greater than the low precision threshold and less than the upper precision threshold, the balanced precision level is set as the precision level. R If A / D is greater than or equal to the precision threshold, the high-recall level is set as the precision level.

[0044] In this way, the precision type can be determined automatically by statistical analysis of the set of sample documents received. To obtain the precision level to use in determining the refined search request, the number of relevant sample documents D R Note that only the total number of sample documents D and the number of sample documents N may be needed. Moreover, it may not be necessary to obtain further information from the content provider to make this determination. In this manner, it may be possible to determine a refined search request with minimal interaction between the user and the content provider and with efficient use of the computing resources of the computing device implementing the method.

[0045] According to one embodiment, the topic relevance score is selected from a topic relevance group, the topic relevance group including a high relevance level representing a highest relevance compared to all other relevance levels therein, and the method further comprises: determining a number D of documents available as search results for a hypothetical search request including a conjunction of the initial search request and a focus statement selecting one or more of the data sources having the high relevance level as a source relevance score; RS from a content provider, the method further comprising: KThis includes querying content providers. K D RS Ratio of D RS / D K If D is less than or equal to the low precision threshold, the high precision level is set as the precision level. RS / D K If D is greater than the low precision threshold and less than the upper precision threshold, the balanced precision level is set as the precision level. RS / D K If the recall level is greater than or equal to the precision threshold, the recall level is set as the precision level.

[0046] This method of determining the level of precision governing the refinement of search requests is based on the total number of hits D shown by the content provider for an initial search request K. K This can be particularly advantageous when D is large compared to the number of sample documents received D, but this is not always the case. In this situation, the set of sample documents may be too small to have statistical relevance, and therefore more accurate decisions can be made by considering the entire statistics of hits rather than just the set of sample documents. However, when it is desirable to deliver high-quality refined search requests by minimizing the number of assumptions to make, it is often advantageous to have a large D and D. K Regardless of the ratio of D RS and D. K It may be advantageous to base the accuracy level decision on

[0047] In one embodiment, the method further includes receiving, in response to the submission, a document relevance score from the user for one or more of the sample documents, where the classification of the data source is further based on the document relevance score, and / or receiving a source relevance score from the user for one or more of the data sources, where a data source having the source relevance score received from the user is excluded from the classification, and / or receiving a keyword relevance score from the user for one or more keywords representative of one or more of the topics, where the classification of the data source is further based on the keyword relevance score.

[0048] Providing the user with the possibility to specify relevance scores for items other than the topic can improve the probability of obtaining a refined search request with high selectivity for relevant documents even when the user is uncertain about which relevance to assign to a particular topic identified by the topic model, or when the identified topics do not seem to give adequate support for the type of relevance predicted by the user. For example, the user can find a particular sample document that is more relevant than all the presented topics and can mark this document as relevant accordingly. Thus, an algorithm implementing the method can obtain the desired increase in selectivity, for example, by focusing the refined search request on data sources from which the sample documents marked as relevant were obtained, or by focusing the refined search request on characteristic keywords of the sample documents, or both. Similarly, the refined search request can be configured to exclude keywords or sources related to irrelevant documents, to include data sources designated by the user as relevant and / or to exclude data sources designated by the user as irrelevant, and / or to include keywords designated by the user as relevant and / or to exclude keywords designated by the user as irrelevant.

[0049] According to one embodiment, the method further includes applying a predetermined default topic relevance score to topics that have not received a topic relevance score and / or applying a predetermined default source relevance score to data sources for which a source relevance score was not determined during classification, which can maximize the number of topics and / or data sources accessible as a basis for determining a refined search request.

[0050] According to one embodiment, the presentation is limited to topics that equal or exceed a predetermined minimum number of hits for the initial search request. This prevents users from marking topics with low statistical significance as relevant or irrelevant because the number of hits for keywords within those topics is too small. Similarly, presentation of statistically insignificant data sources, keywords and / or sample documents is suppressed to prevent users from accidentally marking those data sources, keywords and / or sample documents as relevant or irrelevant.

[0051] Referring now to the drawings, FIG. 1 illustrates an exemplary routine for processing an initial search request K using an exemplary computing environment adapted to retrieve sample documents in response to the initial search request K. A user 101 uses a computing device 10 connected to another computing device (e.g., a server) of a document provider 130 via a computing network 120. The user inputs an initial search request K into the computing device 10, which transmits the initial search request to a content provider via the network 120. Without limitation, the initial search request K may include keywords to be included or excluded in the requested search, or source identifiers (e.g., URLs) of data sources to be included or excluded in the requested search, or both. The initial search request K may further include an identifier that indicates to the content provider 130 that the requested search should be limited to a set of sample documents, where the specifications of the set of sample documents to be returned, such as a maximum number of hits to be included in the set of sample documents, may be predetermined by the content provider 130, or by specifications also included in the initial search request K, or both.

[0052] In response, content provider 130 parses initial search request K and, if initial search request K is valid, performs a search for documents that match initial search request K. Given a specification of the set of sample documents to be returned, document provider 130 aggregates set 132 of sample documents and delivers set 132 over network 120 to computing device 10 of user 101.

[0053] Fig. 2 shows an exemplary topic representation 200 formed by a list of topics 202, 204, 206, 208 and 210, in the non-limiting particular case shown in Fig. 2, where each topic is represented by a list of keywords printed in the style of a tag cloud, i.e. with different font sizes corresponding to the relative frequency of each keyword in documents covering the respective topic. The example shown in Fig. 2 can be used to assign topic relevance scores to one or more of the topics 202, 204, 206, 208 and 210, and, optionally, to assign keyword relevance scores to one or more of the depicted keywords. The particular choice of a tag cloud list of topics as topic representation 200 is merely exemplary, and without limitation, other choices for topic representation 200 can utilize topic maps, tag clouds, tree structures, lists, tables, or combinations thereof.

[0054] FIG. 3 illustrates relationships that can facilitate the handling of relevance scores according to an embodiment of the present invention. A received set of sample documents 132 is analyzed and a single sample document is assigned to a topic by application of a topic model, which is assumed to be known and therefore not described herein. The topics are presented to the user 101 as a topic representation 200 including an interface 300, which allows the user 101 to set a topic relevance score for one or more of the topics. In the non-exhaustive example shown in FIG. 3, each topic can be assigned one topic relevance level from a group including the topic relevance levels "relevant", "partially relevant" and "not relevant". In the particular example of FIG. 3, topic 1 is assigned to the topic relevance level "relevant", which may also be referred to as a high relevance level, topic 2 is assigned to the topic relevance level "partially relevant", which may also be referred to as an intermediate relevance level, and topic p is assigned to the topic relevance level "not relevant", which may also be referred to as a low relevance level.

[0055] FIG. 3 is simplified by showing only four data sources 310 from which six sample documents of the set 132 originate. Topic 1 is populated with one document from source S1 and one document from source Sn, topic 2 is populated with one document from source S1, and topic p is populated with one document from source S2, one document from source S3, and one document from source Sn. Thus, source S1 provides one sample document with high relevance and one document with medium relevance, sources S2 and S3 provide one sample document with low relevance, and source Sn provides one sample document with high relevance and one sample document with low relevance. In the illustrated simplified example, a typical and merely exemplary method of categorizing the data sources 310 into source relevance scores can be to assign weights to the topic relevance scores, e.g., 100% for each document with high relevance, 50% for each document with partial relevance, and 0% for each document with low relevance. In the same example, the averages per data source may yield weights of 75% for source S1, 0% for sources S2 and S3, and 50% for source Sn. In a further non-limiting example, a threshold average weight of 33% may be applied to determine between low and medium source relevance levels, and a threshold average weight of 67% may be applied to determine between high and medium source relevance levels. In the same example, this may result in classification of S1 as a high relevance data source, S2 and S3 as low relevance data sources, and Sn as a medium relevance data source. It is noted that the present invention does not contemplate any restriction to applying any modified or different approach to deriving source relevance scores from topic relevance scores.

[0056] 4 is a flow diagram illustrating steps of an exemplary method for refining an initial search request to a content provider, according to an embodiment of the present invention. The steps of the method are described below from the perspective of a computing device 10 incorporating an algorithm implementing the method. The method includes receiving 402 an initial search request from a user. The initial search request is then transmitted 404 to a content provider 130, for example using a computing or communication network 120. A set 132 of sample documents matching the initial search request is then received 406 from the content provider. Each sample document is assigned a data source identifier representing the data source 310 from which the respective sample document was provided.

[0057] At step 408, a topic model is applied to the received set of sample documents 132. The topic model can identify mutually exclusive topics and each of the sample documents can be assigned to one of the identified topics. Step 408 further includes obtaining topic representations 200 of the topics that can be output to a user. The topic representations are then presented to the user at step 410, for example using an output device of the computing device 10. In response to the presentation at step 410, at step 412, topic relevance scores for one or more of the presented topics are received from the user.

[0058] Next, in step 414, the data sources 310 are sorted according to the topic relevance scores. As a result, one or more of the data sources 310 are assigned a source relevance score derived from the received topic relevance score(s) by applying appropriate logic. Then, in step 416, further logic is applied to determine a refined search request based on the obtained source relevance score(s). The refined search request can be based on the initial search request or can be newly constructed based on the keywords, topic relevance scores and / or source relevance scores obtained during steps 408, 412 and 414. The refined search request is constructed to have an improved selectivity compared to the initial search request with respect to sample documents assigned to one or more of the topics that received the highest topic relevance scores during the user's assignment in step 412, or other documents that match the initial search request and are assigned to one or more of these highest scoring topics if included in the set of sample documents 132.

[0059] Embodiments of the present invention may be practiced using a computing device, which may be referred to as a computer system, client, or server. Referring now to FIG. 5, an example computer system is shown in schematic form. Computer system 10 is merely one example of a suitable computer system and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the present invention described herein. Nevertheless, computer system 10 may implement and / or perform any of the functions described above.

[0060] In computer system 10, computer system / server 12 is operable with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments or configurations or combinations thereof that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0061] The computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0062] As shown in Figure 5, the computer system / server 12 in the computer system 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components, including the system memory 28, to the processor 16. The bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0063] Computer system / server 12 typically includes a variety of computer system readable media, which may be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0064] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 or cache memory 32, or both. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such cases, each may be connected to the bus 18 by one or more data media interfaces. As further shown and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0065] By way of example and not limitation, stored within memory 28 may be a program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. Program modules 42 generally perform the functions and / or methods of embodiments of the invention described herein.

[0066] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, or the like, one or more devices that allow a user to interact with the computer system / server 12, or any device that allows the computer system / server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.), or combinations thereof. Such communication may occur via an input / output (I / O) interface 22. Additionally, the computer system / server 12 may also communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or combinations thereof, via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. Although not shown, it is understood that other hardware and / or software components may be used with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0067] A computer system, such as computer system 10 shown in Figure 5, can be used to perform the operations disclosed herein, such as refining an initial search request to a content provider. Such a computer system can be a stand-alone computer that is not connected to a network and can receive the initial search request, a set of sample documents with corresponding source identifiers, or data to be processed, such as topic relevance scores, through a local interface. However, such operations can equally be performed using a computer system that is connected to a network, such as a communications network or a computing network, or both.

[0068] The present invention may be embodied as a system, method, or computer program product, or combination thereof, at any possible level of technical detail. The computer program product may include a computer readable storage medium or media having computer readable program instructions for causing a processor to carry out aspects of the present invention.

[0069] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium is not to be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electrical signal sent through a wire.

[0070] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.

[0071] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the last scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0072] Aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0073] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both. These computer program instructions, which may direct a computer, other programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, may also be stored in a computer readable medium, such that the instructions stored in the computer readable medium include an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both.

[0074] The computer readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer or other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both.

[0075] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts may represent a module, segment, or portion of code, including one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or these blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.

[0076] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be practiced in conjunction with any other type of computing environment now known or later developed.

Claims

1. 1. A computer-implemented method for refining an initial search request to a content provider, comprising: receiving the initial search request from a user; sending the initial search request to the content provider; receiving from the content provider a set of sample documents and a source identifier for each one of the sample documents, the source identifier identifying a respective one of a data source associated with the respective one of the sample documents; applying a topic model to the set of sample documents to obtain a topic representation that is a description of the topics covered by each one of the sample documents; presenting the topic representation to the user; and receiving a topic relevance score from the user for each one of the topics; sorting the data sources according to the topic relevance scores to obtain a source relevance score for each one of the data sources; determining a refined search request based on the source relevance scores, the refined search request having increased selectivity for documents covering the topics having the highest scores among the topics; The method includes:

2. 2. The method of claim 1, wherein the refined search request includes a conjunction of the initial search request and a source criterion that reduces a number of data sources covered by the refined search request compared to a number of data sources covered by the initial search request.

3. 3. The method of claim 2, wherein the source criteria includes either a focus statement that selects one or more of the data sources based on the source relevance score, or an exclusion statement that deselects one or more of the data sources based on the source relevance score.

4. Categorizing the data sources includes selecting the source relevance scores from a predetermined set of discrete source relevance levels, the method comprising: determining, for each one of the discrete source relevance levels used to determine the source criteria, a total number of hits found for the initial search request in each one of the data sources having the respective one of the discrete source relevance levels; identifying, from said data sources having said respective ones of said discrete source relevance levels, N said data sources having a maximum total number of hits, where N is a predetermined upper limit; Further comprising: determining the refined search request includes populating the source criteria with N identified data sources for each one of the discrete source relevance levels used to determine the source criteria. The method according to any one of claims 2 to 3.

5. 5. The method of claim 1, wherein categorizing the data sources comprises selecting the source relevance scores from a predetermined set of discrete source relevance levels, and determining the refined search request comprises selecting one or more of the source relevance levels to improve selectivity, and the topic relevance scores are selected from a predetermined set of discrete topic relevance levels.

6. 6. The method of claim 1, wherein the source relevance scores and the topic relevance scores are selected from a predetermined set of discrete common relevance levels, and wherein determining the refined search request comprises selecting one or more of the discrete common relevance levels to improve selectivity.

7. 7. The method of claim 6, wherein the predetermined set of discrete common relevance levels includes a high relevance level, classifying the data sources includes assigning the given data source to a high relevance level if a relative frequency of the sample documents associated with the given data source and covering a topic assigned to the high relevance level equals or exceeds a predetermined high relevance threshold, and determining the refined search request includes limiting the refined search request to one or more of the data sources assigned to the high relevance level compared to the initial search request when the high relevance level is used to determine the refined search request.

8. The method of claim 7 , wherein the predetermined high relevance threshold is 100 percent.

9. 9. The method of claim 7, wherein the set of discrete common relevance levels further includes a low relevance level, and wherein classifying the data sources includes assigning the given data source to a low relevance level if a relative frequency of the sample documents associated with the given data source and covering a topic assigned to the low relevance level is equal to or exceeds a first predetermined low relevance threshold, or if a relative frequency of the sample documents associated with the given data source and covering a topic assigned to the high relevance level is equal to or less than a second predetermined low relevance threshold, or both, and wherein determining the refined search request includes excluding from the refined search request one or more of the data sources assigned to the low relevance level compared to the initial search request when the low relevance level is used to determine the refined search request.

10. The method of claim 9 , wherein the first low relevance threshold is 100 percent and the second low relevance threshold is zero percent.

11. The method of any one of claims 1 to 10, further comprising obtaining a precision level, the refined search request being further determined based on the precision level, the precision level being obtained from a predetermined set of discrete precision levels, and the increase in selectivity being based on the precision level.

12. The method of claim 11 , wherein the accuracy level is obtained as an input from the user.

13. the set of precision levels includes a high precision level, a balanced precision level, and a high recall level; Each one of the source relevance scores is selected from a group including a high relevance level, a medium relevance level, and a low relevance level, and classifying the data source includes assigning the given data source to a high relevance level if a relative frequency of sample documents associated with the given data source and covering topics assigned to a high relevance level equals or exceeds a predetermined high relevance threshold, and classifying the data source further includes assigning the given data source to a low relevance level if a relative frequency of sample documents associated with the given data source and covering topics assigned to a low relevance level equals or exceeds a predetermined first low relevance threshold, or if a relative frequency of sample documents associated with the given data source and covering topics assigned to a high relevance level equals or exceeds a predetermined second low relevance threshold, or both, and classifying the data source further includes assigning the given data source to a high relevance level.

13. The method of claim 11, wherein if the precision level is a high precision level, the refined search request comprises a conjunction of the initial search request with a focusing statement selecting one or more of the data sources having a high relevance level as the source relevance score; if the precision level is the balanced precision level, the refined search request comprises a conjunction of the initial search request with a focusing statement including a logical OR of one or more of the data sources having the high relevance level as the source relevance score and one or more of the data sources having the intermediate relevance level as the source relevance score; and if the precision level is a high recall level, the refined search request comprises a conjunction of the initial search request with a focusing statement including a logical negation of one or more of the data sources having the low relevance level as the source relevance score.

14. each said topic relevance score being selected from a topic relevance group, said topic relevance group including a high relevance level indicating a highest relevance compared to all other relevance levels therein; The method comprises: The number D of sample documents covering topics that have a high relevance level as a topic relevance score Ｒ determining a D for the total number D of sample documents in the set of sample documents Ｒ Ratio of D Ｒ if / D is less than or equal to a predetermined low-precision threshold, the high-precision level is set as the precision level; D Ｒ when / D is greater than a predetermined low precision threshold and less than a predetermined upper precision threshold, the balanced precision level is set as the precision level; D Ｒ If / D is greater than or equal to a predetermined high precision threshold, the high recall level is set as the precision level. The method of claim 13.

15. each said topic relevance score being selected from a topic relevance group, said topic relevance group including a high relevance level indicating a highest relevance as compared to all other relevance levels therein; The method comprises: a number D of documents available as a search result for a hypothetical search request that includes a conjunction of the initial search request and a focus statement that selects one or more of the data sources having the high relevance level as the source relevance score; ＲＳ from said content provider; The number D of documents available as a search result for the initial search request Ｋ from said content provider; Further comprising: D Ｋ D against ＲＳ Ratio of D ＲＳ / D Ｋ is less than or equal to a predetermined low-precision threshold, the high-precision level is set as the precision level; D ＲＳ / D Ｋ is greater than a predetermined low precision threshold and less than a predetermined upper precision threshold, the balanced precision level is set as the precision level; D ＲＳ / D Ｋ is greater than or equal to a predetermined high precision threshold, the high recall level is set as the precision level. The method according to any one of claims 13 to 14.

16. receiving from the user a document relevance score for the respective one of the sample documents, wherein classifying the data source is further based on the document relevance score; receiving from the user the source relevance score for the respective one of the data sources, the data source having the source relevance score received from the user being excluded from categorizing the data sources; receiving from the user a keyword relevance score for each one of the keywords representing the each one of the topics, wherein classifying the data source is further based on the keyword relevance score; The method of any one of claims 1 to 15, further comprising:

17. applying a predetermined default topic relevance score to topics that do not receive said topic relevance score; applying a predetermined default source relevance score to said data sources whose source relevance scores were not determined during classification of said data sources; The method of any one of claims 1 to 16, further comprising:

18. The method of any one of claims 1 to 17, wherein the topic representations are limited to topics that equal or exceed a predetermined minimum number of hits for the initial search request.

19. A computer program comprising computer readable program instructions causing a computer to carry out a method according to any one of claims 1 to 18.

20. 1. A computer system for refining an initial search request to a content provider, comprising: one or more processors; one or more computer-readable tangible storage devices; and program instructions stored on at least one of the one or more computer-readable tangible storage devices for execution by at least one of the one or more processors; The program instructions include: receiving said initial search request from a user; sending the initial search request to the content provider; receiving from the content provider a set of sample documents and a source identifier for each one of the sample documents, the source identifier identifying a respective one of data sources associated with the respective one of the sample documents; applying a topic model to the set of sample documents to obtain a topic representation that is a description of the topics covered by each one of the sample documents; presenting the topic representation to the user; receiving a topic relevance score from the user for each one of the topics; sorting the data sources according to the topic relevance scores to obtain a source relevance score for each one of the data sources; determining a refined search request based on the source relevance scores, the refined search request being more selective to documents covering the topics with the highest scores among the topics; It is feasible to Computer system.

Images