Search device, search method, and search program

Abstract

Outputs search results that meet the display criteria. [Solution] The search device includes a calculation means for determining the confidence level of each search result according to the similarity score indicating the degree to which the query matches one or more search results for the query, and an extraction means for outputting a search result from among the one or more search results whose confidence level satisfies the display criteria.

Description

[Technical Field]

[0001] This disclosure relates to a search device, a search method, and a search program. [Background technology]

[0002] One example of a technique for excluding search results unrelated to the query is the technique described in Non-Patent Document 1. Non-Patent Document 1 describes a method for ordering search results by embedding the query and search results into a vector space, calculating the cosine similarity of the obtained vectors, and transforming the cosine similarity using a transformation function based on the query. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Rossi et. al., Relevance Filtering for Embedding-based Retrieval, arXiv:2408.04887v1, 9 Aug 2024 [Overview of the project] [Problems that the invention aims to solve]

[0004] For example, when it is necessary to review a large number of search results in a short time, such as when checking images of disaster situations, it is beneficial to narrow down the number of search results displayed. Criteria for narrowing down search results include, for example, a confidence level of a certain percentage or higher if the user wants to display only reliable search results; a precision level of a certain percentage or higher if the user prioritizes the proportion of correct search results; and a recall level of a certain percentage or higher if the user wants to prevent missing correct candidates.

[0005] In the technology described in Non-Patent Document 1, there was a problem in that in order to display results that met such standards, it was necessary to manually prepare correct labels indicating whether the results were correct or incorrect for all search results, and to manually set thresholds.

[0006] This disclosure has been made in view of the above-mentioned issues, and one exemplary purpose thereof is to provide a technology that outputs search results according to the criteria to be displayed. [Means for solving the problem]

[0007] A search device relating to an exemplary aspect of this disclosure includes one or more search results for a query, a calculation means for determining the confidence level of each search result according to the degree of similarity indicating the degree of consistency with the query, and an extraction means for outputting a search result from among the one or more search results in which the confidence level satisfies the display criteria.

[0008] A search method relating to an illustrative aspect of this disclosure includes determining a confidence level for each search result based on one or more search results for a query and a similarity score indicating the degree of consistency with the query, and outputting a search result from among the one or more search results whose confidence level satisfies the criteria for display.

[0009] An exemplary aspect of this disclosure is a search program for causing a computer to function as a search device, which causes the computer to perform a calculation process that determines the confidence level of each search result according to the similarity score indicating the degree to which the query matches one or more search results for the query, and an extraction process that outputs a search result from among the one or more search results in which the confidence level satisfies the criteria for display. [Effects of the Invention]

[0010] According to an illustrative aspect of this disclosure, one exemplary effect is that it is possible to output search results that meet the display criteria. [Brief explanation of the drawing]

[0011] [Figure 1] It is a block diagram showing the configuration of the search device according to the present disclosure. [Figure 2] It is a flow diagram showing the flow of the search method according to the present disclosure. [Figure 3] It is a block diagram showing the configuration of the search device according to the present disclosure. [Figure 4] It is a flow diagram showing the flow of the search method according to the present disclosure. [Figure 5] It is a flow diagram showing the flow of the search method according to the present disclosure. [Figure 6] It is a flow diagram showing the flow of the search method according to the present disclosure. [Figure 7] It is a block diagram showing the configuration of the search device according to the present disclosure. [Figure 8] It is a flow diagram showing the flow of the search method according to the present disclosure. [Figure 9] It is a graph for explaining an example of the calibration method according to the present disclosure. [Figure 10] It is a graph for explaining an example of the calibration method according to the present disclosure. [Figure 11] It is a flow diagram showing the flow of the search method according to the present disclosure. [Figure 12] It is a block diagram showing the configuration of the search device according to the present disclosure. [Figure 13] It is a block diagram showing the configuration of the search device according to the present disclosure. [Figure 14] It is a block diagram showing the configuration of a computer that functions as the search device according to the present disclosure. MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be exemplified. However, the present invention is not limited to the following exemplary embodiments, and various modifications are possible within the scope defined in the claims. For example, embodiments obtained by appropriately combining the techniques (part or all of an article or method) employed in each of the following exemplary embodiments are also included within the scope of the present invention. In addition, embodiments obtained by appropriately omitting part of the techniques employed in each of the following exemplary embodiments are also included within the scope of the present invention. Furthermore, the effects mentioned in each of the following exemplary embodiments are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. That is, embodiments that do not achieve the effects mentioned in each of the following exemplary embodiments are also included within the scope of the present invention.

[0013] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. The present exemplary embodiment is a basic form for each of the exemplary embodiments described later. The scope of application of each technique employed in the present exemplary embodiment is not limited to the present exemplary embodiment. That is, each technique employed in the present exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, as long as no particular technical obstacle arises. In addition, each technique shown in the drawings referred to for describing the present exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, as long as no particular technical obstacle arises.

[0014] The search device 1 is a device that outputs search results corresponding to display criteria from among one or more search results for a query.

[0015] The "display criteria" are not particularly limited, and examples thereof include criteria related to confidence probability, precision, recall, and F measure.

[0016] Furthermore, the "criteria for display" may also be that evaluation values ​​such as confidence probability, precision, recall, and F-measure are within a range of a predetermined percentage (%) to a predetermined error range (%) (allowable error range) (first criterion for display), and the maximum evaluation value that differs from the first criterion for display (second criterion for display).

[0017] The use of search device 1 is not particularly limited, and it can be suitably used in various search devices or search systems. In particular, for example, when searching from a large number of on-site photographs to understand the situation of a disaster, or when it is necessary to reflect the search results on a map, it is beneficial to display a number of search results corresponding to the user's display criteria, and therefore it can be suitably used in such cases. The same applies when performing searches in police investigation support or in media that handle large amounts of video and image data.

[0018] (Configuration of the search device) The configuration of the search device 1 will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the search device 1. As shown in Figure 1, the search device 1 comprises a calculation unit 12 and an extraction unit 13.

[0019] The calculation unit 12 calculates the confidence level of each of the one or more search results based on the similarity score, which indicates the degree to which the search result matches the query.

[0020] "Similarity, which indicates the degree to which search results and queries are consistent," is an index that shows how semantically similar the search results and what the query represents are. For example, cosine similarity between embedded vectors in a vector space, or relevance calculated by algorithms such as BM25, can be used. Note that the data formats of the search results and queries may be the same or different.

[0021] "Search result confidence" indicates the estimated probability that the search result is correct for the query.

[0022] The extraction unit 13 outputs search results according to the display criteria based on the confidence level of each of the one or more search results.

[0023] "Outputting search results according to display criteria" means outputting search results in a way that the outputted search results meet the display criteria. Alternatively, "Outputting search results according to display criteria" means selecting and outputting search results in a way that the outputted search results meet the display criteria.

[0024] (Effectiveness of the search device) As described above, the search device 1 employs a configuration that calculates confidence levels from the similarity of the search results and outputs search results based on these confidence levels. Therefore, the search device 1 has the effect of being able to output search results that correspond to the display criteria.

[0025] (Search process) The flow of search method S1 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of search method S1. Search method S1 is a method for outputting one or more search results for a query according to the display criteria, and may be executed by search device 1. As shown in Figure 2, search method S1 includes a calculation process S11 and an extraction process S12.

[0026] In calculation process S11, for each of the one or more search results, the confidence level of the search result is calculated from the similarity score, which indicates the degree to which the search result matches the query.

[0027] In extraction process S12, search results are output according to the display criteria based on the confidence level of each of the one or more search results.

[0028] (Effectiveness of search methods) As described above, the search method S1 employs a configuration in which confidence is calculated from the similarity of the search results and the search results are output based on the confidence level. Therefore, the search method S1 can achieve the same effect as the search device 1.

[0029] [Second exemplary embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0030] (Configuration of the search device) The configuration of the search device 2 will be explained with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the search device 2.

[0031] The search device 2 includes a control unit 10. The control unit 10 includes a preprocessing unit 11, a calculation unit 12a, and an extraction unit 13a. The preprocessing unit 11 includes an acquisition unit 111, a first calculation unit 113, a second calculation unit 114, a third calculation unit 115, and an alignment unit 116. The calculation unit 12a includes a fourth calculation unit 121 and a normalization unit 122. The extraction unit 13a includes an estimation unit 131 and a first comparison unit 132.

[0032] The acquisition unit 111 acquires the query, a column of search results containing one or more search results for the query, and the criteria for display.

[0033] The queries and display criteria may be obtained by the acquisition unit 111 from user input via an input device (not shown), or by the acquisition unit 111 from another device. One or more search results for a query may be obtained by the acquisition unit 111 by searching an arbitrary database using the query, or by the acquisition unit 111 from another device.

[0034] The first calculation unit 113 embeds the query into a specific vector space and calculates the query vector. The first calculation unit 113 may use a neural network model to perform the embedding process into the specific vector space.

[0035] The second calculation unit 114 embeds each of the one or more search results included in the search result column into a specific vector space and calculates a search result vector. The second calculation unit 114 may use a neural network model to perform the embedding process into the specific vector space.

[0036] The neural network models used by the first calculation unit 113 and the second calculation unit 114 may be the same or different, but it is preferable to use different neural network models if the data formats of the query and the search results are different (for example, text and images).

[0037] The third calculation unit 115 calculates a similarity score indicating the degree to which each search result matches the query by calculating the similarity (e.g., cosine similarity) between the query vector and the search result vector. The third calculation unit 115 then calculates a similarity column that includes the similarity score for each search result.

[0038] The sorting unit 116 rearranges (sorts) the search results column and the similarity column. In the following description, the order of the similarity column is in descending order based on similarity. Similarly, the order of the search results column is in descending order based on the similarity corresponding to the search results. However, the order of the search results column and the similarity column is not limited to this; any order based on similarity is acceptable, and it may also be in ascending order.

[0039] The calculation unit 12a calculates the confidence level of each search result from the similarity of each search result included in the similarity column. In one embodiment, the calculation unit 12a calculates the confidence level column by converting each similarity to a confidence level indicated by a value between 0.0 and 1.0, for example.

[0040] In detail, first, the fourth calculation unit 121 determines the maximum and minimum similarity values. The fourth calculation unit 121 may determine the maximum value in the similarity column as the maximum similarity value, or it may determine a predetermined value as the maximum similarity value. Furthermore, the fourth calculation unit 121 may determine the minimum value in the similarity column as the minimum similarity value, or it may determine a predetermined value as the minimum similarity value.

[0041] Next, the normalization unit 122 uses the maximum similarity value max and the minimum similarity value min to convert the similarity s into a confidence score p between 0.0 and 1.0 using the following formula (1), thereby calculating a column of confidence scores.

[0042]

number

[0043] The extraction unit 13a calculates the number of output search results according to the display criteria, based on the confidence level of each search result included in the confidence level column.

[0044] In detail, first, the estimation unit 131 estimates a confusion matrix for each candidate number of search results to be output. In one embodiment, the candidate number of search results to be output can be any integer from 1 to N (where N is the total number of search results included in the search result column).

[0045] A "confusion matrix" is a matrix that summarizes the results of binary classification, and is shown by equation (2) below.

[0046]

number

[0047] TP represents the number of correct responses (true positives) among the selected subjects. FP represents the number of incorrect responses (false positives) among the selected subjects. FN represents the number of correct responses (false negatives) among the unselected subjects. TN represents the number of incorrect responses (true negatives) among the unselected subjects. Evaluation values ​​such as precision, recall, and F-measure can be calculated from the confusion matrix. The F-measure may be F1 or any other value.

[0048] In this case, the search results column does not have correct or incorrect labels manually attached to all search results. Therefore, with conventional technology, the number of correct and incorrect answers is unknown, making it impossible to directly calculate the confusion matrix mentioned above, and thus difficult to calculate the evaluation value mentioned above. In contrast, search device 2 can calculate the evaluation value mentioned above by estimating the confusion matrix based on the confidence level.

[0049] In other words, the estimation unit 131 calculates the following (1) to (4) for each candidate number of search results to be output, based on the confidence column.

[0050] (1) The estimated number of correct search results (expected value of TP) included in the candidate search results for the number of search results to be output. (2) Estimated number of incorrect search results included in the candidate search results to be output (expected value of FP) (3) The estimated number of correct search results (expected value of FN) included in the column of search results excluding candidate search results to be output. (4) The estimated number of incorrect search results (expected value of TN) included in the column of search results excluding the candidate search results to be output. In one embodiment, "candidate search results extracted based on the candidate number of search results to be output" refers to the candidate number of search results to be output, starting from the beginning of the search results column. If the search results column is in ascending order, the candidate number of search results to be output will start from the end of the search results column.

[0051] Here, the confidence level of a search result is modeled after the probability that the search result is correct (the probability of being correct), so the confidence level can be used as the probability of being correct. Also, a correct answer can be considered as 1 correct answer, and an incorrect answer as 0 correct answers.

[0052] Therefore, in one embodiment, the estimation unit 131 can estimate the confusion matrix by calculating the expected value of each element (TP, FP, FN, TN) of the confusion matrix for each candidate number of search results to be output, as follows. Note that N represents the total number of search results included in the search results column. p This indicates the number of search results to output.

[0053] (1) Expected value of TP = 1 × (1st confidence level in the confidence level column) + 1 × (2nd confidence level in the confidence level column) + ... + 1 × (Nth confidence level in the confidence level column) p (Second confidence level) In other words, the expected value of TP is the number of candidate search results to output, N. p This represents the sum of the probabilities of each individual search result being correct.

[0054] (2) Expected value of FP = 1 × (1 - (1st confidence in the confidence column)) + 1 × (1 - (2nd confidence in the confidence column)) + ... + 1 × (1 - (Nth confidence in the confidence column) p (Second confidence level)) (1 - confidence level) represents the probability of an incorrect answer; in other words, the expected value of FP is the number of candidate search results that will be output (N). pis the sum of the probabilities of incorrect answers among the search results.

[0055] (3) Expected value of FN = 1 × (the (N p +1)-th confidence value in the confidence sequence) + 1 × (the (N p +2)-th confidence value in the confidence sequence) + ... + 1 × (the N-th confidence value in the confidence sequence) In other words, the expected value of FN is the sum of the probabilities of correct answers in (N-N p ) search results other than the output search result candidates.

[0056] (4) Expected value of TN = 1 × (1-(the (N p +1)-th confidence value in the confidence sequence)) + 1 × (1-(the (N p +2)-th confidence value in the confidence sequence)) + ... + 1 × (1-(the N-th confidence value in the confidence sequence)) In other words, the expected value of TN is the sum of the probabilities of incorrect answers in (N-N p ) search results other than the output search result candidates.

[0057] Then, the estimating unit 131 estimates a confusion matrix for candidates of the number of search results to output from 1 to N (where N is the total number of search results included in the search result sequence), and outputs the confusion matrix as a column together with the column of candidates of the number of search results to output to the first comparing unit 132.

[0058] For each candidate of the number of search results to output, the first comparing unit 132 calculates one or more evaluation values defined by criteria to be displayed from each element of the confusion matrix based on, for example, the following formulas (3) to (5).

[0059] [Mathematics]

[0060] [Mathematics]

[0061] [Mathematics]

[0062] The first comparison unit 132 then identifies candidate numbers of output search results that satisfy the criteria displayed by one or more evaluation values, and outputs them as the number of output search results calculated by the extraction unit 13a.

[0063] In one embodiment, the display criteria may include the smallest difference between the evaluation value and the standard value, the evaluation value being greater than the standard value and the smallest, or the evaluation value being the largest.

[0064] The extraction unit 13a extracts search results to be output from the search results column based on the number of search results to be output. In one embodiment, the extraction unit 13a may extract search results to be output based on the number of search results to be output from the search results column. In particular, the extraction unit 13a may extract several search results to be output from the beginning of the search results column. If the search results column is in ascending order, then several search results to be output will be selected from the end of the search results column.

[0065] Furthermore, the extraction unit 13a may display the output search results on a display device (not shown), or it may transmit the output search results to another device via a communication path.

[0066] (Operation flow of the search device) The operation flow of the search device 2 will be explained with reference to Figures 4 to 6. Figure 4 is a flowchart showing the flow of search method S2. Figure 5 is a flowchart showing the flow of step S24 of search method S2. Figure 6 is a flowchart showing the flow of step S25 of search method S2.

[0067] In the following explanation, we will describe, as an example, a case where the search result column includes search result a (correct), search result b (incorrect), and search result c (correct), and the display criteria are that the recall is closest to 0.95 and the precision is the highest. However, this is not limited to this embodiment.

[0068] First, let's explain with reference to Figure 4. In step S21, the acquisition unit 111 acquires the query, the columns of the search results (a, b, c), and the display criteria.

[0069] In step S22, the first calculation unit 113 calculates the query vector, the second calculation unit 114 calculates the search result vector, and the third calculation unit 115 calculates the similarity based on the query vector and the search result vector to calculate a similarity column (a: 0.5, b: 0.1, c: 0.4).

[0070] In step S23, the sorting unit 116 sorts the search result column and the similarity column based on the similarity to calculate a similarity column sorted in descending order (a:0.5, c:0.4, b:0.1) and a search result column sorted in descending order (a, c, b).

[0071] In step S24, the calculation unit 12a calculates a column of confidence scores arranged in descending order. Specifically, as shown in Figure 5, in step S241, the fourth calculation unit 121 determines the maximum similarity value (0.5) and the minimum similarity value (0.1). Then, in step S241, the normalization unit 122 uses the maximum similarity value (0.5) and the minimum similarity value (0.1) to convert the similarity of the column of similarity scores arranged in descending order (a:0.5, c:0.4, b:0.1) into confidence scores, so in step S243, the calculation unit 12a calculates a column of confidence scores arranged in descending order (a:1.0, c:0.75, b:0.0).

[0072] Returning to Figure 4, in step S25, the extraction unit 13a calculates the number of output search results according to the display criteria, based on the confidence level of each search result included in the confidence level column sorted in descending order.

[0073] In detail, as shown in Figure 6, in step S251, the estimation unit 131 estimates a confusion matrix for each candidate number of search results to be output (1 to 3) based on a column of confidence values ​​arranged in descending order (a: 1.0, c: 0.75, b: 0.0). The confusion matrix corresponding to candidate 1 for the number of search results to output is estimated to be (TP=1.0, FP=1-1.0, FN=0.75+0.0, TN=(1-0.75)+(1-0.0)), the confusion matrix corresponding to candidate 2 for the number of search results to output is estimated to be (TP=1.0+0.75, FP=(1-1.0)+(1-0.75), FN=0.75+0.0, TN=1-0.0), and the confusion matrix corresponding to candidate 3 for the number of search results to output is estimated to be (TP=1.0+0.75+0.0, FP=(1-1.0)+(1-0.75)+(1-0.0), FN=0, TN=0).

[0074] In step S252, the first comparison unit 132 calculates an evaluation value for each candidate for the number of search results to be output, based on the confusion matrix estimated by the estimation unit 131. The evaluation value corresponding to candidate 1 for the number of search results to be output is calculated as (precision = 1.0, recall = 0.57), the evaluation value corresponding to candidate 2 for the number of search results to be output is calculated as (precision = 0.88, recall = 1.0), and the evaluation value corresponding to candidate 3 for the number of search results to be output is calculated as (precision = 0.58, recall = 1.0).

[0075] In step S253, the first comparison unit 132 identifies (2) candidate numbers of search results that satisfy the display criteria (the recall closest to 0.95 and the precision highest) and outputs them as the number of search results to be output.

[0076] Returning to Figure 4, in step S26, the extraction unit 13a outputs search results (a, c) from the columns of search results (a, c, b) arranged in descending order, based on the number of search results to be output (2).

[0077] (modified version) The acquisition unit 111 may directly acquire the columns of search results sorted in descending order and the columns of similarity sorted in descending order. In that case, the preprocessing unit 11 may omit the first calculation unit 113, the second calculation unit 114, the third calculation unit 115, and the sorting unit 116.

[0078] Furthermore, while the above explanation treats search results, similarity, and confidence as separate columns, it is not limited to this. In other words, if the search results, similarity, and confidence are linked, they do not necessarily need to be treated as separate columns. The same applies to other embodiments.

[0079] [Third Exemplary Embodiment] A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiments are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs.

[0080] The configuration of the search device 3 will be explained with reference to Figure 7. Figure 7 is a block diagram showing the configuration of the search device 3.

[0081] The search device 3 includes a control unit 10. The control unit 10 includes a preprocessing unit 11, a calculation unit 12b, and an extraction unit 13b. The calculation unit 12b includes an assignment unit 123 and a calibration unit 124. The extraction unit 13b includes a second comparison unit 133.

[0082] The calculation unit 12b calculates the confidence level of each search result from the similarity of each search result included in the similarity column. In one embodiment, the calculation unit 12b calculates the confidence level column by converting each similarity to a confidence level, for example, represented by a value between 0.0 and 1.0.

[0083] In detail, first, the labeling unit 123 assigns labels to at least some of the search results included in the search result column, indicating whether or not the search results are presumed to be correct based on the search results and the query.

[0084] In one embodiment, the labeling unit 123 may assign labels obtained by inputting the search results and the query into a trained model to at least some of the search results included in the search results column. In another embodiment, the labeling unit 123 may assign labels to at least some of the search results included in the search results column based on user input.

[0085] Next, the calibration unit 124 calibrates the function used to calculate the confidence level from the similarity, based on the labels. In one embodiment, the function calibrated by the calibration unit 124 is a conversion function f that converts the similarity s to the confidence level p, as shown in equation (6) below.

[0086]

number

[0087] In one embodiment, the calibration unit 124 calibrates a function used to calculate confidence from similarity by using labels assigned in a calibration algorithm using known labels as shown below. Histogram binning • Isotonic regression ·Bayesian binning into quantiles • Platform scaling • Temperature scaling Here, we will explain the calibration method using histogram binning. First, the calibration unit 124 generates a histogram for each search result with a label, where the horizontal axis is similarity and the vertical axis is the number of correct labels and the number of incorrect labels. Figure 9 shows an example of a histogram. The shaded areas indicate the number of correct labels, and the dots indicate the number of incorrect labels.

[0088] Then, the calibration unit 124 calibrates the function used to calculate confidence from similarity, based on the generated histogram, so that the relationship between similarity and confidence is as shown in Figure 10.

[0089] Then, the calculation unit 12b calculates a column of confidence scores by converting the similarity scores into confidence scores using the function calibrated by the calibration unit 124.

[0090] The extraction unit 13b calculates the number of output search results according to the display criteria, based on the confidence level of each search result included in the confidence level column.

[0091] In detail, the second comparison unit 133 compares each confidence level included in the confidence level column with the threshold indicated by the display criterion, and counts the number of confidence levels that are equal to or greater than the threshold. The extraction unit 13b calculates the number of confidence levels counted by the second comparison unit 133 as the number of search results to output.

[0092] As described above, the search device 3 is equipped with a calculation unit 12b and an extraction unit 13b instead of the calculation unit 12a and extraction unit 13a of the search device 2. However, similar to the search device 2, it can calculate confidence from similarity, determine the number of search results to output based on confidence, and extract items to be displayed.

[0093] (Operation flow of the search device) The operation flow of the search device 3 will be explained with reference to Figures 4, 8, and 11. Figure 8 is a flowchart showing the flow of step S24. Figure 11 is a flowchart showing the flow of step S25. The search device 3 basically operates the same way as the search device 2, as shown in Figure 4, but the operation of steps S24 and S25 differs from that of the search device 2.

[0094] In the following explanation, we will describe, as an example, a case where the search result column includes search result a (correct), search result b (incorrect), and search result c (correct), and the display criterion is a confidence level of 0.5 or higher. However, this is not limited to this embodiment.

[0095] First, let's refer to Figure 4 for explanation. Steps S21 to S23 are the same as in the second embodiment, so we will omit their explanation.

[0096] In step S24, the calculation unit 12b calculates a column of confidence scores sorted in descending order. Specifically, as shown in Figure 8, in step S244, the labeling unit 123 assigns labels (a: 1 (correct), b: 0 (incorrect)) to at least some of the search results (a, b) included in the column of search results (a, c, b) sorted in descending order, indicating whether the search results are presumed to be correct or not based on the search results and queries. In step S245, the calibration unit 124 calibrates the function used to calculate confidence scores from similarity based on the labels. Then, in step S246, the calculation unit 12b converts the similarity scores to confidence scores using the function calibrated by the calibration unit 124, and in step S247, calculates a column of confidence scores sorted in descending order (a: 1.0, c: 0.75, b: 0.0). Note that this result is the result when the calibration unit 124 uses isotonic regression.

[0097] Returning to Figure 4, in step S25, the extraction unit 13b calculates the number of search results to display according to the criteria, based on the confidence level of each search result included in the confidence column sorted in descending order.

[0098] In detail, as shown in Figure 11, in step S256, the second comparison unit 133 compares each confidence level in the confidence level column (a: 1.0, c: 0.75, b: 0.0) arranged in descending order with the threshold value (0.5) indicated by the display criterion, and counts the number of confidence levels (2) that are equal to or greater than the threshold value. In step S257, the extraction unit 13b calculates the number of confidence levels (2) counted by the second comparison unit 133 as the number of search results to output.

[0099] Returning to Figure 4, in step S26, the extraction unit 13b outputs search results (a, c) from the column of search results (a, c, b) arranged in descending order, based on the number of search results to be output (2).

[0100] [Fourth exemplary embodiment] A fourth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiments are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs.

[0101] The configuration of the search device 4 will be explained with reference to Figure 12. Figure 12 is a block diagram showing the configuration of the search device 4. The search device 4 includes a control unit 10. The control unit 10 includes a preprocessing unit 11, a calculation unit 12a, and an extraction unit 13b. Even with this configuration, similar to the search device 2, it is possible to calculate the confidence level from the similarity, determine the number of search results to output based on the confidence level, and output search results according to the display criteria.

[0102] [Fifth Exemplary Embodiment] A fifth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiments are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs.

[0103] The configuration of the search device 5 will be explained with reference to Figure 13. Figure 13 is a block diagram showing the configuration of the search device 5. The search device 5 includes a control unit 10. The control unit 10 includes a preprocessing unit 11, a calculation unit 12b, and an extraction unit 13a. Even with this configuration, similar to the search device 2, it is possible to calculate the confidence level from the similarity, determine the number of search results to output based on the confidence level, and output search results according to the display criteria. [Sixth Exemplary Embodiment] A sixth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiments are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs.

[0104] In the first to fifth embodiments, we will describe a case in which the display criteria are set to include a first display criterion, a tolerance for error in the first display criterion, and a second display criterion.

[0105] In this case, the extraction unit 13 may maintain a number of candidates for the number of search results to be output based on a first display criterion and an allowable error range, and then determine the number of search results to be output from among these candidates based on a second display criterion.

[0106] For example, if the first display criterion is that a predetermined evaluation value falls within a predetermined percentage range, which is the allowable error range, and the second display criterion is that the maximum evaluation value differs from the first display criterion, the extraction unit 13 may determine multiple candidates for the number of search results to output so that the predetermined evaluation value falls within a predetermined percentage range, which is the allowable error range, and then determine the number of the search results to output that has the maximum evaluation value differing from the first display criterion.

[0107] [Examples of implementation using software] Some or all of the functions of search devices 1 to 5 (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0108] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as Computer C) is shown in Figure 14. Figure 14 is a block diagram showing the hardware configuration of Computer C, which functions as each of the above devices.

[0109] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as each of the above-mentioned devices. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing each of the above-mentioned devices.

[0110] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0111] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0112] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0113] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0114] [Additional Note 1] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims. (Note 1) A calculation means for determining the confidence level of each search result based on one or more search results for a query and the similarity score indicating the degree to which the query is consistent, An extraction means that outputs a search result from among the one or more search results whose reliability meets the display criteria, A search device equipped with the following features. (Note 2) The extraction means determines the number of search results to output, For each candidate number of search results to output, The estimated number of correct search results included in the candidate for the number of search results to be output, corresponding to the candidate for the number of search results to be output. The estimated number of incorrect search results included in the candidate search results to be output, The estimated number of correct search results included in the one or more search results excluding the candidate search results to be output, and The estimated number of incorrect search results included in the one or more search results excluding the candidate search results to be output is A search device as described in Appendix 1, which calculates based on the confidence level of each of the one or more search results. (Note 3) The search device as described in Appendix 1, wherein the extraction means compares the confidence level and threshold of each of the one or more search results in order to determine the number of search results to be output. (Note 4) The search device according to any one of the appendices 1 to 3, wherein the calculation means determines the maximum and minimum values ​​of the similarity, and calculates the confidence level from the similarity using the maximum and minimum values ​​of the similarity. (Note 5) The calculation means is A means for assigning labels to at least some of the one or more search results, indicating whether the search results are presumed to be correct based on the search results and the query, A search device according to any one of appendices 1 to 3, comprising: a calibration means for calibrating a function used to calculate the confidence level from the similarity based on the label. (Note 6) The search device according to Appendix 5, wherein the labeling means assigns the labels obtained by inputting the search results and the queries into a trained model to at least some of the one or more search results. (Note 7) The calculation means calculates a column of confidence scores by converting each similarity score in the column of similarity scores, in which the similarity scores of the one or more search results are arranged in order based on the similarity scores, The search device according to Appendix 1, wherein the extraction means calculates the number of search results to be output based on the confidence column, and extracts the search results to be output from a column of search results in which one or more search results are arranged in order based on the similarity of each detection result, based on the calculated number of search results. (Note 8) A first display criterion, a tolerance for error in the first display criterion, and a second display criterion are set. The extraction means determines the number of search results to be output, The search device according to Appendix 1, which maintains a plurality of candidates as candidates for the number of search results to be output based on the first display criterion and the allowable error range, and determines the number of search results to be output from the plurality of candidates based on the second display criterion. (Note 9) The search device as described in Appendix 1, wherein the criteria displayed are criteria relating to at least one of precision, recall, and F-value (F-measure). (Note 10) The confidence level for each search result is determined based on the similarity score, which indicates the degree to which the query matches one or more search results. From the one or more search results mentioned above, the system outputs the search result whose reliability meets the display criteria. Search methods that include this. (Note 11) A search program for causing a computer to function as a search device, wherein the computer, A calculation process that determines the confidence level for each search result based on one or more search results for a query and the similarity score indicating the degree to which the query is consistent. A search program that performs an extraction process to output search results from the above one or more search results, the search result whose reliability meets the display criteria.

[0115] [Additional Note 2] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0116] (Note 1) A search device that calculates the number of output search results for a query, based on the display criteria, of which one or more search results are available for a query. It comprises at least one processor, and the at least one processor is A calculation process that determines the confidence level for each search result based on one or more search results for a query and the similarity score indicating the degree to which the query is consistent. From the above one or more search results, an extraction process is performed to output search results whose reliability meets the display criteria, A search device that performs this task.

[0117] The search device may also include memory. Furthermore, the memory may store a program that causes at least one processor to perform each of the aforementioned processes. [Explanation of symbols]

[0118] 1-5 ···Search device 11 ···Pre-processing 111...Acquisition part 113...1st calculation section 114...2nd calculation section 115...Third calculation section 116 ···Alignment section 12, 12a, 12b...Calculation section 121...4th calculation section 122...Normalization section 123 ···Granting Section 124 ···Proofreading Department 13, 13a, 13b...extraction part 131... Estimation section 132 ···First Comparison Section 133...Second Comparative Section

Claims

1. A calculation means for determining the confidence level of each search result based on one or more search results for a query and the similarity score indicating the degree to which the query is consistent, An extraction means that outputs a search result from among the one or more search results whose reliability meets the display criteria, A search device equipped with the following features.

2. The extraction means determines the number of search results to output, For each candidate number of search results to output, The estimated number of correct search results included in the candidate for the number of search results to be output, corresponding to the candidate for the number of search results to be output. The estimated number of incorrect search results included in the candidate search results to be output, The estimated number of correct search results included in the one or more search results excluding the candidate search results to be output, and The estimated number of incorrect search results included in the one or more search results excluding the candidate search results to be output is The search device according to claim 1, which calculates based on the confidence level of each of the one or more search results.

3. The search device according to claim 1, wherein the extraction means compares the confidence level and threshold of each of the one or more search results in order to determine the number of search results to be output.

4. The search device according to any one of claims 1 to 3, wherein the calculation means determines the maximum and minimum values ​​of the similarity, and calculates the confidence level from the similarity using the maximum and minimum values ​​of the similarity.

5. The calculation means is A means for assigning labels to at least some of the one or more search results, indicating whether the search results are presumed to be correct based on the search results and the query, A search device according to any one of claims 1 to 3, comprising: calibration means for calibrating a function used to calculate the confidence level from the similarity based on the label.

6. The search device according to claim 5, wherein the labeling means assigns the labels obtained by inputting the search results and the queries into a trained model to at least some of the one or more search results.

7. The calculation means calculates a column of confidence scores by converting each similarity score in the column of similarity scores, in which the similarity scores of the one or more search results are arranged in order based on the similarity scores, The search device according to claim 1, wherein the extraction means calculates the number of search results to be output based on the confidence column, and extracts the search results to be output from a column of search results in which one or more search results are arranged in order based on the similarity of each detection result, based on the calculated number of search results.

8. A first display criterion, a tolerance for error in the first display criterion, and a second display criterion are set. The extraction means determines the number of search results to be output, The search device according to claim 1, which maintains a plurality of candidates as candidates for the number of search results to be output based on the first display criterion and the allowable error range, and determines the number of search results to be output from the plurality of candidates based on the second display criterion.

9. The confidence level for each search result is determined based on the degree of similarity, which indicates the degree to which one or more search results for a query are consistent with the query. From the one or more search results mentioned above, the system outputs the search result whose reliability meets the display criteria. Search methods that include this.

10. A search program for causing a computer to function as a search device, wherein the computer, A calculation process that determines the confidence level for each search result based on one or more search results for a query and the similarity score indicating the degree to which the query is consistent, A search program that performs an extraction process to output search results from the above one or more search results, the search result whose reliability meets the display criteria.

Images