An evaluation dataset construction method and apparatus

Abstract

Embodiments of the present application provide an evaluation dataset construction method and device, and relate to the technical field of Internet. The method comprises: obtaining search statements received by a search engine in a first preset time length, the search statements comprising at least one unit word; querying, from a preset knowledge graph, an expanded unit word corresponding to the at least one unit word, and taking the at least one unit word and the expanded unit word as associated unit words of the search statements; for each associated unit word, searching, from an inverted index library, a target corpus identifier matched by the associated unit word; determining, based on the target corpus identifiers matched by the associated unit words of the search statements and the matching times of each target corpus identifier, a matching level and a matching order value of each target corpus identifier, and constructing an evaluation dataset for the search statements. The embodiments of the present application can efficiently construct a more accurate evaluation dataset.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to an evaluation data set construction method and device. BACKGROUND

[0002] Currently, users can search for the required information by using a search engine. However, if the search accuracy of the search engine is poor, the information returned by the search engine to the user may be information that does not match the search statement. Therefore, it is necessary to evaluate the accuracy of the search engine, and currently the search engine is mainly evaluated by using an evaluation data set. The search statement can be input into the search engine, the search result output by the search engine is obtained, and the search result is compared with the corpus in the evaluation data set of the search statement, so as to evaluate the search accuracy of the search engine.

[0003] Currently, the evaluation data set is constructed manually, which is time-consuming and laborious. In addition, the data covered by the manually constructed evaluation data set is not comprehensive enough, and it is difficult to ensure the accuracy of the evaluation data set. The accuracy of the evaluation data set directly affects the accuracy of the evaluation result of the evaluation of the search engine. Therefore, the efficiency of constructing the evaluation data set is low, and the accuracy of the constructed evaluation data set is low. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide an evaluation data set construction method and device to realize efficient construction of more accurate evaluation data sets. The specific technical solutions are as follows:

[0005] In a first aspect, the embodiments of the present application provide an evaluation data set construction method, which comprises:

[0006] obtaining search statements received by a search engine within a first preset time period, wherein the search statements comprise at least one unit word;

[0007] querying, from a preset knowledge graph, an expanded unit word corresponding to the at least one unit word, and taking the at least one unit word and the expanded unit word as associated unit words of the search statement;

[0008] For each associated unit word, searching, from an inverted index library, a target corpus identifier matched by the associated unit word, wherein the inverted index library is used to store a set of corpus identifiers corresponding to each word in a preset corpus, and the preset corpus comprises corpus searched by the search engine;

[0009] determining, based on the target corpus identifiers matched by the associated unit words of the search statement and the matching times of each target corpus identifier, a matching level and a matching order value of each target corpus identifier;

[0010] The matching level and the matching order value of each target corpus identified by the search statement are constructed as an evaluation dataset of the search statement.

[0011] In a possible implementation, the matching level and the matching order value of each target corpus are determined based on the target corpus identified by each associated unit word of the search statement and the number of matches of each target corpus, and the method comprises the following steps of:

[0012] For each target corpus, a matching level is determined based on the number of matches to which the number of matches of the target corpus belongs and a preset correspondence relationship between the number of matches and the matching level.

[0013] In order of decreasing matching level, the matching order value of each target corpus of each matching level is set to be increasingly larger, and the matching order values of target corpora of the same matching level are the same.

[0014] In a possible implementation, the preset corpus is updated in real time, and the method further comprises the following steps of:

[0015] Every second preset time length, the inverted index library is updated based on the updated preset corpus.

[0016] A target word is obtained from a target corpus identification set that has been updated in the inverted index library.

[0017] From the search statements for which the evaluation dataset has been constructed, a target search statement including the target word is obtained, and an evaluation dataset of the target search statement is reconstructed based on the updated inverted index library.

[0018] In a possible implementation, after the inverted index library is updated based on the updated preset corpus, the method further comprises the following steps of:

[0019] A plurality of new search statements in the search engine within the second preset time length are obtained.

[0020] From the plurality of new search statements, a search statement to be evaluated is selected, and the search statement to be evaluated includes at least one unit word to be evaluated.

[0021] From the preset knowledge graph, an expansion unit word to be evaluated corresponding to the at least one unit word to be evaluated is queried, and the at least one unit word to be evaluated and the expansion unit word to be evaluated are used as associated unit words to be evaluated of the search statement to be evaluated.

[0022] For each associated unit word to be evaluated, a target corpus identification matched by the associated unit word to be evaluated is searched from the updated inverted index library.

[0023] determine a matching level and a matching order value of each to-be-evaluated corpus identifier based on the to-be-evaluated corpus identifier matched by each unit word of the to-be-evaluated unit and the number of matches of each to-be-evaluated corpus identifier;

[0024] construct the matching level and the matching order value of each to-be-evaluated corpus identifier corresponding to the to-be-evaluated search statement as an evaluation data set of the to-be-evaluated search statement.

[0025] In a possible implementation, the screening of the to-be-evaluated search statement from the plurality of new search statements includes:

[0026] randomly screening a preset number of new search statements that are not constructed into an evaluation data set from the plurality of new search statements, and taking the screened new search statements as the to-be-evaluated search statement.

[0027] In a second aspect, an embodiment of the present application provides an evaluation data set construction device, and the device includes:

[0028] an acquisition module configured to acquire search statements received by a search engine within a first preset time length, the search statements including at least one unit word;

[0029] a query module configured to query, from a preset knowledge graph, extended unit words corresponding to the at least one unit word, and take the at least one unit word and the extended unit words as associated unit words of the search statements;

[0030] The query module is further configured to search, for each associated unit word, a target corpus identifier matched by the associated unit word from an inverted index library, and the inverted index library is configured to store a set of corpus identifiers corresponding to each word in a preset corpus, and the preset corpus includes corpus searched by the search engine.

[0031] a determination module configured to determine a matching level and a matching order value of each target corpus identifier based on the target corpus identifier matched by each associated unit word of the search statement and the number of matches of each target corpus identifier;

[0032] a construction module configured to construct the matching level and the matching order value of each target corpus identifier corresponding to the search statement as an evaluation data set of the search statement.

[0033] In a possible implementation, the determination module is specifically configured to:

[0034] determine, for each target corpus identifier, a matching level based on a matching number interval to which the number of matches of the target corpus identifier belongs and a preset corresponding relationship between the matching number interval and the matching level;

[0035] The target corpus identifiers of each matching level are sequentially assigned with matching order values in ascending order according to the matching levels from high to low, and the matching order values of the target corpus identifiers with the same matching level are the same.

[0036] In a possible implementation, the preset corpus is updated in real time, and the device further includes:

[0037] The updating module is configured to update the inverted index library based on the updated preset corpus every second preset time length.

[0038] The query module is further configured to obtain target words corresponding to a target corpus identifier set that has been updated in the inverted index library.

[0039] The construction module is further configured to obtain a target search statement including the target words from search statements that have been constructed with the evaluation data set, and reconstruct the evaluation data set of the target search statement based on the updated inverted index library.

[0040] In a possible implementation, the obtaining module is specifically configured to obtain a plurality of newly added search statements in the search engine within the second preset time length.

[0041] The obtaining module is further configured to filter a search statement to be evaluated from the plurality of newly added search statements, and the search statement to be evaluated includes at least one unit word to be evaluated.

[0042] The query module is further configured to query, from the preset knowledge graph, an expansion unit word to be evaluated corresponding to the at least one unit word to be evaluated, and take the at least one unit word to be evaluated and the expansion unit word to be evaluated as an association unit word to be evaluated of the search statement to be evaluated.

[0043] The query module is further configured to search, for each association unit word to be evaluated, a corpus identifier to be evaluated matched with the association unit word to be evaluated from the updated inverted index library.

[0044] The determination module is further configured to determine, based on the corpus identifier to be evaluated matched with each unit word to be evaluated and the number of times of matching of each corpus identifier to be evaluated, a matching level and a matching order value of each corpus identifier to be evaluated.

[0045] The construction module is further configured to construct the matching level and the matching order value of each corpus identifier to be evaluated corresponding to the search statement to be evaluated as an evaluation data set of the search statement to be evaluated.

[0046] In a possible implementation, the obtaining module is specifically configured to:

[0047] From the plurality of new search statements, a preset number of new search statements not constructed by the evaluation data set are randomly selected, and the selected new search statements are used as the search statements to be evaluated.

[0048] In a third aspect, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0049] The memory is configured to store a computer program.

[0050] The processor is configured to execute the program stored in the memory, and implement the method of the first aspect.

[0051] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the method of the first aspect is implemented.

[0052] In a fifth aspect, a computer program product is provided, which includes executable instructions. When the executable instructions are executed on a computer, the method of the first aspect is implemented.

[0053] With the above technical solution, the search statements received by the search engine within the first preset time period can be obtained, the search statements include at least one unit word, and the unit word is expanded based on a preset knowledge graph to obtain a plurality of associated unit words, so that the subsequent search can search the corpus identifier corresponding to the expanded unit word that has semantic association with the search statement from the inverted index library. After searching the corpus identifier corresponding to each associated unit word, the matching level and the matching order value of each target corpus identifier can be determined based on the target corpus identifier matched by each associated unit word and the matching number of each target corpus identifier, and then the matching level and the matching order value of each target corpus identifier corresponding to the search statement are constructed as the evaluation data set of the search statement. Compared with the manual construction of the evaluation data set, the efficiency of constructing the evaluation data set is improved. Moreover, the evaluation data set constructed by the embodiments of the present application involves the corpus searched by the search engine, that is, the constructed evaluation data set contains more abundant corpus, so that the evaluation data set is more abundant and accurate. It can be seen that the embodiments of the present application can efficiently and accurately construct the evaluation data set.

[0054] Of course, implementing any product or method of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0056] Figure 1 A flowchart of an evaluation dataset construction method provided by an embodiment of the present application is shown in the figure.

[0057] Figure 2 A flowchart of a search engine evaluated by an evaluation dataset based on a search sentence provided by an embodiment of the present application is shown in the figure.

[0058] Figure 3 A structural diagram of an evaluation dataset construction device provided by an embodiment of the present application is shown in the figure.

[0059] Figure 4 A structural diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0061] An evaluation dataset construction method is provided by an embodiment of the present application. The method is applied to an electronic device, which can be a terminal, a server, or the like. As shown in the figure, the method comprises the following steps. Figure 1

[0062] S101, obtaining search sentences received by a search engine in a first preset time period, wherein the search sentences comprise at least one unit word.

[0063] The search sentences can be a question, a phrase, or a word input by a user into the search engine. The first preset time period can be 5 hours, 12 hours, 24 hours, or the like, which can be set according to actual needs. For example, after upgrading the search engine, the search sentences received by the search engine in one day can be obtained to evaluate the search performance of the upgraded search engine.

[0064] After obtaining the search sentences, the search sentences can be processed by word segmentation respectively to obtain at least one unit word included in each search sentence.

[0065] ​Wherein, after the word segmentation processing of the search sentence, the unit words are obtained, and the unit words representing stop words such as "of", "get" and the like are removed from the obtained unit words.

[0066] It can be understood that if the search sentence is a question, a plurality of unit words can be obtained after the word segmentation processing of the search sentence, and if the search sentence is a word, one unit word can be obtained after the word segmentation processing of the search sentence.

[0067] S102, query the at least one unit word corresponding to the expansion unit word from the preset knowledge graph, and take the at least one unit word and the expansion unit word as the associated unit word of the search sentence.

[0068] Wherein, the preset knowledge graph is a kind of graph data structure. The node in the preset knowledge graph is a unit word, and the unit words can be connected by different connection modes, and different connection modes represent different semantic relationships, such as error correction relationship, approximate relationship, containing relationship and the like. If two unit words are not connected to each other, it indicates that there is no semantic relationship between the two unit words.

[0069] The preset knowledge graph can be constructed based on structured data, semi-structured data and unstructured data. The structured data can be obtained from the corpus summarized by artificial, and the unstructured data can be obtained from the full corpus that can be searched by the search engine.

[0070] It can be understood that for each unit word, the expansion unit word corresponding to the unit word queried from the preset knowledge graph is a unit word having a semantic relationship with the unit word, for example, a unit word having an approximate relationship with the unit word, or a unit word having a containing relationship with the unit word.

[0071] For example, after the word segmentation processing of the search sentence, unit word A and unit word B are obtained, in the preset knowledge graph, unit word A and unit word M are connected to each other, and unit word B and unit word O and unit word N are connected to each other. That is, the expansion unit word corresponding to unit word A can be queried in the preset knowledge graph: unit word M; the expansion unit word corresponding to unit word B: unit word O and unit word N. Further, unit word A, unit word B, unit word M, unit word N and unit word O can be taken as the associated unit word of the search sentence.

[0072] S103, for each associated unit word, search the target corpus identifier matched with the associated unit word from the inverted index library.

[0073] The inverted index library is configured to store a set of corpus identifiers corresponding to each word in a preset corpus. The preset corpus includes corpus searched by a search engine. A basic unit of each corpus is a document, for example, a document included in an Internet web page, an article, or an answer to a question, etc. The preset corpus can be a corpus in a full corpus library of the search engine, and the full corpus library includes full corpus searchable by the search engine.

[0074] The set of corpus identifiers includes at least one corpus identifier, which is used to identify a corpus searchable by the search engine and can be used as an index of the corpus. The corpus identifier can be a string of numbers, a string of letters, etc. For example, the corpus identifier of document 1 can be 201906100002, and the corpus identifier of document 2 can be 201908190005. The corpus identifier is used to mark the corpus in the embodiment of the present application, which can save the storage space occupied by the inverted index library.

[0075] For each association unit word obtained in S102, the association unit word can be searched in the inverted index library, and then a set of corpus identifiers corresponding to the association unit word is searched, and each corpus identifier in the set of corpus identifiers is obtained. Each corpus identifier in the set of corpus identifiers is a target corpus identifier matched by the association unit word.

[0076] It should be noted that each corpus identifier in the set of corpus identifiers corresponding to a word identifies a corpus including the word.

[0077] The inverted index library can be a hash table library with a word as a key and a set of corpus identifiers as a value.

[0078] As an example, the correspondence between the word and the corpus identifier is shown in Table 1.

[0079] Table 1

[0080] Words in the full corpus Corpus identification set Word 1 ["201909040002”，"202002280005”] Word 2 ["201902210001”] Word 3 ["201908190006”，"201906100002”]

[0081] In Table 1, the set of corpus identifiers corresponding to word 1 is [“201909040002”, “202002280005”], the set of corpus identifiers corresponding to word 2 is [“201902210001”], and the set of corpus identifiers corresponding to word 3 is [“201908190006”, “201906100002”].

[0082] The corpus identified by "201909040002" and the corpus identified by "202002280005" both include word 1, the corpus identified by "201902210001" includes word 2, and the corpus identified by "201908190006" and the corpus identified by "201906100002" both include word 3.

[0083] S104, based on the target corpus identification matched by each associated unit word of the search statement and the matching number of each target corpus identification, determining the matching level and matching order value of each target corpus identification.

[0084] S105, constructing the matching level and matching order value of each target corpus identification corresponding to the search statement as the evaluation data set of the search statement.

[0085] The evaluation data set includes each target corpus identification corresponding to the search statement, the matching level and the matching order value of each corpus identification. After the evaluation data set of each search statement is constructed, the search statement and the evaluation data set of the search statement can be added to the evaluation corpus library, and then according to the actual demand, for example, when the search engine is upgraded and evaluated, the search statement and the evaluation data set of the search statement can be obtained from the evaluation corpus library to evaluate the search engine.

[0086] By adopting the above technical solution, the search statement received by the search engine within the first preset time period can be obtained, the search statement includes at least one unit word, and the unit word is expanded based on the preset knowledge graph to obtain a plurality of associated unit words, so that the subsequent search can search the corpus identification corresponding to the expanded unit word having semantic association with the search statement from the inverted index library. After each associated unit word corresponding corpus identification is searched, the matching level and matching order value of each target corpus identification can be determined based on the target corpus identification matched by each associated unit word and the matching number of each target corpus identification, and then the matching level and matching order value of each target corpus identification corresponding to the search statement can be constructed as the evaluation data set of the search statement. Compared with the manual construction of the evaluation data set, the efficiency of constructing the evaluation data set is improved. Moreover, the evaluation data set constructed by the embodiment of the present application involves the corpus searched by the search engine, that is, the evaluation data set constructed contains more abundant corpus, so that the evaluation data set is more abundant and accurate. It can be seen that the embodiment of the present application can efficiently and accurately construct the evaluation data set.

[0087] In an embodiment of the present application, based on the target corpus identification matched by each associated unit word of the search statement and the matching number of each target corpus identification, the matching level and matching order value of each target corpus identification are determined, which can be specifically implemented as:

[0088] Step A, for each target corpus identifier, based on the match number interval to which the number of matches of the target corpus identifier belongs and the preset correspondence between the match number interval and the match level, the match level is determined.

[0089] Wherein, in the inverted index library, different association unit words may correspond to the same corpus identifier, so the same target corpus identifier may be searched in the inverted index library multiple times.

[0090] As an example, the association unit words of the search statement include: association unit word A, association unit word B and association unit word C. First, for association unit word A, search the first target corpus identifier and the second target corpus identifier matched by association unit word A from the inverted index library, then record the match number of the first target corpus identifier as 1, and record the match number of the second target corpus identifier as 1; for association unit word B, search the first target corpus identifier and the third target corpus identifier matched by association unit word B from the inverted index library, then add 1 to the match number of the first target corpus identifier, and record the match number of the third target corpus identifier as 1; for association unit word C, search the first target corpus identifier and the second target corpus identifier matched by association unit word C from the inverted index library, then add 1 to the match number of the first target corpus identifier, and add 1 to the match number of the second target corpus identifier. Therefore, it can be obtained that the match number of the first target corpus identifier is 3, the match number of the second target corpus identifier is 2, and the match number of the third target corpus identifier is 1.

[0091] It can be understood that the more the match number of a target corpus identifier is, the more the number of association unit words of the search statement included in the corpus identified by the target corpus identifier is, and thus it can be indicated that the match level of the corpus identified by the target corpus identifier and the search statement is higher. Conversely, the less the match number of a target corpus identifier is, the less the number of association unit words of the search statement included in the corpus identified by the target corpus identifier is, and thus it can be indicated that the match level of the corpus identified by the target corpus identifier and the search statement is lower.

[0092] Wherein, each match level corresponds to a match number interval, and the match number interval corresponding to each match level can be predefined. As an example, it can be specified that the match number interval of 1-5 corresponds to the match level of "general", the match number interval of 6-10 corresponds to the match level of "good", and the match number interval of 11-15 corresponds to the match level of "excellent", wherein the highest match level is "excellent", the medium match level is "good", and the lowest match level is "general". It should be noted that only three match levels are shown in this example, and in actual implementation, the number of match levels can be two or four, and the number of match levels is not limited by the embodiments of the present application.

[0093] Through the above examples, if it is determined that the matching times of the target corpus identifier is 4, since the matching times 4 belongs to the matching times interval 1-5, the matching level of the target corpus identifier is general; if it is determined that the matching times of the target corpus identifier is 12, since the matching times 12 belongs to the matching times interval 11-15, the matching level of the target corpus identifier is excellent.

[0094] Step B, in order of high to low of the matching level, set the matching order value of each matching level target corpus identifier to be increased in turn, and the matching order value of the target corpus identifiers with the same matching level is the same.

[0095] Wherein, the higher the matching level of a target corpus identifier is, the smaller the matching order value set for the target corpus identifier is.

[0096] For example, if a total of 5 target corpus identifiers are obtained, wherein the matching level of the first target corpus identifier and the second target corpus identifier is excellent, the matching level of the third target corpus identifier and the fourth target corpus identifier is good, and the matching level of the fifth target corpus identifier is general. According to the order from high to low of the matching level, the matching order value of the first target corpus identifier and the second target corpus identifier with the highest matching level is set to 1 first, then the matching order value of the third target corpus identifier and the fourth target corpus identifier with the moderate matching level is set to 2, and finally the matching order value of the fifth target corpus identifier with the lowest matching level is set to 3.

[0097] Using the above technical solution, since a search engine with higher search accuracy will preferentially output search results with higher matching levels with search statements, when the evaluation data set constructed by the embodiment of the application is used to evaluate the accuracy of the search engine, it can be judged whether the search engine preferentially outputs search results with higher matching levels with search statements based on the matching level and the matching order value of each target corpus identifier and each corpus identifier, that is, the evaluation data set constructed by the embodiment of the application can accurately evaluate the accuracy of the search engine.

[0098] In an embodiment of the application, the preset corpus is updated in real time, and the preset corpus needs to be involved when the evaluation data set is constructed by the embodiment of the application. Therefore, in order to ensure the accuracy of the evaluation data set used for evaluating the search engine, the evaluation data set of the search statement can be updated in time, and the updating steps of the evaluation data set of the search statement are as follows:

[0099] Step 1, update the inverted index library based on the preset corpus every second preset time length.

[0100] The second preset time length can be set according to actual needs. For example, if the evaluation data set of the search statement is updated every half day, the second preset time length can be set to 12 hours, and if the evaluation data set of the search statement is updated every day, the second preset time length can be set to 24 hours.

[0101] The corpus identifiers in the corpus identifier set in the inverted index library are used to identify the corpus included in the preset corpus, so when the full preset corpus is updated, the corpus identifier set corresponding to each word in the inverted index library will also change, and therefore the inverted index library needs to be updated.

[0102] As an example, taking the preset corpus as the corpus in the full corpus library, whenever the corpus in the full corpus library is updated, the update type of the updated corpus and the timestamp at the time of update are recorded. The update type includes: an added type, a deleted type and a modified type. The added type indicates that the updated corpus is a corpus newly added to the full corpus library, the deleted type indicates that the updated corpus is a corpus that has been deleted from the full corpus library, and the modified type indicates that the updated corpus is a corpus that has been modified.

[0103] The timestamp of the updated corpus can be used to determine whether the updated corpus is updated within the second preset time, and if so, the inverted index library can be updated based on the updated corpus. The implementation of updating the inverted index library based on the updated corpus is as follows:

[0104] If the update type of the updated corpus is the added type, the updated corpus is subjected to word segmentation processing to obtain at least one unit word included in the updated corpus. For each unit word included in the updated corpus, if the inverted index library includes a word identical to the unit word, the corpus identifier of the updated corpus is added to the corpus identifier set corresponding to the word identical to the unit word. If the inverted index library does not include a word identical to the unit word, the unit word is added to the inverted index library, and a corpus identifier set corresponding to the unit word is created in the inverted index library, and the corpus identifier of the updated corpus is added to the newly created corpus identifier set.

[0105] If the update type of the updated corpus is the deleted type, the corpus identifier of the updated corpus is searched for and deleted from the corpus identifier set corresponding to each word in the inverted index library. If the corpus identifier set corresponding to a word only includes the corpus identifier of the updated corpus, the word and the corpus identifier set corresponding to the word are deleted from the inverted index library.

[0106] If the update type of the updated corpus is the modification type, the corpus identifier of the updated corpus is searched and deleted from the corpus identifier set corresponding to each term in the inverted index library. If the corpus identifier set corresponding to a term only includes the corpus identifier of the updated corpus, the term and the corpus identifier set corresponding to the term are deleted from the inverted index library. Then, the updated corpus is processed to obtain at least one unit word included in the updated corpus. For each unit word included in the updated corpus, if the inverted index library includes a term identical to the unit word, the corpus identifier of the updated corpus is added to the corpus identifier set corresponding to the term identical to the unit word. If the inverted index library does not include a term identical to the unit word, the unit word is added to the inverted index library, a corpus identifier set corresponding to the unit word is created in the inverted index library, and the corpus identifier of the updated corpus is added to the newly created corpus identifier set.

[0107] Step 2, obtaining a target term corresponding to a target corpus identifier set that has been updated in the inverted index library.

[0108] Step 3, obtaining a target search statement including the target term from search statements whose evaluation data sets have been constructed, and reconstructing the evaluation data set of the target search statement based on the updated inverted index library.

[0109] After the inverted index library is updated, in the full corpus identifier set in the inverted index library, only the corpus identifier in the target corpus identifier set corresponding to the target term has been changed, and the corpus identifiers in the corpus identifier sets corresponding to other terms have not been changed, that is, only the evaluation accuracy of the evaluation data set of the target search statement including the target term has been deteriorated, and the evaluation accuracy of the evaluation data sets of other search statements is still high. The embodiment of the present application does not reconstruct the evaluation data sets of all search statements, but only reconstructs the evaluation data set of the target search statement, thereby saving the time consumption of reconstructing the evaluation data set.

[0110] After obtaining the target search sentence, the steps of reconstructing the evaluation data set of the target search sentence based on the updated inverted index library are similar to S102-S105. First, the target search sentence is segmented to obtain at least one target unit word included in the target search sentence. For each target unit word, a target expansion unit word corresponding to the target unit word is queried in the preset knowledge graph, and the at least one target unit word and the target expansion unit word are taken as target associated unit words of the target search sentence. For each target associated unit word, the updated corpus identifier matched with the target associated unit word is searched from the updated inverted index library. Based on the updated corpus identifier matched with each target associated unit word and the matching number of each updated corpus identifier, the matching level and the matching order value of each updated corpus identifier are determined. The matching level and the matching order value of each updated corpus identifier corresponding to the target search sentence are constructed as the evaluation data set of the target search sentence, and the reconstruction of the evaluation data set of the target search sentence is completed.

[0111] By adopting the technical solution, the updated inverted index library is obtained based on the updated preset corpus, that is, the updated inverted index library includes the full corpus in the updated preset corpus, and thus the evaluation data set with more accurate evaluation results can be constructed based on the updated inverted index library.

[0112] In an embodiment of the present application, after updating the inverted index library based on the updated full corpus library, the newly added search sentences in the search engine within a second preset time period can be obtained. The to-be-evaluated search sentences are selected from the plurality of newly added search sentences, and the to-be-evaluated search sentences include at least one to-be-evaluated unit word.

[0113] Then, the evaluation data set of the to-be-evaluated search sentence can be constructed based on the updated inverted index library and the preset knowledge graph.

[0114] The method of constructing the evaluation data set of the to-be-evaluated search sentence is as follows: the to-be-evaluated expansion unit word corresponding to the at least one to-be-evaluated unit word is queried from the preset knowledge graph, and the at least one to-be-evaluated unit word and the to-be-evaluated expansion unit word are taken as to-be-evaluated associated unit words of the to-be-evaluated search sentence. Then, for each to-be-evaluated associated unit word, the to-be-evaluated corpus identifier matched with the to-be-evaluated associated unit word is searched from the updated inverted index library. Based on the to-be-evaluated corpus identifier matched with each to-be-evaluated unit word and the matching number of each to-be-evaluated corpus identifier, the matching level and the matching order value of each to-be-evaluated corpus identifier are determined. The matching level and the matching order value of each to-be-evaluated corpus identifier corresponding to the to-be-evaluated search sentence are constructed as the evaluation data set of the to-be-evaluated search sentence.

[0115] The method of constructing the evaluation data set of the to-be-evaluated search sentence is the same as the method of constructing the evaluation data set of the search sentence, and the related description in the above embodiments can be referred to.

[0116] It can be understood that, within the second preset time length, the user will input new search statements into the search engine. For example, if it is stipulated to rebuild the evaluation dataset every 48 hours, within the 48 hours, the user will input multiple new search statements into the search engine. For the new search statements in the search engine, a preset number of to-be-evaluated search statements need to be selected from the new search statements, the preset number is less than or equal to the total number of the new search statements, and then the evaluation dataset is built for the to-be-evaluated search statements, so that repeated construction of the evaluation dataset can be avoided.

[0117] In an embodiment of the present application, among the multiple new search statements, the evaluation dataset has been built for some new search statements, and the evaluation dataset of other new search statements has not been built. Therefore, from the multiple new search statements, a preset number of new search statements for which the evaluation dataset has not been built are randomly selected, and the selected new search statements are used as the to-be-evaluated search statements.

[0118] For example, each time the user inputs a search statement into the search engine, the search engine can record the search statement and the time stamp when the search statement is input. Then, according to the time stamp of each search statement, the new search statements in the search engine within the second preset time length can be obtained. The electronic device can randomly select one new search statement from the multiple new search statements in the search engine. If the new search statement does not exist in the evaluation corpus, it means that the evaluation dataset for the new search statement has not been created, and then the new search statement is used as the to-be-evaluated search statement. If the new search statement exists in the evaluation corpus, it means that the evaluation dataset for the new search statement has been created, and then a new new search statement is randomly selected from the multiple new search statements in the search engine, and it is determined whether the newly selected new search statement exists in the evaluation corpus. The above process is repeated until a preset number of to-be-evaluated search statements are obtained, and then the evaluation dataset is created for the obtained to-be-evaluated search statements based on the updated inverted index library and the preset knowledge graph.

[0119] In addition, the preset knowledge graph will be updated from time to time. After the preset knowledge graph is updated, the expansion unit word corresponding to each unit word included in each search statement will change, that is, the associated unit word of each search statement will also change. Therefore, in order to build a more accurate evaluation dataset, after the preset knowledge graph is updated, the evaluation corpus of each search statement can be updated based on the updated preset knowledge graph.

[0120] The steps of updating the evaluation corpus of the search statement based on the updated preset knowledge graph are as follows:

[0121] First, the search statement is segmented, and at least one unit word included in the search statement is obtained. For each unit word, an updated and expanded unit word corresponding to the unit word is queried in the updated preset knowledge graph, and the at least one unit word and the updated and expanded unit word are taken as updated associated unit words of the search statement. For each updated associated unit word, an updated corpus identifier matched with the updated associated unit word is searched from the inverted index library, and based on the updated corpus identifiers matched with the respective updated associated unit words, an evaluation data set of the search statement is constructed, and the evaluation data set of the search statement is updated.

[0122] As shown in Figure 2 , a flowchart for evaluating a search engine based on an evaluation data set of a search statement provided by an embodiment of the present application can include the following steps:

[0123] S201, collect corpus from a full corpus library.

[0124] S202, segment the collected corpus.

[0125] For example, first, the corpus is segmented to obtain a plurality of words, and then the words representing stop words are removed from the obtained plurality of words.

[0126] S203, construct an inverted index library.

[0127] The inverted index library can be a hash word table library with words as keys and corpus identifier sets as values. The corpus identifier set includes at least one corpus identifier, and the corpus identifier is used to mark the corpus in the full corpus library.

[0128] S204, obtain a search statement received by a search engine.

[0129] S205, segment the search statement to obtain at least one unit word included in the search statement.

[0130] After obtaining the search statement and segmenting the search statement to obtain the unit words, the unit words representing stop words can be removed from the obtained unit words.

[0131] For example, the obtained search statement is "12508f-af main control board abnormal restart problem", and after segmenting the search statement, the obtained unit words can be stored in the form of a list, such as ["12508f-af", "main control board", "abnormal restart"].

[0132] S206, query at least one unit word corresponding to an expanded unit word from a preset knowledge graph, and take at least one unit word and the expanded unit word as associated unit words of the search statement.

[0133] wherein the queried extended unit word has a semantic relationship with the unit word.

[0134] For example, for the unit word "12508f-af", the extended unit words "s12508f-af", "s12508f", "12508f" and "s12508" can be queried in the preset knowledge graph, and for the unit word "abnormal restart", the extended unit word "power-off restart" can be queried in the preset knowledge graph, and then the obtained associated unit words ["12508f-af", "main control board", "abnormal restart", "s12508f-af", "s12508f", "12508f", "s12508", "power-off restart"] can be stored in the form of a list.

[0135] S207, for each associated unit word, searching a target corpus identifier matching the associated unit word from the inverted index library.

[0136] S208, for each target corpus identifier, determining a matching level and a matching order value of the target corpus identifier.

[0137] For example, it can be specified that the matching frequency interval of 1-5 corresponds to the matching level of "general", the matching frequency interval of 6-10 corresponds to the matching level of "good", and the matching frequency interval of 11-15 corresponds to the matching level of "excellent". Then, according to the order from high to low of the matching level, the target corpus identifiers of each matching level are set to have increasing matching order values, the matching order value of the matching level "excellent" is set to 1, the matching order value of the matching level "good" is set to 2, and the matching order value of the matching level "general" is set to 3.

[0138] S209, based on the target corpus identifiers, the matching levels and the matching order values of the corpus identifiers, constructing an evaluation data set of the search statement.

[0139] In an implementation manner, the evaluation data set of the search statement can be constructed by a Discounted Cumulative Gain (DCG) method.

[0140] In the DCG method, the matching level of a target corpus identifier corresponds to a rel value, and the matching order value of the target corpus identifier corresponds to an i value, for example, the matching level "excellent" corresponds to the rel value of 3 and the i value of 1; the matching level "good" corresponds to the rel value of 2 and the i value of 2; and the matching level "general" corresponds to the rel value of 1 and the i value of 3.

[0141] As an example, the constructed evaluation data set can be: [

[0143] ["201909040002", "202002280005"]: [rel=3, i=1],

[0144] ["201902140001", "201906100001", "201912260003"]: [rel=2, i=2],

[0145] ["201908190006", "202002280004", "201911260001", "201902210001"]: [rel=1, i=3]

[0146] ].

[0147] Wherein, the matching level of the corpus identified as "201909040002" and "202002280005" with the search statement is "excellent", the matching level of the corpus identified as "201902140001", "201906100001" and "201912260003" with the search statement is "good", and the matching level of the corpus identified as "201908190006", "202002280004", "201911260001" and "201902210001" with the search statement is "general".

[0148] The above S201-S209 is the construction process of the evaluation data set. After the evaluation data set is constructed, the search engine can be evaluated based on the evaluation data set every certain period of time, such as 1 day, 3 days, etc., or the search engine after upgrading can be evaluated based on the evaluation data set. When the search engine is evaluated, at least one evaluation data set can be randomly selected from the evaluation corpus to evaluate the search engine. The evaluation process of the search engine includes the following S210-S214.

[0149] S210, obtaining a search statement, and calculating the ideal DCG value of the search engine based on the evaluation data set of the search statement.

[0150] For example, the search statement is "12508f-af master control board abnormal restart problem", and the evaluation data set of the search statement is: [

[0152] ["201909040002", "202002280005"]: [rel=3, i=1],

[0153] ["201902140001", "201906100001", "201912260003"]: [rel=2, i=2],

[0154] ["201908190006", "202002280004", "201911260001", "201902210001"]: [rel=1, i=3]

[0155] ].

[0156] The ideal DCG value can be calculated according to the following formula:

[0157]

[0158] wherein DCG1 is the ideal DCG value, p is the total number of target corpus identifiers in the evaluation data set of the search statement, i j is the i value corresponding to the jth target corpus identifier in the evaluation data set, rel j is the rel value corresponding to the jth target corpus identifier in the evaluation data set, for example, the total number of target corpus identifiers in the evaluation data set is 9, the first corpus identifier in the evaluation data set is "201909040002", that is, the rel1 value is 3, and the i1 value is 1.

[0159] S211, input the search statement into the search engine to obtain a preset number of ordered arranged output results output by the search engine.

[0160] For example, input the search statement "12508f-af master control board abnormal restart problem" into the search engine, and select four ordered output results: the first result, the second result, the third result and the fourth result from a large number of output results output by the search engine. Among them, the first result is the first output by the search engine, and the fourth result is the last output by the search engine.

[0161] S212, for each output result output by the search engine, determine the arrangement order of the output result, and determine the matching level of the output result and the search statement based on the evaluation data set of the search statement.

[0162] Among them, for each output result output by the search engine, the corpus identifier of the output result is determined, and then the rel value corresponding to the output result in the corpus evaluation set is determined based on the corpus identifier of the output result.

[0163] For example, the ranking order of the first output result is 1, the corpus identifier is "201909040002", and based on the evaluation data set of the search statement, it is determined that the rel corresponding to the first output result is 3; the ranking order of the second output result is 2, the corpus identifier is "202002280005", and based on the evaluation data set of the search statement, it is determined that the rel corresponding to the second output result is 3; the ranking order of the third output result is 3, the corpus identifier is "201908190006", and based on the evaluation data set of the search statement, it is determined that the rel corresponding to the third output result is 1; the ranking order of the fourth output result is 4, the corpus identifier is "201906100001", and based on the evaluation data set of the search statement, it is determined that the rel corresponding to the fourth output result is 2.

[0164] S213, calculate the actual DCG value of the search engine.

[0165] wherein the actual DCG value can be calculated according to the following formula:

[0166]

[0167] wherein DCG2 is the actual DCG value, q is the total number of obtained output results, k j is the ranking order of the gth output result, rel g is the rel value corresponding to the corpus identifier of the gth output result in the evaluation data set. For example, the total number of obtained output results is 4, the ranking order of the first output result is 1, and the rel value corresponding to the corpus identifier of the first output result is 1, that is, k1 is 1 and rel1 is 3.

[0168] S214, calculate the nDCG value based on the ideal DCG value and the actual DCG value.

[0169] In an implementation manner, the actual DCG value can be divided by the ideal DCG value to obtain a normalize discounted cumulative gain (nDCG) value. The closer the nDCG value is to 1, the closer the actual DCG value of the search engine is to the actual DCG value, and thus it can be indicated that the search accuracy of the search engine is higher. The farther the nDCG value is from 1, the farther the actual DCG value of the search engine is from the actual DCG value, and thus it can be indicated that the search accuracy of the search engine is lower.

[0170] It can be seen from the calculation formula of the DCG value that when the actual DCG value of the search engine is calculated, not only the matching degree of the output result of the search engine and the search statement is considered, but also the output order of the search engine is considered. The search engine outputs the output result with a higher matching degree with the search statement more preferentially, which indicates that the search accuracy of the search engine is higher. Therefore, the DCG evaluation method can make the evaluation result of the search engine more accurate.

[0171] It can be understood that, Figure 2 The process of evaluating the search engine based on the evaluation data set of the search statement is introduced. In actual implementation, the process shown in Figure 2 may be executed for multiple search statements, and then multiple nDCG values are calculated. The search engine is evaluated based on the average value of the multiple nDCG values. In this way, the search engine can be accurately evaluated.

[0172] Based on the same inventive concept, the embodiments of the present application also provide an evaluation data set construction device, as shown in Figure 3 The device comprises:

[0173] The acquisition module 301 is configured to acquire search statements received by the search engine within a first preset time period. The search statements comprise at least one unit word.

[0174] The query module 302 is configured to query, from a preset knowledge graph, an expanded unit word corresponding to the at least one unit word, and to take the at least one unit word and the expanded unit word as associated unit words of the search statement.

[0175] The query module 302 is further configured to search, for each associated unit word, a target corpus identifier matched by the associated unit word from an inverted index library. The inverted index library is configured to store a set of corpus identifiers corresponding to each word in a preset corpus. The preset corpus comprises corpus searched by the search engine.

[0176] The determination module 303 is configured to determine a matching level and a matching order value of each target corpus identifier based on the target corpus identifiers matched by the associated unit words of the search statement and the matching times of each target corpus identifier.

[0177] The construction module 304 is configured to construct the matching level and the matching order value of each target corpus identifier corresponding to the search statement into an evaluation data set of the search statement.

[0178] Optionally, the determination module 303 is specifically configured to:

[0179] For each target corpus identifier, the matching level is determined based on a matching time interval to which the matching times of the target corpus identifier belong and a preset corresponding relationship between the matching time interval and the matching level.

[0180] In order from high to low in the matching level, a matching order value is set for each target corpus identifier in the matching level in turn, and the matching order values of the target corpus identifiers with the same matching level are the same.

[0181] Optionally, the preset corpus is updated in real time, and the device further comprises:

[0182] The updating module is configured to update the inverted index library based on the updated preset corpus every second preset time length.

[0183] The query module 302 is further configured to obtain a target word from the target corpus identifier set that has been updated in the inverted index library.

[0184] The construction module 304 is further configured to obtain a target search statement including the target word from the search statements that have been constructed into the evaluation data set, and reconstruct the evaluation data set of the target search statement based on the updated inverted index library.

[0185] Optionally, the obtaining module 301 is further configured to obtain a plurality of new search statements in the search engine within a second preset time length.

[0186] The obtaining module 301 is further configured to select a preset number of evaluation search statements from the plurality of new search statements, the evaluation search statements including at least one evaluation unit word.

[0187] The query module 302 is further configured to query at least one evaluation expansion unit word corresponding to the at least one evaluation unit word from the preset knowledge graph, and take the at least one evaluation unit word and the evaluation expansion unit word as evaluation associated unit words of the evaluation search statement.

[0188] The query module 302 is further configured to search, for each evaluation associated unit word, an evaluation corpus identifier matching the evaluation associated unit word from the updated inverted index library.

[0189] The determination module 303 is further configured to determine a matching level and a matching order value of each evaluation corpus identifier based on the evaluation corpus identifier matching each evaluation unit word and the matching number of each evaluation corpus identifier.

[0190] The construction module 304 is further configured to construct the matching level and the matching order value of each evaluation corpus identifier corresponding to the evaluation search statement into the evaluation data set of the evaluation search statement.

[0191] Optionally, the obtaining module 301 is specifically configured to:

[0192] randomly select a preset number of new search statements that have not been constructed into the evaluation data set from the plurality of new search statements, and take the selected new search statements as the evaluation search statements.

[0193] The embodiment of the present application also provides an electronic device, such as Figure 4 As shown in the figure, the electronic device comprises a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 complete mutual communication through the communication bus 404,

[0194] The memory 403 is used for storing a computer program.

[0195] The processor 401 is used for executing the program stored in the memory 403, so as to realize the steps of any data set evaluation method in the above method embodiment.

[0196] The communication bus mentioned in the above electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0197] The communication interface is used for communication between the above electronic device and other devices.

[0198] The memory can comprise a random access memory (RAM) and can also comprise a non-volatile memory (NVM), for example at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0199] The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0200] In a further implementation provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the data set construction methods in the above method embodiments.

[0201] In a further implementation provided in the present application, a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the steps of any of the data set construction methods in the above method embodiments.

[0202] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the above processes or functions are entirely or partially generated. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0203] It should be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0204] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0205] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An evaluation dataset construction method, characterized by, The method comprises: acquiring search statements received by a search engine in a first preset time period, the search statements comprising at least one unit word; querying, from a preset knowledge graph, extended unit words corresponding to the at least one unit word, and taking the at least one unit word and the extended unit words as associated unit words of the search statements; for each associated unit word, searching, from an inverted index library, target corpus identifiers matched by the associated unit word, the inverted index library being configured to store a set of corpus identifiers corresponding to each word in a preset corpus, the preset corpus comprising corpora searched by the search engine; determining, based on the target corpus identifiers matched by each associated unit word of the search statement and the number of matches of each target corpus identifier, a match level and a match order value of each target corpus identifier; constructing the match level and the match order value of each target corpus identifier corresponding to the search statement into an evaluation data set of the search statement.

2. The method of claim 1, wherein, The determining, based on the target corpus identifiers matched by each associated unit word of the search statement and the number of matches of each target corpus identifier, a match level and a match order value of each target corpus identifier comprises: for each target corpus identifier, determining a match level based on a match number interval to which the number of matches of the target corpus identifier belongs and a preset correspondence relationship between the match number interval and the match level; setting, in order of decreasing match level, a match order value that increases successively for target corpus identifiers of each match level, the match order values of target corpus identifiers of the same match level being the same.

3. The method of claim 1, wherein, The preset corpus is updated in real time, and the method further comprises: updating the inverted index library based on the updated preset corpus every second preset time period; acquiring target words corresponding to a set of target corpus identifiers that have been updated in the inverted index library; from search statements for which evaluation data sets have been constructed, acquiring a target search statement comprising the target words, and reconstructing an evaluation data set of the target search statement based on the updated inverted index library.

4. The method of claim 3, wherein, After the updating of the inverted index library based on the updated preset corpus, the method further comprises: acquiring a plurality of newly added search statements in the search engine in the second preset time period; selecting, from the plurality of newly added search statements, a search statement to be evaluated, the search statement to be evaluated comprising at least one unit word to be evaluated; querying, from the preset knowledge graph, extended unit words corresponding to the at least one unit word to be evaluated, and taking the at least one unit word to be evaluated and the extended unit words to be evaluated as associated unit words to be evaluated of the search statement to be evaluated; for each associated unit word to be evaluated, searching, from the updated inverted index library, corpus identifiers to be evaluated matched by the associated unit word to be evaluated; determining, based on the corpus identifiers to be evaluated matched by each associated unit word to be evaluated and the number of matches of each corpus identifier to be evaluated, a match level and a match order value of each corpus identifier to be evaluated; constructing the match level and the match order value of each corpus identifier to be evaluated corresponding to the search statement to be evaluated into an evaluation data set of the search statement to be evaluated.

5. The method of claim 4, wherein, The screening of the to-be-evaluated search statements from the plurality of newly-added search statements comprises: randomly screening a preset number of newly-added search statements that have not been constructed into an evaluation dataset from the plurality of newly-added search statements, and taking the screened newly-added search statements as the to-be-evaluated search statements.

6. An evaluation dataset construction apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire search statements received by a search engine within a first preset time period, the search statements comprising at least one unit word; a query module configured to query, from a preset knowledge graph, extended unit words corresponding to the at least one unit word, and take the at least one unit word and the extended unit words as associated unit words of the search statements; the query module is further configured to search, for each associated unit word, a target corpus identifier matched by the associated unit word from an inverted index library, the inverted index library being configured to store a set of corpus identifiers corresponding to each word in a preset corpus, the preset corpus comprising corpora searched by the search engine; a determination module configured to determine, based on the target corpus identifiers matched by each associated unit word of the search statement and the number of matches of each target corpus identifier, a matching level and a matching order value of each target corpus identifier; a construction module configured to construct the matching level and the matching order value of each target corpus identifier corresponding to the search statement into an evaluation dataset of the search statement.

7. The apparatus of claim 6, wherein, The determination module is specifically configured to: determine, for each target corpus identifier, a matching level based on a matching number interval to which the number of matches of the target corpus identifier belongs and a preset correspondence relationship between the matching number interval and the matching level; set, in order of the matching level from high to low, a matching order value that increases successively for target corpus identifiers of each matching level, the matching order values of target corpus identifiers with the same matching level being the same.

8. The apparatus of claim 6, wherein, The preset corpus is updated in real time, and the device further comprises: an update module configured to update the inverted index library based on the updated preset corpus every second preset time period; the query module is further configured to acquire target words corresponding to a set of target corpus identifiers that have been updated in the inverted index library; the construction module is further configured to acquire, from search statements that have been constructed into an evaluation dataset, a target search statement comprising the target words, and reconstruct an evaluation dataset of the target search statement based on the updated inverted index library.

9. The device of claim 8, wherein: the acquisition module is further configured to acquire a plurality of newly-added search statements in the search engine within the second preset time period; the acquisition module is further configured to screen to-be-evaluated search statements from the plurality of newly-added search statements, the to-be-evaluated search statements comprising at least one to-be-evaluated unit word; the query module is further configured to query, from the preset knowledge graph, to-be-evaluated extended unit words corresponding to the at least one to-be-evaluated unit word, and take the at least one to-be-evaluated unit word and the to-be-evaluated extended unit words as to-be-evaluated associated unit words of the to-be-evaluated search statements; The query module is further configured to search, for each to-be-evaluated associated unit word, to-be-evaluated corpus identifiers matched by the to-be-evaluated associated unit word from the updated inverted index library; The determination module is further configured to determine, based on the to-be-evaluated corpus identifiers matched by each to-be-evaluated unit word and the number of matches of each to-be-evaluated corpus identifier, a match level and a match order value of each to-be-evaluated corpus identifier; The construction module is further configured to construct the match level and the match order value of each to-be-evaluated corpus identifier corresponding to the to-be-evaluated search statement as an evaluation data set of the to-be-evaluated search statement.

10. The apparatus of claim 9, wherein, The obtaining module is specifically configured to: randomly select, from the plurality of new search statements, a preset number of new search statements that have not been constructed into evaluation data sets, and select the new search statements as to-be-evaluated search statements.

Images