Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating performance of document retrieval

Abstract

Automatically detects a performance degradation in document retrieval. When document data is appended to the database (300), the learning unit (260) updates the language model (61) using machine learning. The performance evaluation unit (270) calculates a first statistic related to the position of each specific document data in the multiple specific document data based on the document retrieval results where each of the first tags attached to the multiple specific document data is set as a retrieval query (TQRY). The performance evaluation unit (270) calculates a second statistic related to the position of each specific document data in the multiple specific document data based on the document retrieval results where each of the second tags attached to the multiple specific document data is set as a retrieval query (TQRY). If the change in the first statistic caused by the update of the language model (61) accompanying the appending of at least one document data is greater than a first threshold, and the change in the second statistic caused by the update is greater than a second threshold (Th2), the performance evaluation unit (270) detects a performance degradation in document retrieval.

Description

Technical Field

[0001] This invention relates to a method, system, and apparatus for evaluating the performance of document retrieval. Background Technology

[0002] Previously, a structure for retrieving desired information from a database was known. For example, Japanese Patent Application Publication No. 2003-99439 (Patent Document 1) discloses an analysis processing method retrieval device that maintains various analysis processing methods as a database, retrieves the desired analysis processing method from the database, and displays it. According to this analysis processing method retrieval device, intuitively easy-to-grasp keywords are set as attribute data characterizing the analysis processing method. Therefore, by setting these attribute data as search keywords, analysis processing methods that match the target processing method can be easily retrieved.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2003-99439 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] Because new information is sometimes added to the database containing information that is the object of retrieval, the search results obtained using the same search query may vary depending on the time of retrieval. Depending on the information added to the database, the tendency of the information contained in the database may change drastically, resulting in documents with relatively low relevance to the search query being ranked higher. However, the analysis processing method retrieval apparatus disclosed in Patent Document 1 does not take into account the performance degradation of document retrieval associated with changes in the tendency of the information contained in the database.

[0008] This invention was made to solve the problem of automatically detecting performance degradation in document retrieval.

[0009] Solution for solving the problem

[0010] One aspect of the invention relates to a method for evaluating document retrieval performance, which retrieves multiple associated document data related to a retrieval query from a database containing multiple document data, and ranks the multiple associated document data according to the relevance of each associated document data to the retrieval query. This relevance is calculated based on features of each document data in the multiple document data extracted by a language model, and features of the retrieval query extracted by the same language model, which is a model generated using machine learning on the database. For each specific document data within the multiple document data, a first tag associated with that specific document data and a second tag serving as a superordinate concept of the first tag are pre-attached. The method includes the following steps: updating a language model via machine learning when at least one document data has been appended to a database; calculating a first statistic related to the position of each specific document data in the plurality of specific document data based on the results of a document retrieval query using each of the first tags among the plurality of first tags appended to the plurality of specific document data as the retrieval query; calculating a second statistic related to the position of each specific document data in the plurality of specific document data based on the results of a document retrieval query using each of the second tags among the plurality of second tags appended to the plurality of specific document data as the retrieval query; and detecting a performance degradation in document retrieval if the change in the first statistic caused by the update of the language model accompanying the appending of at least one document data is greater than a first threshold, and the change in the second statistic caused by the update is greater than a second threshold.

[0011] Other aspects of the present invention relate to a system for evaluating document retrieval performance, retrieving multiple associated document data related to a retrieval query from a database containing multiple document data, and ranking the multiple associated document data according to the relevance of each associated document data to the retrieval query. This relevance is calculated based on features of each document data in the multiple document data extracted by a language model, and features of the retrieval query extracted by the language model, which is a model generated using machine learning with the database. For each specific document data within multiple specific document data contained in the multiple document data, a first tag associated with that specific document data and a second tag serving as a superordinate concept of the first tag are pre-attached. The system includes a terminal device and a server device. The terminal device is used to input a retrieval query. The server device receives the retrieval query from the terminal device to perform document retrieval. The server device includes a learning unit and a performance evaluation unit. When at least one document data is appended to the database, the learning unit updates the language model using machine learning. The performance evaluation unit evaluates the performance of the document retrieval. The performance evaluation unit calculates a first statistic related to the ranking of each specific document in the multiple specific document data based on the document retrieval results where each of the first tags attached to multiple specific document data is used as a retrieval query. The performance evaluation unit also calculates a second statistic related to the ranking of each specific document in the multiple specific document data based on the document retrieval results where each of the second tags attached to multiple specific document data is used as a retrieval query. The performance evaluation unit detects a performance degradation in document retrieval if the change in the first statistic due to an update of the language model accompanying the addition of at least one document exceeds a first threshold, and the change in the second statistic due to the update exceeds a second threshold.

[0012] Other aspects of the present invention relate to an apparatus for evaluating document retrieval performance, retrieving multiple associated document data related to a retrieval query from a database containing multiple document data, and ranking the multiple associated document data according to the relevance of each associated document data to the retrieval query. This relevance is calculated based on features of each document data in the multiple document data extracted by a language model, and features of the retrieval query extracted by the language model, which is a model generated using machine learning with the database. For each specific document data within multiple specific document data contained in the multiple document data, a first tag associated with that specific document data and a second tag serving as a superordinate concept of the first tag are pre-attached. The apparatus includes an input / output unit, a retrieval unit, a learning unit, and a performance evaluation unit. The input / output unit is used to input a retrieval query. The retrieval unit accepts a retrieval query and performs document retrieval. When at least one document data is added to the database, the learning unit updates the language model using machine learning. The performance evaluation unit evaluates the document retrieval performance. The performance evaluation unit calculates a first statistic related to the ranking of each specific document in the multiple specific document data based on the document retrieval results where each of the first tags attached to the multiple specific document data is used as a retrieval query. The performance evaluation unit also calculates a second statistic related to the ranking of each specific document in the multiple specific document data based on the document retrieval results where each of the second tags attached to the multiple specific document data is used as a retrieval query. The performance evaluation unit detects a performance degradation if the change in the first statistic due to an update of the language model accompanying the addition of at least one document data exceeds a first threshold, and the change in the second statistic due to the update exceeds a second threshold.

[0013] The effects of the invention

[0014] According to the method, system, and apparatus of the present invention, a decrease in document retrieval performance is detected when the change in a first statistical value caused by an update of a language model accompanying the addition of at least one document data is greater than a first threshold, and the change in a second statistical value caused by the update is greater than a second threshold, thereby enabling automatic detection of a decrease in document retrieval performance. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating the structure of an example document retrieval system as an implementation method.

[0016] Figure 2 It is shown Figure 1 A block diagram of the hardware structure of the computer and its input / output units.

[0017] Figure 3It is shown Figure 1 A block diagram of the functional structure of the server device and database.

[0018] Figure 4 This is a diagram showing an example of analysis case data contained in the analysis case database.

[0019] Figure 5 This shows the case when the search query is "purine body". Figure 1 An example of a search results window displayed on a monitor.

[0020] Figure 6 This indicates the case where the search query is "nucleic acid". Figure 1 An example of a search results window displayed on a monitor.

[0021] Figure 7 It is shown in Figure 1 A flowchart summarizing the retrieval process performed in the case analysis retrieval system.

[0022] Figure 8 This is a diagram illustrating the structure formed in the test query database that associates the first label, the second label, and the identifiers of specific analysis case data.

[0023] Figure 9 This is a graph showing the frequency distribution curves of the positions of multiple specific analysis case data when each of the multiple first tags is set as a retrieval query.

[0024] Figure 10 This is a graph showing the frequency distribution curves of the positions of data from multiple specific analysis cases when each of the multiple second labels is set as a retrieval query.

[0025] Figure 11 It is shown in Figure 3 A diagram outlining the process of performance evaluation in a server device.

[0026] Figure 12 It is shown Figure 3 A block diagram of the hardware structure of the server device.

[0027] Figure 13 This is a block diagram illustrating the structure of an information processing apparatus according to a variation of the embodiment.

[0028] Figure 14 It is shown Figure 13 A block diagram of the hardware structure of an information processing device. Detailed Implementation

[0029] The embodiments will now be described in detail with reference to the accompanying drawings. Furthermore, in the following description, the same or equivalent parts in the drawings will be labeled with the same symbols, and their descriptions will not be repeated in principle.

[0030] Figure 1 This is a block diagram illustrating the structure of an analysis case retrieval system 1000, which is an example of a document retrieval system according to an implementation method. Figure 1 As shown, the analysis case retrieval system 1000 includes a terminal device 100, a server device 200, and a database 300. The terminal device 100 and the server device 200 are connected via a network NW. The database 300 includes multiple analysis case data (document data). The network NW includes, for example, the Internet, a WAN (Wan Area Network), or a LAN (Lan Area Network).

[0031] Terminal device 100 includes computer 120 and input / output unit 130. Input / output unit 130 includes display 131, keyboard 132 and mouse 133. Display 131, keyboard 132 and mouse 133 are connected to computer 120.

[0032] The search window Wn1 and the cursor Cr are displayed on the monitor 131. The user manipulates the cursor Cr by operating the mouse 133. The user enters the search query QRY in the search window Wn1 by operating the keyboard 132. Figure 1 The following situation is illustrated: In order to retrieve analytical case data (document data) containing analytical devices, analytical methods and analytical conditions suitable for the analysis of purine bodies, the user enters the string "purine body" as the search query QRY in the search window Wn1.

[0033] In the analysis case retrieval system 1000, a search query input by a user is sent from the terminal device 100 to the server device 200 via the network NW. The server device 200 retrieves multiple association analysis case data (associative document data) related to the search query QRY from the database 300. The server device 200 sorts each association analysis case data in the multiple association analysis case data according to its relevance to the search query QRY, and sends the search results to the terminal device 100. Based on the search results from the server device 200, the terminal device 100 displays the multiple association analysis case data in order of their positions.

[0034] Figure 2 It is shown Figure 1 A block diagram of the hardware structure of the computer 120 and the input / output unit 130. (See diagram below.) Figure 2As shown, the computer 120 includes a processor 121, a communication interface 124, and a memory 122 and a hard disk 123 as storage units. These structures are interconnected via a bus 125 in a manner that enables communication.

[0035] Hard disk 123 is a non-volatile storage device. For example, hard disk 123 stores the operating system (OS) program 41 and the client program 42 of the case analysis and retrieval application. Except... Figure 2 In addition to the data shown, the hard disk 123 also stores, for example, settings and outputs of various applications. The memory 122 is a volatile storage device, such as DRAM (Dynamic Random Access Memory).

[0036] Processor 121 includes a CPU (Central Processing Unit). Processor 121 may also include a GPU (Graphics Processing Unit). Processor 121 reads the program stored in hard disk 123 into memory 122 and executes the program. Processor 121 is connected to network NW via communication interface 124.

[0037] The input / output unit 130 includes a display 131, a mouse 133, and a keyboard 132. The display 131, mouse 133, and keyboard 132 are connected to the computer 120. The display 131 displays the GUI (Graphical User Interface) of the case analysis retrieval application. The user inputs desired information into the case analysis retrieval application by referring to the display on the display 131 and operating the keyboard 132 or the mouse 133.

[0038] Figure 3 It is shown Figure 1 A block diagram illustrating the functional structure of the server device 200 and the database 300. (See diagram below.) Figure 3 As shown, the server device 200 includes a communication unit 210, a vector generation unit 220, a retrieval unit 230, a display control unit 250, a learning unit 260, a performance evaluation unit 270, and an analysis case generation unit 280. The database 300 includes an analysis case database 310 and a test query database 320.

[0039] The analysis case database 310 stores multiple analysis case data (analysis reports, technical papers, or patent documents, etc.) that can be used as search targets. The multiple analysis case data stored in the analysis case database 310 can also be classified into analysis case data that can generally be made public and analysis case data that can only be made public to specific users. The analysis case generation unit 280 generates new analysis case data and appends it to the analysis case database 310. The analysis case generation unit 280 can also append multiple new analysis case data to the analysis case database 310 at once. The analysis case generation unit 280 notifies the learning unit 260 that new analysis case data has been appended to the analysis case database 310. The analysis case generation unit 280 can also be configured as a separate device from the server device 200. The setting and updating of information in the database 200 can also be performed automatically at predetermined intervals.

[0040] Figure 4 This is a diagram showing analysis case data D1, which is an example of analysis case data included in analysis case database 310. (See diagram for example.) Figure 4 As shown, Analytical Case Data D1 contains information related to an analytical case involving the determination of total purine bodies in food using high-performance liquid chromatography (HPLC). In this analytical case, the specific compounds analyzed are purine bodies such as adenine and guanine. Analytical Case Data D1 is the analytical case data that a user would prefer to rank highly in search results when setting "purine bodies" as the search query QRY. Furthermore, since purine bodies are also classified as nucleic acid molecules, Analytical Case Data D1 is also the analytical case data that a user would prefer to rank highly in search results when setting "nucleic acid" or compounds associated with "nucleic acid" as the superior concept of "purine body" as the search query QRY.

[0041] Refer again Figure 3 The learning unit 260 uses machine learning from the analysis case database 310 to enable the language model 61 to learn the distributed representation (vector) of the specific natural language corresponding to the analysis case data, thereby setting the language model 61 as a fully learned model. The analysis case database 310 functions as a corpus in this machine learning. In the machine learning using the analysis case database 310, natural language processing, including lexical analysis, is performed on multiple analysis case data contained in the analysis case database 310. In lexical analysis, the string being analyzed is segmented into the smallest meaningful linguistic units (morphemes or words). The specific natural language corresponding to the analysis case data refers to the natural language used to represent the information contained in the analysis case data.

[0042] Language model 61 can be either a counting-based language model using a co-occurrence matrix or an inference-based language model using a neural network. Examples of inference-based language models include the CBOW (Continuous Bag-of-Words) model and skip-gram model for word2vec used to generate distributed representations of words, and the PV-DM (Paragraph Vector Distributed Memory) model and PV-DBOW (Paragraph Vector Distributed Bag-of-Words) model for doc2vec used to generate distributed representations of documents.

[0043] The vector generation unit 220 uses the learned language model 61 to generate word vectors representing the features of words contained in the analysis example data, sentence vectors representing the features of sentences contained in the analysis example data, and document vectors representing the features of documents for each analysis example data in the analysis example data contained in the analysis example database 310. The vector generation unit 220 vectorizes each analysis example data in the analysis example data contained in the analysis example database 310 and saves the data obtained by vectorizing the analysis example data in association with the analysis example data in the analysis example database 310.

[0044] The communication unit 210 receives the search query QRY information from the terminal device 100 via the network NW, and outputs the search query QRY to the vector generation unit 220. The search query QRY is a string represented in natural language.

[0045] For a search query QRY, the vector generation unit 220 segments the query QRY into the smallest linguistic units through lexical analysis. The vector generation unit 220 uses the language model 61 to calculate the word vector WVC representing each linguistic unit. The vector generation unit 220 outputs the word vector WVC information to the retrieval unit 230.

[0046] The retrieval unit 230 retrieves multiple related analysis case data containing features similar to the word vector WVC from multiple analysis case data stored in the analysis case database 310. Specifically, for each analysis case data in the multiple analysis case data, the retrieval unit 230 calculates the relevance between the search query QRY and the analysis case data based on the distance between the vector of the search query QRY and the vector of the analysis case data. For example, cosine distance (cosine similarity) can be used as this distance. The smaller the distance between two vectors, the greater the relevance between the two linguistic units represented by the two vectors. The retrieval unit 230 sorts the multiple related analysis case data whose relevance to the search query QRY is greater than a predetermined threshold. The retrieval unit 230 outputs ranking information RNK, indicating the position of the multiple related analysis case data, to the display control unit 250.

[0047] Furthermore, the retrieval performed by the retrieval unit 230 is a semantic retrieval that interprets the content of the analytical examples to be retrieved based on the strings (keywords) contained in the search query QRY. Semantic retrieval determines the search targets that match the search string based on the meaning of the keywords, the intended search, and the purpose. That is, semantic retrieval does not capture keywords as simple strings, but rather captures keywords as strings that contain meaning including the search request.

[0048] Furthermore, in the retrieval performed by the retrieval unit 230, semantic retrieval and keyword retrieval can be used in combination. Keyword retrieval is used to retrieve analytical example data containing strings consistent with keywords. That is, the relevance of the retrieval query to the analytical example data can be calculated based on the distance between the vector of keywords and the vector of analytical example data, as well as the number of keywords contained in the analytical example data. For example, analytical example data containing more keywords can be ranked higher, while analytical example data without keywords can be ranked lower than analytical example data containing keywords through semantic retrieval. By using semantic retrieval and keyword retrieval in combination, the clarity of the retrieval criteria for semantic retrieval can be improved.

[0049] The display control unit 250 generates screen information DSP for the search results displayed on the display 131 of the terminal device 100 based on the ranking information RNK. The display control unit 250 sends the screen information DSP to the terminal device 100 via the communication unit 210. In the terminal device 100, the names of multiple correlation analysis case data are displayed on the display 131 in rank order according to the screen information DSP.

[0050] Figure 5 This shows the case where the search query QRY is "purine body". Figure 1An example diagram of the search results window Wn2 displayed on monitor 131. Figure 5 As shown, in the search results window Wn2, the names of the association analysis case data D1 to D11 are displayed in that order along with their positions 1 to 11. Each of the association analysis case data D1 to D11 has a hyperlink to that case data. Among the multiple analysis case data contained in the analysis case database 310, the analysis case data D1 is the most relevant to the search query QRY for "purine body".

[0051] Figure 6 This shows the case where the search query QRY is "nucleic acid". Figure 1 An example diagram of the search results window Wn2 displayed on monitor 131. Figure 6 As shown, in the search results window Wn2, the names and positions 1 to 11 of the association analysis case data D5, D1, D9, D3, D2, D6 to D8, D4, D10, and D11 are displayed in this order. Each of the association analysis case data D1 to D11 has a hyperlink to that case data. Among the multiple analysis case data contained in the analysis case database 310, the analysis case data D5 is most relevant to the search query QRY "nucleic acid".

[0052] Figure 7 It is shown in Figure 1 A diagram illustrating the summary flow of the retrieval process performed in the analysis case retrieval system 1000. The steps are simply described below as S. Figure 7 As shown, in S10, the user inputs a search query QRY into the terminal device 100, and the process proceeds to S20. In S20, the vector generation unit 220 calculates the vector of the search query QRY and the process proceeds to S30. In S30, the retrieval unit 230 retrieves multiple association analysis case data associated with the search query QRY from the analysis case database 310, sorts the multiple association analysis case data, and the process proceeds to S40. In S40, the display 131 of the terminal device 100 displays the search results, and the search process ends.

[0053] Refer again Figure 3 When new analysis case data is added to the analysis case database 310, the learning unit 260 performs machine learning using the analysis case database 310 to update the language model 61. The vector generation unit 220 uses the updated language model 61 to update the vectors of each analysis case data in the multiple analysis case data contained in the analysis case database 310.

[0054] When new analytical case data is appended to the analytical case database 310, the vector of the retrieval query QRY and the vectors of individual analytical case data within the multiple analytical case data contained in the analytical case database 310 may change. Therefore, even using the same retrieval query QRY, the search results before appending new analytical case data to the analytical case database 310 may differ from the search results after appending new analytical case data to the analytical case database 310. Based on the information appended to the analytical case database 310, the bias of the information contained in the analytical case database 310 may change drastically, resulting in analytical case data with relatively low relevance to the retrieval query QRY being ranked higher.

[0055] Therefore, in the analytical case retrieval system 1000, when new analytical case data is added to the analytical case database 310, multiple predetermined test queries are used to calculate the change in the frequency distribution of the positions of the multiple analytical case data as a change in statistical values ​​representing the characteristics of that distribution. According to the analytical case retrieval system 1000, since changes in retrieval performance can be quantitatively calculated, a decrease in retrieval performance can be automatically detected based on a predetermined benchmark.

[0056] The first tag contains a string associated with that specific analytical case data. For example, the first tag may contain a string representing the name of the compound being analyzed as the primary subject of the analytical case data, or a string representing the name of the apparatus used to perform the analysis of that compound.

[0057] The second label contains a string representing the superordinate concept of the first label. If the first label contains a string representing the name of a compound, the second label contains a string representing the chemical or biological superordinate concept of that compound. For example, if the first label is "purine body" or "purine base," the second label could be, for example, "nucleic acid," "alkaloid," or "base." The first and second labels for a specific analytical case data can be automatically appended to that specific analytical case data or appended by the operator.

[0058] Alternatively, for example, if the first label is a string that serves as the name of the instrument, such as LCMS (Liquid Chromatograph Mass Spectrometer)-XYZW (where XYZW is the model number of the instrument), the second label can also be a quadrupole LC / MS, a time-of-flight LC / MS, or a simple LC / MS.

[0059] Because the first and second tags are linked together by concepts related to the compound being analyzed or the apparatus used to analyze the compound, it is possible to detect with high precision any performance degradation in document retrieval against databases containing document data that includes information related to chemical or biological analysis.

[0060] Figure 8 This is a diagram illustrating the structure formed in the test query database 320 that associates the first label, the second label, and the identifiers of specific analysis case data. For example... Figure 8 As shown, for each of the specific analytical case data Da11, Da12, and Da13, a first label Ca1 and a second label CA are attached. For each of the specific analytical case data Da21, Da22, and Da23, a first label Ca2 and a second label CA are attached. The second label CA is a superordinate concept of the first labels Ca1 and Ca2, encompassing the first labels Ca1 and Ca2. For each of the specific analytical case data D1, D3, and D6, "purine body" as the first label and "nucleic acid" as the second label are attached. For the specific analytical case data D5, a first label Cb1 and "nucleic acid" as the second label are attached. For the specific analytical case data D9, a first label Cb2 and "nucleic acid" as the second label are attached. "Nucleic acid" is a superordinate concept of "purine body," the first labels Cb1 and Cb2, encompassing "purine body," the first labels Cb1 and Cb2. Furthermore, the specific analytical case data D1, D3, D5, D6, and D9 are related to... Figure 5 and Figure 6 The analysis case data D1, D3, D5, D6, and D9 shown in the figure correspond to respectively.

[0061] Refer again Figure 3 The vector generation unit 220 outputs a notification to the performance evaluation unit 270, indicating that the vector update of multiple analysis case data contained in the analysis case database 310, which is associated with the addition of new analysis case data to the analysis case database 310, has been completed. Based on this notification, the performance evaluation unit 270 retrieves multiple test queries from the test query database 320 and performs a retrieval of analysis case data for each of these multiple test queries as a retrieval query TQRY.

[0062] The performance evaluation unit 270 obtains the ranking information RNK related to the search query TQRY from the retrieval unit 230. The performance evaluation unit 270 counts the frequency of the position of each specific analysis case data in the ranking information RNK related to multiple first tags in multiple test queries, and calculates a statistical value (first statistical value) related to the first tag. The performance evaluation unit 270 counts the frequency of the position of each specific analysis case data in the ranking information RNK related to multiple second tags in multiple test queries, and calculates a statistical value (second statistical value) related to the multiple second tags. The statistical values ​​related to the first tags and the statistical values ​​related to the second tags are respectively saved to the storage unit in association with the time when the statistical value is calculated.

[0063] If the change in the statistical value related to the first tag caused by the update of language model 61 exceeds a predetermined threshold Th1 (first threshold), and the change in the statistical value related to the second tag caused by the update of language model 61 exceeds a predetermined threshold Th2 (second threshold), the performance evaluation unit 270 detects a decrease in retrieval performance. Thresholds Th1 and Th2 can be appropriately determined through actual experiments or simulations. Furthermore, the change in statistical value is the absolute value of the difference between the statistical value after new analytical case data is added to the analytical case database 310 and the statistical value before the new analytical case data is added to the analytical case database 310. Furthermore, in the retrieval performed by the performance evaluation unit 270, a decrease in retrieval performance is observed. Figure 7 The same processing applies to S20 and S30.

[0064] Given a search query TQRY where "purine body" is the first tag, the ranking of search results and... Figure 5 The positions shown are the same. (Refer to...) Figure 8 and Figure 5The positions of specific analytical case data D1, D3, and D6, which were appended with "purine body," were 1st, 3rd, and 6th, respectively. Therefore, the frequencies of the 1st, 3rd, and 6th positions related to the first tag increased by 1. Similarly, when the search query TQRY was the first tag Ca1, the frequencies of the positions of specific analytical case data Da11, Da12, and Da13 in the search results related to the first tag increased by 1. When the search query TQRY was the first tag Ca2, the frequencies of the positions of specific analytical case data Da21, Da22, and Da23 in the search results related to the first tag increased by 1. When the search query TQRY was the first tag Cb1, the frequencies of the position of specific analytical case data D5 in the search results related to the first tag increased by 1. When the search query TQRY was the first tag Cb2, the frequencies of the position of specific analytical case data D9 in the search results related to the first tag increased by 1.

[0065] Let's consider the search results ranking when the search query TQRY is "nucleic acid" as the second tag. Figure 6 The positions shown are the same. (Refer to...) Figure 8 and Figure 6 The positions of the specific analytical case data D1, D3, D5, D6, and D9, which were appended with "nucleic acid," were 2nd, 4th, 1st, 6th, and 3rd, respectively. Therefore, the frequencies associated with the second tag at each of the 2nd, 4th, 1st, 6th, and 3rd positions increased by 1. When the search query TQRY is the second tag CA, the frequencies associated with the second tag at each of the specific analytical case data Da11, Da12, Da13, Da21, Da22, and Da23 in the search results increased by 1.

[0066] Figure 9 This is a graph showing the frequency distribution curves of the positions of data from multiple specific analysis cases when each of the multiple first labels is set to retrieve the query TQRY. Figure 9 In the diagram, the solid line represents the distribution curve before new analytical case data is added to the analytical case database 310, and the dashed line represents the distribution curve after new analytical case data is added to the analytical case database 310. Additionally, in Figure 9 In calculus, the mode is used as a statistical value to represent a characteristic of the distribution. Figure 10 The same applies to the middle. Furthermore, statistical values ​​representing the characteristics of a distribution can be values ​​other than the mode; for example, they can be the median or the mean.

[0067] like Figure 9As shown, the mode related to the first label before adding new analysis case data to the analysis case database 310 is Vs11. The mode related to the first label after adding new analysis case data to the analysis case database 310 is Vs12. The change value of the mode related to the first label is Vc1.

[0068] Figure 10 This is a graph showing the frequency distribution of the positions of data from multiple specific analysis cases when each of the multiple second labels is set to retrieve the query TQRY. For example... Figure 10 As shown, the mode related to the second label before adding new analysis case data to the analysis case database 310 is Vs21. The mode related to the second label after adding new analysis case data to the analysis case database 310 is Vs22. The change in the mode related to the second label is Vc2. <Vc1)。

[0069] Second tags typically contain multiple distinct first tags. The number of analytical case data with second tags attached is greater than the number of analytical case data with first tags attached as subordinate concepts of the second tags. Therefore, regarding changes in the ranking of search results caused by updates to language model 61, changes are more likely to occur when the first tag is set as the search query than when the second tag is set as the search query. In accordance with the likelihood of changes in the ranking of search results caused by updates to language model 61, it is desirable that the threshold Th1 for statistics related to the first tag be greater than the threshold Th2 for statistics related to the second tag. By making the threshold Th1 greater than Th2, it is possible to consistently detect anomalous signs exhibited by changes in statistics related to the first tag and changes in statistics related to the second tag.

[0070] Figure 11 It is shown in Figure 3 A diagram illustrating the summary flow of performance evaluation processing performed in server device 200. (See diagram for example.) Figure 11 As shown, in S110, the analysis case generation unit 280 adds new analysis case data to the analysis case database 310 and proceeds to S120. In S120, the learning unit 260 performs machine learning using the analysis case database 310 to update the language model 61 and the vectors of each analysis case data in the multiple analysis case data contained in the analysis case database 310, and proceeds to S130. In S130, the retrieval unit 230 performs a retrieval of analysis case data that sets each of the multiple first tags and multiple second tags registered in the test query database 320 as the retrieval query TQRY, and proceeds to S140.

[0071] In S140, the performance evaluation unit 270 calculates a statistical value Vs1 related to the first tag based on the retrieval results of the analysis case data where each of the multiple first tags is set as the retrieval query TQRY, and then proceeds to S150. In S150, the performance evaluation unit 270 calculates a statistical value Vs2 related to the second tag based on the retrieval results of the analysis case data where each of the multiple second tags is set as the retrieval query TQRY, and then proceeds to S160.

[0072] In S160, the performance evaluation unit 270 determines whether the change value Vc1 of the statistical value Vs1 is greater than the threshold Th1. If the change value Vc1 is less than the threshold Th1 ("No" in S160), the performance evaluation unit 270 allows the normal search process to begin in S190 and ends the process. If the change value Vc1 is greater than the threshold Th1 ("Yes" in S160), the performance evaluation unit 270 causes the process to proceed to S170.

[0073] In S170, the performance evaluation unit 270 determines whether the change value Vc2 of the statistical value Vs2 is greater than the threshold Th2. If the change value Vc2 is less than the threshold Th2 ("No" in S170), the performance evaluation unit 270 allows the normal retrieval process to begin in S190 and ends the process. If the change value Vc2 is greater than the threshold Th2 ("Yes" in S170), the performance evaluation unit 270 detects a decrease in retrieval performance in S180 and ends the process. In the analysis case retrieval system 1000, since the performance evaluation process includes adding new analysis case data to the analysis case database 310, the performance evaluation of adding new analysis case data to the analysis case database 310 and document retrieval can be performed simultaneously and continuously.

[0074] Reasons for decreased retrieval performance could include, for example, a deviation in the tendency of newly added analytical case data to the analytical case database 310 from the tendency of multiple analytical case data already included in the analytical case database 310, or an incompatibility between the analytical case database 310 and the hyperparameters of machine learning. Upon detection of decreased retrieval performance, the performance degradation is notified to pre-determined relevant personnel (e.g., the administrator of the analytical case retrieval system 1000). Alternatively, upon detection of decreased retrieval performance, the newly added analytical case data can be deleted from the analytical case database 310, and the language model 61 prior to the addition of the analytical case data and the vectors of each analytical case data from the multiple analytical case data sets can be used in the retrieval of the analytical case data.

[0075] Figure 12 It is shown Figure 3 A block diagram of the hardware structure of server device 200. (See diagram below.) Figure 12As shown, the server device 200 includes a processor 201, a communication interface 204 serving as a communication unit 210, an input / output unit 205, and a memory 202 and a hard disk 203 serving as storage units. These structures are interconnected via a bus 206 in a manner that enables communication.

[0076] Hard disk 203 is a non-volatile storage device. Hard disk 203 stores, for example, the operating system (OS) program 51, the case retrieval server program 52, the performance evaluation program 53, the machine learning program 54, and the language model 61. Except... Figure 12 In addition to the data shown, the hard disk 203 stores, for example, settings and outputs of various applications (e.g., statistics related to the first tag and statistics related to the second tag). The memory 202 is a volatile storage device, such as DRAM (Dynamic Random Access Memory).

[0077] Processor 201 includes a CPU (Central Processing Unit). Processor 201 may also include a GPU (Graphics Processing Unit). Processor 201 reads programs stored in hard disk 203 into memory 202 and executes the programs, thereby implementing various functions of server device 200. For example, processor 201 executing the analysis case retrieval server program 52 functions as a vector generation unit 220, a retrieval unit 230, a display control unit 250, and an analysis case generation unit 280. Processor 201 executing the performance evaluation program 53 functions as a vector generation unit 220 and a performance evaluation unit 270. Processor 201 executing the machine learning program 54 functions as a vector generation unit 220 and a learning unit 260. Processor 201 is connected to network NW via communication interface 204.

[0078] In this embodiment, a system in which a vector generation unit 220, a retrieval unit 230, a display control unit 250, a learning unit 260, a performance evaluation unit 270, and an analysis case generation unit 280 are formed in a server device 200 separate from the terminal device 100 for user input of retrieval queries is described. The vector generation unit, retrieval unit, display control unit, learning unit, performance evaluation unit, and analysis case generation unit may also be formed in the device for user input of retrieval queries.

[0079] Figure 13 This is a block diagram illustrating the structure of the information processing apparatus 100A according to a modified embodiment. Figure 13As shown, the information processing apparatus 100A includes a vector generation unit 220A, a retrieval unit 230A, a display control unit 250A, a learning unit 260A, a performance evaluation unit 270A, and an analysis case generation unit 280A. The vector generation unit 220A, retrieval unit 230A, display control unit 250A, learning unit 260A, performance evaluation unit 270A, and analysis case generation unit 280A have the same... Figure 3 The vector generation unit 220, retrieval unit 230, display control unit 250, learning unit 260, performance evaluation unit 270, and analysis case generation unit 280 each have the same functions, therefore, the description of these functions will not be repeated. The database 300 is connected to the information processing device 100A.

[0080] Figure 14 It is shown Figure 13 A block diagram of the hardware structure of the information processing device 100A. The hardware structure of the information processing device 100A is as follows: Figure 2 Added to hard drive 123 Figure 12 The performance evaluation program 53, the machine learning program 54, and the language model 61 are included, and the analysis case retrieval client program 42 is replaced with the analysis case retrieval program 42A. Otherwise, they are the same and will not be described again.

[0081] like Figure 14 As shown, the processor 121 executing the analysis case retrieval program 42A functions as a vector generation unit 220A, a retrieval unit 230A, a display control unit 250A, and an analysis case generation unit 280A. The processor 121 executing the performance evaluation program 53 functions as a vector generation unit 220A and a performance evaluation unit 270A. The processor 121 executing the machine learning program 54 functions as a vector generation unit 220A and a learning unit 260A.

[0082] Based on the system and method described in the embodiments and the apparatus described in the variations, it is possible to automatically detect a decrease in document retrieval performance.

[0083] [Way]

[0084] Those skilled in the art will understand that the exemplary embodiments described above are specific examples of the following methods.

[0085] (First item) One approach involves a method for evaluating document retrieval performance, retrieving multiple related document data associated with a retrieval query from a database containing multiple document data, and ranking the multiple related document data according to the relevance of each related document data to the retrieval query. This relevance is calculated based on features of each document data in the multiple document data extracted by a language model, and features of the retrieval query extracted by the same language model, which is a model generated using machine learning on the database. For each specific document data within the multiple document data, a first label associated with that specific document data and a second label serving as a superordinate concept of the first label are pre-attached. The method includes the following steps: updating a language model via machine learning when at least one document data has been appended to a database; calculating a first statistic related to the position of each specific document data in the plurality of specific document data based on the results of a document retrieval query using each of the first tags among the plurality of first tags appended to the plurality of specific document data as the retrieval query; calculating a second statistic related to the position of each specific document data in the plurality of specific document data based on the results of a document retrieval query using each of the second tags among the plurality of second tags appended to the plurality of specific document data as the retrieval query; and detecting a performance degradation in document retrieval if the change in the first statistic caused by the update of the language model accompanying the appending of at least one document data is greater than a first threshold, and the change in the second statistic caused by the update is greater than a second threshold.

[0086] According to the method described in the first paragraph, a performance degradation in document retrieval is detected when the change in a first statistical value caused by an update of the language model accompanying the addition of at least one document data is greater than a first threshold, and the change in a second statistical value caused by the update is greater than a second threshold, thereby enabling automatic detection of a performance degradation in document retrieval.

[0087] (Second item) In the method described in the first item, the first tag contains a string representing the name of a compound that is set as the object of analysis in specific document data to which the first tag is attached. The second tag contains a string representing the chemical or biological superordinate concept of the compound.

[0088] According to the method described in the second paragraph, since the first tag and the second tag are associated with each other through a concept related to the compound being analyzed, it is possible to detect with high precision the performance degradation of document retrieval for databases containing document data including information related to chemical or biological analysis.

[0089] (Third item) In the method described in the first or second item, the language model is able to generate distributed representations of each word and sentence contained in multiple document data sets, as well as distributed representations of each document data set within the multiple document data sets. The relevance between the related document data set and the retrieval query is calculated using the distance between the distributed representations of each related document data set and the distributed representation of the retrieval query.

[0090] According to the method described in the third item, the degree of relevance between the related document data and the retrieval query is calculated by using the distance between the distributed representation of each related document data in multiple related document data and the distributed representation of the retrieval query. This degree of relevance can be used as an indicator of the semantic similarity between the retrieval query and each related document data in multiple related document data.

[0091] (Fourth item) In the method described in the third item, the relevance between the associated document data and the retrieval query is calculated based on the distance between the distributed representation of each associated document data in multiple associated document data and the distributed representation of the retrieval query, as well as the number of retrieval queries contained in the associated document data.

[0092] According to the method described in the fourth item, the number of search queries contained in the related document data is also used in the calculation of the relevance, thereby improving the clarity of the search basis.

[0093] (Fifth item) The method described in any of the first to fourth items includes the step of appending at least one document data to the database.

[0094] According to the method described in item five, performance evaluation of adding new document data to the database and document retrieval can be performed in an integrated and continuous manner.

[0095] (Sixth item) In any of the methods described in the first to fifth items, the first threshold is greater than the second threshold.

[0096] According to the method described in item six, it is possible to consistently detect abnormal signs exhibited by each variation in the first and second statistical values.

[0097] (Seventh) A system according to one method is used to evaluate the performance of document retrieval, retrieving multiple related document data associated with a retrieval query from a database containing multiple document data, and ranking the multiple related document data according to the relevance of each related document data to the retrieval query. This relevance is calculated based on features of each document data in the multiple document data extracted by a language model, and features of the retrieval query extracted by the language model, which is a model generated using machine learning with the database. For each specific document data within multiple specific document data contained in the multiple document data, a first label associated with that specific document data and a second label serving as a superordinate concept of the first label are pre-attached. The system includes a terminal device and a server device. The terminal device is used to input a retrieval query. The server device receives the retrieval query from the terminal device to perform document retrieval. The server device includes a learning unit and a performance evaluation unit. When at least one document data is appended to the database, the learning unit updates the language model using machine learning. The performance evaluation unit evaluates the performance of the document retrieval. The performance evaluation unit calculates a first statistic related to the ranking of each specific document in the multiple specific document data based on the document retrieval results where each of the first tags attached to multiple specific document data is used as a retrieval query. The performance evaluation unit also calculates a second statistic related to the ranking of each specific document in the multiple specific document data based on the document retrieval results where each of the second tags attached to multiple specific document data is used as a retrieval query. The performance evaluation unit detects a performance degradation in document retrieval if the change in the first statistic due to an update of the language model accompanying the addition of at least one document exceeds a first threshold, and the change in the second statistic due to the update exceeds a second threshold.

[0098] According to the system described in item 7, a performance degradation in document retrieval is detected when the change in a first statistic caused by an update of the language model accompanying the addition of at least one document data is greater than a first threshold, and the change in a second statistic caused by the update is greater than a second threshold, thereby enabling automatic detection of a performance degradation in document retrieval.

[0099] (Eighth item) An apparatus according to one method is used to evaluate the performance of document retrieval, retrieves multiple related document data associated with a retrieval query from a database containing multiple document data, and ranks the multiple related document data according to the relevance of each related document data in the multiple related document data to the retrieval query. This relevance is calculated based on features of each document data in the multiple document data extracted by a language model, and features of the retrieval query extracted by the language model, which is a model generated using machine learning with the database. For each specific document data in the multiple document data, a first label associated with that specific document data and a second label as a superordinate concept of the first label are pre-attached. The apparatus includes an input / output unit, a retrieval unit, a learning unit, and a performance evaluation unit. The input / output unit is used to input a retrieval query. The retrieval unit accepts a retrieval query to perform document retrieval. When at least one document data is appended to the database, the learning unit updates the language model using machine learning. The performance evaluation unit evaluates the performance of the document retrieval. The performance evaluation unit calculates a first statistic related to the ranking of each specific document in the multiple specific document data based on the document retrieval results where each of the first tags attached to the multiple specific document data is used as a retrieval query. The performance evaluation unit also calculates a second statistic related to the ranking of each specific document in the multiple specific document data based on the document retrieval results where each of the second tags attached to the multiple specific document data is used as a retrieval query. The performance evaluation unit detects a performance degradation if the change in the first statistic due to an update of the language model accompanying the addition of at least one document data exceeds a first threshold, and the change in the second statistic due to the update exceeds a second threshold.

[0100] According to the device described in the eighth item, a decrease in document retrieval performance is detected when the change in a first statistical value caused by an update of the language model accompanying the addition of at least one document data is greater than a first threshold, and the change in a second statistical value caused by the update is greater than a second threshold, thereby enabling automatic detection of a decrease in document retrieval performance.

[0101] Furthermore, regarding the above-described embodiments and variations, including combinations not mentioned in the specification, structures described in the embodiments are intended to be appropriately combined from the outset of the application, without causing problems or contradictions.

[0102] It should be considered that the embodiments disclosed herein are illustrative in all respects and not restrictive. The scope of the invention is set forth in the claims, not in the foregoing description, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0103] Explanation of reference numerals in the attached figures

[0104] 61: Language model; 100: Terminal device; 100A: Information processing device; 120: Computer; 121, 201: Processor; 122, 202: Memory; 123, 203: Hard disk; 124, 204: Communication interface; 125, 206: Bus; 130, 205: Input / output unit; 131: Display; 132: Keyboard; 133: Mouse; 200: Server device; 210: Communication unit; 220, 220A: Vector generation unit; 230, 230A: Retrieval unit; 250, 250A: Display control unit; 260, 260A: Learning unit; 27 0, 270A: Performance Evaluation Department; 280, 280A: Analysis Case Generation Department; 300: Database; 310: Analysis Case Database; 320: Test Query Database; 1000: Analysis Case Retrieval System; CA: Second Tag; Ca1, Ca2, Cb1, Cb2: First Tag; D1, D2, D3, D4, D5, D6, D8, D9, D10, D11: Analysis Case Data; NW: Network; QRY, TQRY: Retrieval Query; Th1, Th2: Threshold; Vc1, Vc2: Variation Value; Vs1, Vs2: Statistical Value; WVC: Word Vector.

Claims

1. A method for evaluating the performance of document retrieval, comprising retrieving multiple related document data associated with a retrieval query from a database containing multiple document data, and sorting the multiple related document data according to the relevance of each related document data to the retrieval query, wherein, The relevance is calculated based on the features of each document in the plurality of document data extracted by the language model, and the features of the retrieval query extracted by the language model, which is a model generated using machine learning on the database. For each retrieval object document data within the plurality of retrieval object document data contained in the plurality of document data, a first tag associated with that retrieval object document data and a second tag serving as a superordinate concept of the first tag are pre-attached. The method includes the following steps: When at least one document data is appended to the database, the language model is updated using the machine learning method. A first statistical value related to the position of each of the multiple first tags attached to the multiple search object document data is calculated based on the document retrieval result of the search query. Based on the document retrieval results of the search query, using each of the second tags from the plurality of second tags attached to the plurality of search object document data as the search query, a second statistical value related to the position of each search object document data in the plurality of search object document data is calculated; and The performance degradation is detected when the change in the first statistic caused by the update of the language model accompanying the addition of the at least one document data is greater than a first threshold, and the change in the second statistic caused by the update is greater than a second threshold.

2. The method according to claim 1, characterized in that, The first tag contains a string representing the name of a compound that is set as an analysis object in the retrieval object document data with the first tag attached. The second tag contains a string representing a higher-level chemical or biological concept of the compound.

3. The method according to claim 1, characterized in that, The language model is capable of generating distributed representations of each word and sentence contained in the multiple document data, as well as distributed representations of each document data within the multiple document data. The relevance is calculated using the distance between the distributed representation of each of the multiple related document data and the distributed representation of the retrieval query.

4. The method according to claim 3, characterized in that, The relevance is calculated based on the distance and the number of search queries contained in each of the multiple related document data.

5. The method according to claim 1, characterized in that, It also includes the step of appending the at least one document data to the database.

6. The method according to claim 1, characterized in that, The first threshold is greater than the second threshold.

7. A system for evaluating the performance of document retrieval, comprising retrieving multiple related document data associated with a retrieval query from a database containing multiple document data, and sorting the multiple related document data according to the relevance of each related document data to the retrieval query, wherein, The relevance is calculated based on the features of each document in the plurality of document data extracted by the language model, and the features of the retrieval query extracted by the language model, which is a model generated using machine learning on the database. For each retrieval object document data within the plurality of retrieval object document data contained in the plurality of document data, a first tag associated with that retrieval object document data and a second tag serving as a superordinate concept of the first tag are pre-attached. The system has the following features: A terminal device for inputting the search query; as well as A server device receives the search query from the terminal device to perform the document search. The server device includes: The learning unit, upon appending at least one document data to the database, updates the language model using the machine learning method; and The performance evaluation department evaluates the aforementioned performance. The performance evaluation unit performs the following processing: A first statistical value related to the position of each of the multiple first tags attached to the multiple search object document data is calculated based on the document retrieval result of the search query. Based on the document retrieval results of the search query, using each of the second tags from the plurality of second tags attached to the plurality of search object document data as the search query, a second statistical value related to the position of each search object document data in the plurality of search object document data is calculated; and The performance degradation is detected when the change in the first statistic caused by the update of the language model accompanying the addition of the at least one document data is greater than a first threshold, and the change in the second statistic caused by the update is greater than a second threshold.

8. An apparatus for evaluating the performance of document retrieval, comprising retrieving multiple associated document data related to a retrieval query from a database containing multiple document data, and sorting the multiple associated document data according to the relevance of each associated document data to the retrieval query, wherein, The relevance is calculated based on the features of each document in the plurality of document data extracted by the language model, and the features of the retrieval query extracted by the language model, which is a model generated using machine learning on the database. For each retrieval object document data within the plurality of retrieval object document data contained in the plurality of document data, a first tag associated with that retrieval object document data and a second tag serving as a superordinate concept of the first tag are pre-attached. The device includes: An input / output unit is used to input the search query; The retrieval department accepts the retrieval query and performs the document retrieval; The learning unit updates the language model using machine learning when at least one document data is added to the database. as well as The performance evaluation department evaluates the aforementioned performance. The performance evaluation unit performs the following processing: A first statistical value related to the position of each of the multiple first tags attached to the multiple search object document data is calculated based on the document retrieval result of the search query. A second statistical value related to the position of each of the multiple second tags attached to the multiple search object document data is calculated based on the document retrieval result of the search query. as well as The performance degradation is detected when the change in the first statistic caused by the update of the language model accompanying the addition of the at least one document data is greater than a first threshold, and the change in the second statistic caused by the update is greater than a second threshold.

Images