Document query method, electronic device and computer program product

Abstract

The invention provides a document query method, electronic equipment and a computer program product. The document query method comprises the steps of determining a target document hit by a query keyword from a document index according to the query keyword; calculating a correlation score between the query keyword and the title of the target document; calculating a correlation score between the query keyword and the text of the target document; according to the relevance score of the title and the relevance score of the text, determining the relevance score of the query keyword and the target document; and generating a document query result according to the correlation score of the target document.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a document query method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] With the rapid development of information technology, document query technology has been widely applied in the fields of knowledge management, information query and data analysis.

[0003] In the related art, a document matching a query keyword is determined according to the similarity between the query keyword and the title of the document. Such a document query method is too single, which may result in inaccurate document query results. SUMMARY

[0004] The present disclosure provides a document query method, an electronic device, a storage medium and a program product.

[0005] According to one aspect of the present disclosure, a document query method is provided, comprising: determining a target document hit by a query keyword from a document index according to the query keyword; calculating a relevance score of the query keyword and a title of the target document; calculating a relevance score of the query keyword and a body of the target document; determining a relevance score of the query keyword and the target document according to the relevance score of the title and the relevance score of the body; and generating a document query result according to the relevance score of the target document.

[0006] According to the technical solution of one aspect, the relevance score of the target document and the query keyword is calculated according to the relevance scores of the title and the body of the target document with respect to the query keyword, and the document query result is generated according to the relevance score of the target document, thereby comprehensively considering the title and the body of the document, improving the coverage of the document query, increasing the possibility of hitting relevant documents, and enabling more documents related to the query keyword to be retrieved, thereby improving the comprehensiveness and accuracy of the document query.

[0007] According to the document query method of at least one embodiment of the present disclosure, calculating the relevance score of the query keyword and the title of the target document comprises: determining the term frequency of the query keyword in the title of the target document; determining the inverse document frequency of the query keyword in the titles of multiple documents; and calculating the relevance score of the query keyword and the title of the target document according to the term frequency in the title and the inverse document frequency in the title.

[0008] According to the technical solution of the present embodiment, the relevance of the query keyword and the title of the target document is accurately quantified according to the term frequency and the inverse document frequency, the distinguishing ability of the title in the document query is enhanced, the document query result is more in line with the user's intention, and the accuracy of the document query is improved.

[0009] According to the document query method of at least one embodiment of the present disclosure, the word frequency of the query keyword in the title of the target document is determined, including: determining the total number of words in the title of the target document and the position of the query keyword in the title of the target document; determining the number of occurrences of the query keyword in the title of the target document based on the position of the query keyword in the title of the target document; and dividing the number of occurrences in the title by the total number of words in the title to obtain the word frequency of the query keyword in the title of the target document.

[0010] According to the technical solution of the present embodiment, a word frequency determination algorithm is provided, which determines the number of occurrences of the query keyword in the title based on the position, provides a reliable data basis for the calculation of the word frequency, ensures the accuracy of the word frequency calculation, and reduces the workload of the pre-generated document index process.

[0011] According to the document query method of at least one embodiment of the present disclosure, the inverse document frequency of the query keyword in the title of the target document is determined, including: determining the title hit by the query keyword from the document index according to the query keyword; and dividing the total number of titles of multiple documents by the number of titles hit by the query keyword to obtain the inverse document frequency of the query keyword in the title of the target document.

[0012] According to the technical solution of the present embodiment, an inverse document frequency determination algorithm is provided, which quickly determines the title hit by the query keyword through the document index, and then calculates the inverse document frequency of the query keyword in the title according to the total number of titles of multiple documents and the number of hit titles, so that the rarity of the query keyword in the title of the document library can be judged through the inverse document frequency, accurate data support is provided for document query, and the workload of the pre-generated document index process is reduced.

[0013] According to the document query method of at least one embodiment of the present disclosure, the relevance score of the query keyword and the title of the target document is calculated according to the word frequency in the title and the inverse document frequency in the title, including: multiplying the word frequency in the title by the inverse document frequency in the title to obtain the relevance score of the query keyword and the title of the target document.

[0014] According to the technical solution of the present embodiment, the relevance score is calculated by multiplying the word frequency and the inverse document frequency, which realizes the simple and efficient application of the TF-IDF algorithm, and ensures that the calculation of the title relevance score is accurate and simple.

[0015] According to the document query method of at least one embodiment of the present disclosure, the relevance score of the query keyword and the body of the target document is calculated, including: determining the term frequency of the query keyword in the body of the target document; determining the inverse document frequency of the query keyword in the bodies of multiple documents; and calculating the relevance score of the query keyword and the body of the target document according to the term frequency in the body and the inverse document frequency in the body.

[0016] According to the technical solution of the present embodiment, the relevance of the query keyword and the body of the target document is accurately quantified according to the term frequency and the inverse document frequency, the distinguishing ability of the body in document query is enhanced, the document query result is more in line with the user's intention, and the accuracy of the document query is improved.

[0017] According to the document query method of at least one embodiment of the present disclosure, the term frequency of the query keyword in the body of the target document is determined, including: determining the total number of words in the body of the target document and the position of the query keyword in the body of the target document; determining the number of times the query keyword appears in the body of the target document based on the position of the query keyword in the body of the target document; and dividing the number of times appearing in the body by the total number of words in the body to obtain the term frequency of the query keyword in the body of the target document.

[0018] According to the technical solution of the present embodiment, a term frequency determination algorithm is provided, which determines the number of times the query keyword appears in the title based on the position, provides a reliable data basis for the calculation of the term frequency, ensures the accuracy of the term frequency calculation, and at the same time, reduces the workload of the early generation of the document index process.

[0019] According to the document query method of at least one embodiment of the present disclosure, the inverse document frequency of the query keyword in the bodies of multiple documents is determined, including: determining the body hit by the query keyword from the document index according to the query keyword; and taking the logarithm of the total number of bodies of multiple documents divided by the number of bodies hit by the query keyword to obtain the inverse document frequency of the query keyword in the bodies of multiple documents.

[0020] According to the technical solution of the present embodiment, an inverse document frequency determination algorithm is provided, which quickly determines the body hit by the query keyword through the document index, and then calculates the inverse document frequency of the query keyword in the body according to the total number of bodies of multiple documents and the number of hit bodies, so that the inverse document frequency can be used to judge the rarity of the query keyword in the bodies of the document library, providing accurate data support for document query, and at the same time, reducing the workload of the early generation of the document index process.

[0021] According to at least one embodiment of the document query method of this disclosure, calculating the relevance score between the query keyword and the text of the target document based on the word frequency in the text and the inverse document frequency in the text includes: multiplying the word frequency in the text by the inverse document frequency in the text to obtain the relevance score between the query keyword and the text of the target document.

[0022] According to the technical solution of this embodiment, the relevance score is calculated by multiplying the term frequency by the inverse document frequency, which realizes the simple and efficient application of the TF-IDF algorithm and ensures that the calculation of the text relevance score is both accurate and convenient.

[0023] According to at least one embodiment of the document query method of this disclosure, determining the relevance score between the query keyword and the target document based on the relevance score of the title and the relevance score of the body text includes: performing a weighted summation of the relevance scores of the title and the body text to obtain the relevance score between the query keyword and the target document.

[0024] According to the technical solution of this embodiment, by weighted summing of the relevance scores of the title and the body text, the importance of the title and the body text is comprehensively considered, which can flexibly adapt to the document query needs of different application scenarios and improve the comprehensiveness and adaptability of document query.

[0025] A document query method according to at least one embodiment of the present disclosure generates document query results based on the relevance scores of the target documents, comprising: when there are multiple target documents, sorting the multiple target documents according to the relevance scores of the target documents; and generating the document query results based on the sorting results.

[0026] According to the technical solution of this embodiment, by sorting the target documents according to their relevance scores, the presentation order of document query results is optimized, making it easier for users to quickly obtain the most relevant documents and improving the practicality and user experience of document query results.

[0027] A document query method according to at least one embodiment of the present disclosure generates a document query result based on a sorting result, comprising: determining the position of the query keyword in a plurality of target documents; embedding style tags at the positions to obtain a plurality of embedded target documents, the style tags being used to instruct a display device to highlight the query keyword in the target documents; and arranging the plurality of embedded target documents according to the sorting result to obtain the document query result.

[0028] According to the technical solution of the embodiment, the highlighting of the query keyword is realized by embedding the style label at the position of the query keyword, the visualization effect of the document query result is significantly improved, the user can quickly locate the query keyword in the target document, and the readability and interactivity of the document query result are enhanced. Compared with the scheme without embedding the style label, the positioning efficiency of the query keyword in the displayed target document can be improved by 150%, and the information acquisition experience of the user is significantly improved.

[0029] According to the document query method of at least one embodiment of the present disclosure, before determining the target document hit by the query keyword from the document index, the method further comprises: obtaining a document; parsing the document to determine the title and the body of the document; performing tokenization processing on the title and the body of the document respectively to determine the keywords in the title and the body of the document; and generating the document index according to the keywords, the document identifier, the body identifier, the title identifier corresponding to the keywords, the position information of the keywords in the body, and the position information of the keywords in the title.

[0030] According to the technical solution of the embodiment, by constructing the inverted index containing the title and the body, the double-field keyword retrieval and positioning are supported, and an efficient index generation method is provided, which lays a solid data foundation for subsequent document query tasks.

[0031] According to the document query method of at least one embodiment of the present disclosure, the tokenization processing on the title and the body of the document respectively comprises: obtaining document description information; performing consistency check on the document based on the document description information; and in the case that the consistency check is passed, performing tokenization processing on the title and the body of the document respectively.

[0032] According to the technical solution of the embodiment, the consistency of the document is checked based on the document description information, the accuracy of the obtained document is ensured, and then the tokenization processing is performed in the case that the obtained document is accurate, so that the waste of computing resources caused by meaningless tokenization processing on inaccurate documents is avoided, and the accuracy of the document index is improved, so that the document index is consistent with the document corresponding to the document description information.

[0033] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing a computer program; and a processor executing the computer program stored in the memory, so that the processor executes the document query method of any embodiment of the present disclosure.

[0034] According to still another aspect of the present disclosure, a readable storage medium is provided, the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the document query method of any embodiment of the present disclosure.

[0035] According to yet another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the document query method of any embodiment of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.

[0037] Figure 1 is a flowchart of a document query method according to an embodiment of the present disclosure.

[0038] Figure 2 is a process diagram of calculating a title relevance score according to an embodiment of the present disclosure.

[0039] Figure 3 is a process diagram of calculating a title term frequency according to an embodiment of the present disclosure.

[0040] Figure 4 is a process diagram of calculating a title inverse document frequency according to an embodiment of the present disclosure.

[0041] Figure 5 is a process diagram of calculating a body relevance score according to an embodiment of the present disclosure.

[0042] Figure 6 is a process diagram of calculating a body term frequency according to an embodiment of the present disclosure.

[0043] Figure 7 is a process diagram of calculating a body inverse document frequency according to an embodiment of the present disclosure.

[0044] Figure 8 is a process diagram of generating a document query result according to an embodiment of the present disclosure.

[0045] Figure 9 is a process diagram of generating a document query result according to another embodiment of the present disclosure.

[0046] Figure 10 is a process diagram of generating a document index according to an embodiment of the present disclosure.

[0047] Figure 11 is a process diagram of a word segmentation process according to an embodiment of the present disclosure.

[0048] Figure 12is a flowchart of a document query method according to another embodiment of the present disclosure.

[0049] Figure 13 is a schematic structural block diagram of a document query device according to an embodiment of the present disclosure.

[0050] Figure 14 is a structural schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] The present disclosure will be further described below in conjunction with the accompanying drawings and examples. It can be understood that the specific examples described herein are only for explanation of the related content, and not a limitation of the present disclosure. In addition, it should be noted that only the parts related to the present disclosure are shown in the drawings for ease of description.

[0052] It should be noted that the embodiments and features in the embodiments of the present disclosure can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0053] The inventors have found through careful research that the content of the title in the document is less, and there may be a problem that the title content does not match the content of the body. Therefore, if only the title in the document is used to retrieve and query the document matching the query keyword, it is likely that the content of the body of the queried document does not match the query keyword, causing the user to be troubled by inaccurate document query results and poor document query effect.

[0054] To this end, the present disclosure proposes the following technical solutions, in which the relevance score of the document and the query keyword is determined according to the relevance scores of the query keyword and the title and body of the target document, and the title and body of the document are comprehensively considered, thereby improving the coverage of the document query, increasing the possibility of hitting relevant documents, and thus improving the accuracy of the document query result.

[0055] In order to facilitate the description and make the technical solutions of the present disclosure easier to understand, before describing the technical solutions of the present disclosure, the terms of the present disclosure are first described.

[0056] The inverted index is a data structure that maps a word appearing in a document to the location of the document containing the word.

[0057] The term frequency is the ratio of the number of times a word appears in a specific text to the total number of words in the specific text, reflecting the importance of the word in the specific text.

[0058] The inverse document frequency measures the rarity of a word in the entire document set, and is usually the logarithm of the total number of documents divided by the number of documents containing the word.

[0059] The document query method of the present disclosure can be used for an electronic device to automatically query documents according to a query keyword. In the present disclosure, the electronic device includes, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a personal computer, a wearable device, a teller machine, and the like.

[0060] Figure 1 The overall flowchart of the document query method of one embodiment of the present disclosure is shown. As shown in the method M100 includes steps S110 to S150. The method can be executed by an electronic device such as a mobile phone, a tablet computer, and the like. Figure 1

[0061] In step S110, according to the query keyword, a target document hit by the query keyword is determined from a document index.

[0062] The document index can be pre-generated and stored locally on the electronic device or on other devices that can be interacted with the electronic device, so as to facilitate the electronic device to obtain.

[0063] The plurality of documents that can be queried by the document index can all respectively include a body and a title, or some documents only include a body and the remaining documents respectively include a body and a title.

[0064] In one example, the document index adopts the form of an inverted index, and the document index can include a plurality of keywords and the identification of the keyword-containing document title and / or the identification of the keyword-containing document body mapped by each keyword, that is, in the case where a keyword only appears in a title, the document index only includes the identification of the keyword-containing document title; in the case where a keyword only appears in a body, the document index only includes the identification of the keyword-containing document body; in the case where a keyword appears in a title and a body respectively, the document index respectively includes the identification of the keyword-containing document title and the identification of the keyword-containing document body.

[0065] Exemplarily, the query keyword can be parsed from the received query request, for example, in the case where the query request is a text, the text can be segmented based on an IK segmenter or a pinyin segmenter, so as to obtain one or more query keywords. If the query keyword in the query request is one, the relevance score of the one query keyword and the target document can be directly taken as the relevance score of the target document; if the query keyword in the query request is multiple, the relevance score of each query keyword and the target document can be calculated through the method of steps S120 to S140, and then the sum of the relevance scores of each query keyword and the target document is taken as the relevance score of the target document.

[0066] ​Further, after the query keyword is determined, the query keyword can be accurately matched with the plurality of keywords in the document index. If there is a keyword identical to the query keyword, the document corresponding to the title identifier or the text identifier corresponding to the keyword identical to the query keyword can be determined as the target document hit by the query keyword, that is, the document in which at least one of the title and the text is hit by the query keyword is the target document. If there is no keyword identical to the query keyword, the similar keyword in the plurality of keywords in the document index can be determined by means of fuzzy matching (for example, Wildcard query), and then the document corresponding to the title identifier or the text identifier corresponding to the similar keyword can be determined as the target document hit by the query keyword.

[0067] In step S120, the relevance score of the query keyword and the title of the target document is calculated.

[0068] Exemplarily, the query keyword and the title of the target document can be respectively converted into vectors, and then the similarity between the vectors is taken as the relevance score of the title. Other algorithms for calculating the relevance score of the title can also be used, which are not limited herein.

[0069] In step S130, the relevance score of the query keyword and the text of the target document is calculated.

[0070] Exemplarily, the query keyword and the text of the target document can be respectively converted into vectors, and then the similarity between the vectors is taken as the relevance score of the text. Other algorithms for calculating the relevance score of the text can also be used, which are not limited herein.

[0071] It is worth noting that step S120 can be performed before step S130, after step S130, or simultaneously with step S130, which is not limited herein.

[0072] In step S140, the relevance score of the query keyword and the target document is determined according to the relevance score of the title and the relevance score of the text.

[0073] In the case where the target document is multiple, the relevance score of the query keyword and the title of each target document in the plurality of target documents is calculated, and the relevance score of the query keyword and the text of each target document in the plurality of target documents is calculated, and then for each target document, the relevance score of the target document is determined according to the relevance score of the title and the relevance score of the text of the target document. The relevance score of the target document can more accurately represent the relevance between the target document and the query keyword.

[0074] In the case where there are multiple titles in one target document, the relevance scores of the titles can be calculated respectively, and the sum of the relevance scores of the titles is taken as the title relevance score of the target document. The text relevance score is calculated in the same way.

[0075] In step S150, the document query result is generated according to the relevance score of the target document.

[0076] After obtaining the document query result, the document query result can be displayed to the user for review.

[0077] The document query method of the embodiments of the present disclosure calculates the relevance score of the target document and the query keyword according to the relevance scores of the query keyword and the title and text of the target document, and generates the document query result according to the relevance score of the target document, thereby comprehensively considering the title and text of the document, improving the coverage of the document query, increasing the possibility of hitting relevant documents, and enabling more documents related to the query keyword to be retrieved, thereby improving the comprehensiveness and accuracy of the document query. Compared with the related technical solutions, the coverage of the document query of the present method is improved by 300%, significantly improving the information acquisition experience of the user.

[0078] In addition, in some embodiments, a monitoring board based on Prometheus and Grafana can also be constructed to form a full-link quality monitoring system, so that the background management personnel can understand the running state of the document query method in real time. The quality monitoring indicators can include one or more of data synchronization delay, index freshness, and highlight rendering accuracy. In one example, the data synchronization delay reaches SLI≤1s, the index freshness reaches 99.9% of the documents can be used for document query within 5min after input, and the highlight rendering accuracy reaches F1-score≥0.98. The stress test of the document index engine can use JMeter to simulate 100 concurrent queries, and the fault self-healing ability is verified through chaos engineering.

[0079] Regarding step S120, in some embodiments of the present disclosure, steps S121 to S123 as shown in Figure 2 may be included.

[0080] In step S121, the word frequency of the query keyword in the title of the target document is determined.

[0081] In one example, the word frequency of each keyword can be included in the document index, and thus the word frequency of the query keyword in the title of the target document can be directly obtained from the document index, thereby improving the document query efficiency.

[0082] In another example, the word frequency of the query keyword in the title of the target document can be calculated on site by a word frequency determination algorithm, thereby reducing the workload of the pre-generated document index process.

[0083] In step S122, the inverse document frequency of the query keyword in the title of the plurality of documents is determined.

[0084] In one example, the inverse document frequency of each keyword can be included in the document index, and then the inverse document frequency of the query keyword in the title of the plurality of documents can be directly obtained from the document index, thereby improving the document query efficiency.

[0085] In another example, the inverse document frequency of the query keyword in the title of the plurality of documents can be calculated on site by an inverse document frequency determination algorithm, thereby reducing the workload of the pre-generated document index process.

[0086] In step S123, the relevance score of the query keyword to the title of the target document is calculated according to the word frequency in the title and the inverse document frequency in the title.

[0087] Exemplarily, after the word frequency and the inverse document frequency are determined, the relevance score of the query keyword to the title of the target document can be calculated based on a relevance score calculation algorithm in the related art, which includes but is not limited to a term frequency-inverse document frequency (TF-IDF) algorithm or a best match 25 (BM25) algorithm.

[0088] The document query method of the above-mentioned embodiment accurately quantifies the relevance of the query keyword to the title of the target document according to the word frequency and the inverse document frequency, enhances the distinguishing ability of the title in the document query, makes the document query result more in line with the user's intention, and improves the accuracy of the document query.

[0089] Regarding step S121, in some embodiments of the present disclosure, steps S1211 to S1213 as shown in Figure 3 may be included.

[0090] In step S1211, the total number of words in the title of the target document and the position of the query keyword in the title of the target document are determined.

[0091] The total number of words in the title of the target document and the position of the query keyword in the title of the target document can be obtained from the document index.

[0092] In step S1212, the number of times the query keyword appears in the title of the target document is determined based on the position of the query keyword in the title of the target document.

[0093] Exemplarily, one position can be considered as one occurrence, and then the number of positions of the query keyword in the title of the target document can be taken as the number of occurrences of the query keyword in the title of the target document.

[0094] In step S1213, the number of occurrences in the title is divided by the total number of words in the title to obtain the term frequency of the query keyword in the title of the target document.

[0095] Exemplarily, the value obtained after dividing the number of occurrences by the total number of words can be taken as the term frequency.

[0096] The document query method of the above-mentioned embodiment provides a term frequency determination algorithm, which determines the number of occurrences of the query keyword in the title based on the position, provides a reliable data basis for the calculation of the term frequency, ensures the accuracy of the calculation of the term frequency, and reduces the workload of the pre-generated document index process.

[0097] Regarding step S122, in some embodiments of the present disclosure, steps S1221 to S1222 as shown in the following table can be included. Figure 4

[0098] In step S1221, the title hit by the query keyword is determined from the document index according to the query keyword.

[0099] The document index can include the title identifier corresponding to each keyword, and then the title hit by the query keyword can be directly obtained from the document index.

[0100] In step S1222, the total number of titles of multiple documents is divided by the number of titles hit by the query keyword, and then the logarithm is taken to obtain the inverse document frequency of the query keyword in the titles of the multiple documents.

[0101] The document query method of the above-mentioned embodiment provides an inverse document frequency determination algorithm, which quickly determines the title hit by the query keyword through the document index, and then calculates the inverse document frequency of the query keyword in the title according to the total number of titles of multiple documents and the number of hit titles, so that the rarity of the query keyword in the title of the document library can be judged through the inverse document frequency, accurate data support is provided for the document query, and the workload of the pre-generated document index process is reduced.

[0102] Regarding step S123, in some embodiments of the present disclosure, the term frequency in the title can be multiplied by the inverse document frequency in the title to obtain the relevance score of the query keyword and the title of the target document.

[0103] Exemplarily, the product value of the term frequency of the keyword in the title of the target document and the inverse document frequency of the query keyword in the titles of the multiple documents is the relevance score of the query keyword and the title of the target document.​

[0104] The document query method of the above embodiment realizes the application of the simple and efficient TF-IDF algorithm by multiplying the term frequency and the inverse document frequency to calculate the relevance score, and ensures that the calculation of the title relevance score is both accurate and simple.

[0105] Regarding step S130, in some embodiments of the present disclosure, steps S131-S133 as shown in Figure 5 may be included.

[0106] In step S131, the term frequency of the query keyword in the body of the target document is determined.

[0107] In step S132, the inverse document frequency of the query keyword in the body of the plurality of documents is determined.

[0108] In step S133, the relevance score of the query keyword and the body of the target document is calculated according to the term frequency in the body and the inverse document frequency in the body.

[0109] The content related to steps S131-S133 can refer to the description of steps S121-S123, and for the sake of brevity, will not be repeated here.

[0110] The document query method of the above embodiment accurately quantifies the relevance of the query keyword and the body of the target document according to the term frequency and the inverse document frequency, enhances the distinguishing ability of the body in document query, makes the document query result more in line with the user's intention, and improves the accuracy of document query.

[0111] Regarding step S131, in some embodiments of the present disclosure, steps S1311-S1313 as shown in Figure 6 may be included.

[0112] In step S1311, the total number of words in the body of the target document and the position of the query keyword in the body of the target document are determined.

[0113] In step S1312, the number of times the query keyword appears in the body of the target document is determined based on the position of the query keyword in the body of the target document.

[0114] In step S1313, the number of times the query keyword appears in the body is divided by the total number of words in the body to obtain the term frequency of the query keyword in the body of the target document.

[0115] The content related to steps S1311-S1313 can refer to the description of steps S1211-S1213, and for the sake of brevity, will not be repeated here.

[0116] The document query method described above provides a word frequency determination algorithm, which determines the number of times the query keyword appears in the title based on its position, providing a reliable data foundation for word frequency calculation, ensuring the accuracy of word frequency calculation, and reducing the workload of the early document index generation process.

[0117] Regarding step S132, in some embodiments of this disclosure, it may include, for example... Figure 7 Steps S1321 to S1322 are shown.

[0118] In step S1321, the text matching the query keywords is determined from the document index based on the query keywords.

[0119] In step S1322, the total number of texts in multiple documents is divided by the number of texts in which the query keyword is matched, and then the logarithm is taken to obtain the inverse document frequency of the query keyword in the texts of multiple documents.

[0120] The content related to steps S1321 to S1322 can be found in the description of steps S1221 to S1222, and will not be repeated here for the sake of brevity.

[0121] The document query method described above provides an inverse document frequency (IVF) determination algorithm. This algorithm quickly determines the text in which the query keyword is matched by the document index, and then calculates the IVF of the query keyword in the text based on the total number of texts in multiple documents and the number of matched texts. Thus, the IVF can be used to determine the rarity of the query keyword in the text of the document database, providing accurate data support for document querying. At the same time, it reduces the workload of generating the document index in the early stage.

[0122] Regarding step S133, in some embodiments of this disclosure, it may include: multiplying the word frequency in the text by the inverse document frequency in the text to obtain the relevance score between the query keyword and the text of the target document.

[0123] For example, the product of the term frequency of the keyword in the body of the target document and the inverse document frequency of the query keyword in the body of multiple documents is the relevance score between the query keyword and the body of the target document.

[0124] The document query method described above calculates the relevance score by multiplying the term frequency by the inverse document frequency, thus realizing a simple and efficient application of the TF-IDF algorithm and ensuring that the calculation of the text relevance score is both accurate and convenient.

[0125] Regarding step S140, in some embodiments of this disclosure, it may include: weighted summation of the relevance score of the title and the relevance score of the body text to obtain the relevance score between the query keywords and the target document.

[0126] Exemplarily, the relevance score of the query keyword to the target document = the relevance score of the title x the first weight + the relevance score of the body text x the second weight, and the first weight and the second weight can be set according to requirements.

[0127] In one example, the first weight is 3 times the second weight, so that the relevance score of the target document is more focused on the matching of the title and the query keyword, and the accuracy of the document query is improved.

[0128] It should be noted that the specific numerical values mentioned above are only for the purpose of illustrating the embodiments of the present disclosure in detail, and should not be understood as a limitation of the present disclosure. In other examples or implementations or embodiments, other numerical values can be selected according to the present disclosure, which are not specifically limited here.

[0129] The document query method of the above-mentioned embodiments can flexibly adapt to the document query requirements of different application scenarios by weighted sum of the relevance score of the title and the relevance score of the body text, and comprehensively consider the importance of the title and the body text, thereby improving the comprehensiveness and adaptability of the document query.

[0130] Regarding step S150, in some embodiments of the present disclosure, steps S151 and S152 as shown in Figure 8 may be included.

[0131] In step S151, in the case where the number of target documents is multiple, the multiple target documents are sorted according to the relevance scores of the target documents.

[0132] Exemplarily, the multiple target documents can be sorted in descending order of the relevance scores of the target documents.

[0133] In step S152, the document query result is generated according to the sorting result.

[0134] All the sorted target documents can be used as the document query result, or the top K target documents after sorting can be used as the document query result, or the target documents are further processed (for example, adding style tags and / or conditional filtering) to obtain the document query result, which is not limited here.

[0135] The document query method of the above-mentioned embodiments can optimize the presentation order of the document query result by sorting the target documents according to the relevance scores of the target documents, facilitate the user to quickly obtain the most relevant document, and improve the practicality and user experience of the document query result.

[0136] Regarding step S152, in some embodiments of the present disclosure, steps S1521 to S1523 as shown in Figure 9 may be included.

[0137] In step S1521, the positions of the query keywords in the plurality of target documents are determined.

[0138] The document index can include the position information of each keyword in the title and the body of each document, and thus the positions of the query keywords in the plurality of target documents can be obtained through the document index.

[0139] In step S1522, a style tag is embedded at the positions, to obtain a plurality of embedded target documents, the style tag being used to instruct the display device to highlight the query keywords in the target documents.

[0140] The style tag can be set according to requirements.

[0141] In one example, an HTML start tag is embedded before the starting character position of each query keyword in the target document and embedding HTML end tags after the end character position of each query keyword in the target document , to obtain the target document after the HTML / CSS style tag is embedded. The highlight is a CSS style class, which can include the following style attributes: the background color is set to red or a high-contrast color, and / or, the font color or bold style is optional.

[0142] In another example, the highlighting of the query keywords in the target document can be implemented through the unified highlighting strategy and / or the fast-vector highlighting strategy in the related art, and the response time is controlled within 500 ms (99% percentile), so as to improve the display effect of the document query result and facilitate reading.

[0143] In step S1523, the plurality of embedded target documents are arranged according to the sorting result, to obtain the document query result.

[0144] The sorted embedded target documents can be used as the document query result, or the first K sorted embedded target documents can be used as the document query result, or the embedded target documents are further processed (for example, conditional filtering) to obtain the document query result, which is not limited herein.

[0145] The document query method of the above embodiment realizes the highlighting of the query keywords by embedding a style tag at the positions of the query keywords, significantly improves the visualization effect of the document query result, enables the user to quickly locate the query keywords in the target document, and enhances the readability and interactivity of the document query result. Compared with the scheme without embedding the style tag, the positioning efficiency of the query keywords in the displayed target document can be improved by 150% by using the method, and the information acquisition experience of the user is significantly improved.

[0146] In some embodiments of the present disclosure, before step S110, the method can further include the following steps: Figure 10The steps S160 to S190 are shown.

[0147] In step S160, the document is acquired.

[0148] Exemplarily, the document can be acquired through historical document data and / or newly added document data. The newly added document data can be a document newly input by a relevant person or a document newly published by a document publishing platform.

[0149] The format of the document can be a pdf format or a word format.

[0150] In step S170, the document is parsed to determine the title and the body of the document.

[0151] In the case where the document is in a pdf format, the document can be converted into a text format through parsing.

[0152] Exemplarily, the pdfdocument tool can be called to parse the document first. If the parsing is successful, the title information and the body information in the document can be determined. If the parsing fails, the itext tool can be called to parse the document, so as to ensure the smooth determination of the title information and the body information through double-engine redundant processing, and improve the data integrity.

[0153] In step S180, the title and the body of the document are respectively processed for word segmentation to determine the keywords in the title and the body of the document.

[0154] The word segmentation processing can be implemented through a word segmentation algorithm in the related art.

[0155] In step S190, a document index is generated according to the keywords and the document identifier, the body identifier, the title identifier, the position information of the keywords in the body, and the position information of the keywords in the title corresponding to the keywords.

[0156] Exemplarily, the document index and the document can be stored in an Elasticsearch search engine in a multi-shard distributed manner, so as to improve the efficiency of subsequent document query.

[0157] The document index can be an inverted index, and the title (text / keyword double field), the body (text), and the metadata field can be defined through a custom mapping (mapping).

[0158] The document index can reserve a refresh_interval parameter interface, so as to support subsequent cold and hot data hierarchical storage optimization.

[0159] Exemplarily, a change data capture (CDC) listening module can be constructed based on a database transaction log (binlog), and a change event asynchronous distribution is achieved through events. Then, after generating a document index, if it is detected that a new document is input by a relevant person or a new document is published by a document publishing platform, the relevant information of the new document input by the relevant person or the new document published by the document publishing platform is synchronized to the document index, so as to realize automatic update of the document index. In addition, the synchronization rate can be controlled based on a back pressure mechanism, so as to ensure that the throughput is stable at 2000+ docs / sec during peak period.

[0160] The document query method of the above embodiment supports double-field keyword search and positioning by constructing an inverted index containing a title and a body, and provides an efficient index generation method, thereby laying a solid data foundation for subsequent document query tasks.

[0161] As to step S180, in some embodiments of the present disclosure, steps S181 to S183 as shown in the following can also be included. Figure 11

[0162] In step S181, document description information is acquired.

[0163] Exemplarily, the document description information of each document can be pre-stored in a database, and then the document description information can be acquired from the database.

[0164] The document description information can include one or more of a title, a storage path and a document id of the document.

[0165] In step S182, consistency check is performed on the document based on the document description information.

[0166] Exemplarily, the document includes a title, and the document description information also includes the title of the document. The title in the document and the title in the document description information can be respectively converted into hash values. Then, if the hash values obtained by the conversion are the same, it is determined that the consistency check passes; otherwise, it is determined that the consistency check fails.

[0167] In the case of consistency check failure, an exception log can be recorded, and a relevant person is prompted to handle manually.

[0168] In step S183, in the case of passing the consistency check, the title and the body of the document are respectively subjected to word segmentation processing.

[0169] ​The document query method of the above-mentioned embodiments performs consistency check on the document based on the document description information, ensures the accuracy of the obtained document, and then performs word segmentation processing in the case that the obtained document is accurate, thereby avoiding waste of computing resources caused by meaningless word segmentation processing on inaccurate documents, improving the accuracy of the document index, and ensuring that the document index corresponds to the document described by the document description information.

[0170] Please refer to Figure 12 In one example, the document query method can include the following steps S201 to S216, and the content related to steps S201 to S216 can refer to the description of the above-mentioned embodiments. For the sake of brevity, the details are not repeated here.

[0171] In step S201, the document and the document description information are obtained.

[0172] In step S202, the document is parsed to determine the title and the body of the document.

[0173] In step S203, the document is subjected to consistency check based on the document description information.

[0174] In step S204, it is determined whether the consistency check is passed. If yes, step S205 is entered, otherwise step S206 is entered.

[0175] In step S205, in the case that the consistency check is passed, the title and the body of the document are subjected to word segmentation processing respectively to determine the keywords in the title and the body of the document.

[0176] In step S206, in the case that the consistency check is not passed, an abnormal log is recorded and relevant personnel is prompted to handle manually.

[0177] In step S207, the document index is generated according to the keywords, the document identifier corresponding to the keywords, the body identifier, the title identifier, the position information of the keywords in the body, and the position information of the keywords in the title.

[0178] In step S208, the query request is received.

[0179] In step S209, the query keywords are extracted from the query request.

[0180] In step S210, the target document hit by the query keywords is determined from the document index according to the query keywords.

[0181] In step S211, the relevance score of the query keywords and the title of the target document is calculated.

[0182] In step S212, the relevance score of the query keywords and the body of the target document is calculated.

[0183] In step S213, the relevance score of the query keyword and the target document is determined according to the relevance score of the title and the relevance score of the body.

[0184] In step S214, the plurality of target documents are sorted according to the relevance scores of the target documents.

[0185] In step S215, style tags are embedded at the positions of the query keyword in the plurality of target documents, to obtain a plurality of embedded target documents.

[0186] In step S216, the plurality of embedded target documents are arranged according to the sorting result, to obtain the document query result.

[0187] Based on any one of the above embodiments, the present disclosure further provides a document query device.

[0188] Figure 13 FIG. 1 is a structural schematic block diagram of a document query device according to an embodiment of the present disclosure.

[0189] As shown in FIG. 1, the document query device comprises a first determining module 110, a first calculating module 120, a second calculating module 130, a second determining module 140, and a generating module 150. Figure 13 The first determining module 110 is configured to determine, according to a query keyword, target documents hit by the query keyword from a document index. The first calculating module 120 is configured to calculate a relevance score of the query keyword and a title of the target document. The second calculating module 130 is configured to calculate a relevance score of the query keyword and a body of the target document. The second determining module 140 is configured to determine, according to the relevance score of the title and the relevance score of the body, the relevance score of the query keyword and the target document. The generating module 150 is configured to generate a document query result according to the relevance score of the target document.

[0190] The above document query device can be in the form of computer software, and each module of the above document query device can be realized by a computer software module.

[0191] In some embodiments of the present disclosure, the first calculating module 120 is configured to determine a term frequency of the query keyword in the title of the target document, determine an inverse document frequency of the query keyword in the titles of the plurality of documents, and calculate the relevance score of the query keyword and the title of the target document according to the term frequency in the title and the inverse document frequency in the title.

[0192] In some embodiments of the present disclosure, the first calculating module 120 is configured to: determine the total number of words in the title of the target document and the position of the query keyword in the title of the target document; determine the number of occurrences of the query keyword in the title of the target document based on the position of the query keyword in the title of the target document; and divide the number of occurrences in the title by the total number of words in the title to obtain the term frequency of the query keyword in the title of the target document.

[0193] In some embodiments of the present disclosure, the first calculating module 120 is configured to: determine the title hit by the query keyword from the document index according to the query keyword; and divide the total number of titles of the plurality of documents by the number of titles hit by the query keyword to obtain the inverse document frequency of the query keyword in the titles of the plurality of documents.

[0194] In some embodiments of the present disclosure, the first calculating module 120 is configured to: multiply the term frequency in the title by the inverse document frequency in the title to obtain the relevance score of the query keyword to the title of the target document.

[0195] In some embodiments of the present disclosure, the second calculating module 130 is configured to: determine the term frequency of the query keyword in the body of the target document; determine the inverse document frequency of the query keyword in the bodies of the plurality of documents; and calculate the relevance score of the query keyword to the body of the target document according to the term frequency in the body and the inverse document frequency in the body.

[0196] In some embodiments of the present disclosure, the second calculating module 130 is configured to: determine the total number of words in the body of the target document and the position of the query keyword in the body of the target document; determine the number of occurrences of the query keyword in the body of the target document based on the position of the query keyword in the body of the target document; and divide the number of occurrences in the body by the total number of words in the body to obtain the term frequency of the query keyword in the body of the target document.

[0197] In some embodiments of the present disclosure, the second calculating module 130 is configured to: determine the body hit by the query keyword from the document index according to the query keyword; and divide the total number of bodies of the plurality of documents by the number of bodies hit by the query keyword to obtain the inverse document frequency of the query keyword in the bodies of the plurality of documents.

[0198] In some embodiments of the present disclosure, the second calculating module 130 is configured to: multiply the term frequency in the body by the inverse document frequency in the body to obtain the relevance score of the query keyword to the body of the target document.

[0199] In some embodiments of the present disclosure, the second determining module 140 is configured to: perform weighted summation on the relevance score of the title and the relevance score of the body to obtain the relevance score of the query keyword to the target document.

[0200] In some embodiments of the present disclosure, the generating module 150 is configured to: in the case that the number of target documents is multiple, rank the multiple target documents according to the relevance scores of the target documents; and generate the document query result according to the ranking result.

[0201] In some embodiments of the present disclosure, the generating module 150 is configured to: determine the positions of the query keywords in the multiple target documents; embed a style tag in the positions, to obtain multiple embedded target documents, the style tag being used to instruct the display device to highlight the query keywords in the target documents; and arrange the multiple embedded target documents according to the ranking result, to obtain the document query result.

[0202] In some embodiments of the present disclosure, the document query apparatus further comprises: an obtaining module configured to obtain a document; an analyzing module configured to analyze the document to determine a title and a body of the document; a word segmentation module configured to respectively perform word segmentation processing on the title and the body of the document to determine keywords in the title and the body of the document; and a second generating module configured to generate a document index according to the keywords and document identifiers, body identifiers, title identifiers, position information of the keywords in the body, and position information of the keywords in the title corresponding to the keywords.

[0203] In some embodiments of the present disclosure, the word segmentation module is configured to: obtain document description information; perform consistency verification on the document based on the document description information; and in the case that the consistency verification is passed, respectively perform word segmentation processing on the title and the body of the document.

[0204] The implementation processes of the functions and roles of the various modules in the above apparatus are specifically described in the implementation processes of the corresponding steps in the above method, and will not be described here.

[0205] The execution subject of the document query method in the specific embodiments of the present disclosure can be a mobile phone, a computer, or other electronic devices.

[0206] Therefore, based on any one of the above embodiments, the present disclosure further provides an electronic device which can execute the document query method of any one of the above embodiments of the present disclosure.

[0207] Figure 14 FIG. 1 is a structural schematic block diagram of an electronic device 1000 according to an embodiment of the present disclosure.

[0208] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and on overall design constraints. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. The bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connection line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0209] For ease of explanation, some steps of the above method are described in correspondence with modules. It should be understood that the corresponding module performing one or more steps of the above method can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer readable medium for implementation by a processor, or implemented by some combination.

[0210] The present disclosure also provides a readable storage medium having a computer program stored therein, the computer program being executed by a processor to implement the above method. The "readable storage medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in conjunction with an instruction execution system, apparatus or device. More specific examples of readable storage media include the following: an electrical connection having one or more wires (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM), etc.

[0211] The present disclosure also provides a computer program product, and the method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the flow or function of the present disclosure is executed in whole or in part.

[0212] The computer program or instructions can be stored in or transferred from one computer-readable medium to another, e.g., from one website or computer to another website or computer, by way of wired or wireless communication. The computer-readable medium can be any available medium or a combination of one or more of the available media, which is accessible by a computer or a data storage device such as a server, a data center, etc. The available medium can be a magnetic medium, e.g., a floppy diskette, a hard disk drive, a magnetic tape; an optical medium, e.g., a compact disk, a DVD; or a semiconductor medium, e.g., a solid state disk. The computer-readable medium can be a volatile or non-volatile medium, or can include both volatile and non-volatile media.

[0213] Those skilled in the art will appreciate that embodiments of the disclosure can be supplied as a method, a device, or a computer program product. Accordingly, the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the disclosure can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, magnetic disks, CD-ROMs, optical storage media, etc.) embodying computer program code.

[0214] The disclosure is described in reference to the flowchart and / or block diagrams of the method, apparatus, and computer program product according to the disclosure. It should be understood that each flow and / or block in the flowchart and / or block diagrams, and a combination of flows and / or blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowchart and / or block diagrams. Figure 1 The flow or flows and / or blocks in a flowchart and / or block diagrams Figure 1 The flow or flows and / or blocks in a flowchart and / or block diagrams

[0215] The computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which are executed in the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagrams. Figure 1 The flow or flows and / or blocks in a flowchart and / or block diagrams Figure 1 The flow or flows and / or blocks in a flowchart and / or block diagrams

[0216] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices, to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide the function of realizing the processes specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0217] In the description of the present specification, the description of the terms "one embodiment / way", "some embodiments / ways", "example", "specific example", or "some examples" and the like means that the specific features, structures, or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present disclosure. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / way or example. Also, the specific features, structures, or characteristics described can be combined in any appropriate manner in any one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples without contradiction.

[0218] The person skilled in the art should understand that the above-mentioned embodiments are only for clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. Based on the above disclosure, other changes or modifications can also be made by the person skilled in the art, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A document retrieval method, characterized in that, include: Based on the query keywords, determine the target documents matched by the query keywords from the document index; Calculate the relevance score between the query keywords and the title of the target document; Calculate the relevance score between the query keywords and the main text of the target document; Based on the relevance score of the title and the relevance score of the body text, determine the relevance score between the query keywords and the target document; as well as Document query results are generated based on the relevance score of the target document.

2. The document query method according to claim 1, characterized in that, Calculate the relevance score between the query keywords and the title of the target document, including: Determine the word frequency of the query keywords in the title of the target document; Determine the inverse document frequency of the query keyword in the titles of multiple documents; and The relevance score between the query keyword and the title of the target document is calculated based on the word frequency and inverse document frequency in the title.

3. The document query method according to claim 1, characterized in that, Calculating the relevance score between the query keywords and the body of the target document includes: Determine the word frequency of the query keywords in the body of the target document; Determine the inverse document frequency of the query keyword in the body of multiple documents; and The relevance score between the query keyword and the text of the target document is calculated based on the term frequency and inverse document frequency in the text.

4. The document query method according to claim 1, characterized in that, Based on the relevance score of the title and the relevance score of the body text, the relevance score between the query keywords and the target document is determined, including: The relevance scores of the title and the body text are weighted and summed to obtain the relevance score between the query keywords and the target document.

5. The document query method according to claim 1, characterized in that, Document query results are generated based on the relevance score of the target document, including: When there are multiple target documents, the target documents are sorted according to their relevance scores; and The document query results are generated based on the sorting results.

6. The document query method according to claim 5, characterized in that, The document query results are generated based on the sorting results, including: Determine the position of the query keyword in multiple target documents; Embedding style tags at the specified locations yields multiple embedded target documents. These style tags instruct the display device to highlight the query keywords within the target documents. The multiple embedded target documents are arranged according to the sorting result to obtain the document query result.

7. The document query method according to claim 1, characterized in that, Before determining the target document that the query keywords hit from the document index, the process also includes: Get the document; The document is parsed to determine its title and body text; The title and body of the document are segmented into words to determine the keywords in the title and body; and The document index is generated based on the keywords and their corresponding document identifiers, body identifiers, title identifiers, the position information of the keywords in the body text, and the position information of the keywords in the title.

8. The document query method according to claim 7, characterized in that, The title and body of the document are segmented into words, including: Retrieve document description information; The document is subjected to consistency verification based on the document description information; and If the consistency check passes, the title and body of the document are segmented into words respectively.

9. An electronic device, characterized in that, include: Memory, the memory storing computer programs; as well as A processor that executes a computer program stored in the memory, causing the processor to perform the document query method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the document query method according to any one of claims 1 to 8.

Images