Document retrieval method, device, equipment and storage medium

Abstract

The application relates to the technical field of data query, and discloses a document retrieval method, device and equipment and a storage medium, the method comprising the following steps: obtaining a target word string of a search term input by a user, and determining the semantic importance of the target word string; obtaining a candidate document matched with the target word string, and determining the correlation between the target word string and the candidate document; optimizing the correlation according to the semantic importance to obtain an optimized correlation; and displaying the candidate document after the candidate document is sorted according to the optimized correlation. By determining the semantic importance of the target word string and optimizing the correlation between the target word string and the candidate document according to the semantic importance, the application avoids the situation that when the candidate document is a document matched with a word with low importance, the candidate document is also displayed if the correlation of the candidate document is high, makes the retrieval result more in line with the user demand, and effectively improves the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data query, and in particular to a document retrieval method, device, equipment and storage medium. BACKGROUND

[0002] With the development of Internet technology, information on the Internet is growing explosively, and more and more users search for the required documents through the network. The document retrieval method in the related art is to query and retrieve candidate documents matched with each vocabulary in the search term, then calculate the relevance of each candidate document and the matched vocabulary, and sort and display according to the relevance of each candidate document.

[0003] However, in the above-mentioned method, when the candidate document is a document matched with a vocabulary of low importance, if the relevance of the candidate document is high, the candidate document will also be displayed, which leads to the fact that the retrieval result cannot meet the user's demand and affects the user experience. SUMMARY

[0004] The main purpose of the present application is to provide a document retrieval method, device, equipment and storage medium, which aims to solve the technical problem that the prior art cannot meet the user's expectation and affects the user experience when the candidate document is a document matched with a vocabulary of low importance and the relevance of the candidate document is high.

[0005] To achieve the above-mentioned purpose, the present application provides a document retrieval method, which is applied to an online ranking model, and the method comprises the following steps:

[0006] obtaining a target word string of a search term input by a user, and determining the semantic importance of the target word string;

[0007] obtaining a candidate document matched with the target word string, and determining the relevance between the target word string and the candidate document;

[0008] optimizing the relevance according to the semantic importance to obtain an optimized relevance;

[0009] displaying the candidate document after sorting according to the optimized relevance.

[0010] Optionally, the step of determining the semantic importance of the target word string comprises:

[0011] obtaining the importance of each word in the target word string relative to the original words of the search term;

[0012] determining the length of the segmented sentence of the search term according to the number of each word;

[0013] The original word importance is optimized based on the length of the segmented sentence to obtain semantic importance of the target word string.

[0014] Optionally, the step of optimizing the original word importance based on the length of the segmented sentence to obtain semantic importance of the target word string comprises:

[0015] The original word importance is uniformly processed based on the length of the segmented sentence by a preset word importance optimization formula to obtain semantic importance of the target word string, wherein the preset word importance optimization formula is:

[0016] W = Important (Sent) * len (Sent),

[0017] In the formula, W is the uniform word importance, Important (C) is the original word importance, and len (Sent) is the length of the segmented sentence.

[0018] Optionally, the step of optimizing the relevance based on the semantic importance to obtain the optimized relevance comprises:

[0019] The relevance is optimized based on the semantic importance by a preset relevance optimization formula to obtain optimized segment relevance of each segment, wherein the preset relevance optimization formula is:

[0020] S tw = W i *S i ,

[0021] In the formula, S tw is the optimized segment relevance, W i is the uniform word importance of word i in the target word string, and S i is the relevance of word i in the target word string and the candidate document.

[0022] The optimized segment relevance of each segment is superimposed by a preset relevance superimposition formula to obtain the optimized relevance of the target word string, wherein the preset relevance superimposition formula is:

[0023]

[0024] In the formula, R doc is the optimized relevance, and S tw is the optimized segment relevance.

[0025] Optionally, the step of obtaining the candidate document matched with the target word string comprises:

[0026] query a preset inverted database to obtain matching word strings matched with each word in the target word string;

[0027] concurrently query a preset forward database based on the matching word strings to obtain matching documents matched with the matching word strings;

[0028] recall the matching documents and take the matching documents as candidate documents matched with the target word string.

[0029] Optionally, before the step of obtaining the candidate documents matched with the target word string, the method further comprises:

[0030] obtain a sample document, parse the sample document according to a corresponding parsing strategy selected according to the format of the sample document, and obtain a parsed document with a unified format;

[0031] perform word segmentation on the title of the parsed document by using a multi-granularity word segmentation granularity, and obtain an inverted index;

[0032] construct a preset inverted database based on the inverted index, and construct a preset forward database according to a target sample document corresponding to the inverted index.

[0033] Optionally, the construction of the online ranking model comprises:

[0034] obtain initial data composed of preset documents, and label the initial data according to preset industry keywords in the preset documents to obtain labeled data;

[0035] train a transformer model by using the labeled data to obtain an offline ranking model, wherein the output result of the offline ranking model is a single-layer result obtained by merging the output results of the last two layers of the transformer model;

[0036] predict preset unlabeled document data based on the offline ranking model to obtain a sample relevance result of the preset unlabeled document data;

[0037] train an XGBoost model according to the sample relevance result to obtain an online ranking model.

[0038] In addition, to achieve the above object, the application further provides a document retrieval device, which comprises:

[0039] a semantic importance module configured to obtain a target word string of a retrieval word input by a user and determine the semantic importance of the target word string;

[0040] a document relevance module configured to obtain candidate documents matched with the target word string and determine the relevance between the target word string and the candidate documents.

[0041] a relevance degree optimization module configured to optimize the relevance degree according to the semantic importance degree, to obtain an optimized relevance degree;

[0042] a candidate document display module configured to display the candidate documents after sorting according to the optimized relevance degree.

[0043] In addition, to achieve the above object, the present application further provides a document retrieval device, which comprises a memory, a processor and a document retrieval program stored in the memory and executable on the processor, and the document retrieval program is configured to implement the steps of the document retrieval method as described above.

[0044] In addition, to achieve the above object, the present application further provides a storage medium, which stores a document retrieval program, and the document retrieval program is executable on a processor to implement the steps of the document retrieval method as described above.

[0045] The present application provides a document retrieval method, device, equipment and storage medium, which obtains a target word string of a search term input by a user, and determines a semantic importance degree of the target word string; then obtains candidate documents matched with the target word string, and determines a relevance degree between the target word string and the candidate documents; finally, the relevance degree is optimized according to the semantic importance degree, to obtain an optimized relevance degree; and the candidate documents are displayed after sorting according to the optimized relevance degree. By determining the semantic importance degree of the target word string, and optimizing the relevance degree between the target word string and the candidate documents according to the semantic importance degree, the present application avoids the situation that when the candidate documents are documents matched with a word with a low importance degree, the candidate documents are also displayed if the relevance degree of the candidate documents is high, so that the search result is more in line with the user's demand, and the user experience is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A document retrieval device structure schematic diagram of a hardware running environment related to an embodiment of the present application;

[0047] Figure 2 A flowchart of a first embodiment of the document retrieval method of the present application;

[0048] Figure 3 A flowchart of a second embodiment of the document retrieval method of the present application;

[0049] Figure 4 A flowchart of a third embodiment of the document retrieval method of the present application;

[0050] Figure 5 A document retrieval overall flowchart of the third embodiment of the document retrieval method of the present application;

[0051] Figure 6 Fig. 13 is a schematic diagram of output results of an offline ranking model in the third embodiment of the document retrieval method of the present application;

[0052] Figure 7 Fig. 14 is a schematic diagram of online ranking model training in the third embodiment of the document retrieval method of the present application;

[0053] Figure 8 Fig. 1 is a structure block diagram of the first embodiment of the document retrieval device of the present application.

[0054] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the present application.

[0056] Reference Figure 1 , Figure 1 Fig. 2 is a structure schematic diagram of a document retrieval device related to the hardware running environment of the embodiment of the present application.

[0057] As Figure 1 shown, the document retrieval device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 can include a display screen, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 can be a high-speed random access memory (RAM), and can also be a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art can understand that Figure 1 the structure shown in the foregoing embodiments does not constitute a limitation on the document retrieval device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0059] As Figure 1As shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a document retrieval program.

[0060] In Figure 1 In the document retrieval device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the document retrieval device of the present application can be arranged in the document retrieval device, and the document retrieval device calls the document retrieval program stored in the memory 1005 through the processor 1001 and executes the document retrieval method provided by the embodiments of the present application.

[0061] The embodiments of the present application provide a document retrieval method, which is described with reference to Figure 2 , Figure 2 The flowchart of the first embodiment of the document retrieval method of the present application.

[0062] In this embodiment, the document retrieval method comprises the following steps:

[0063] Step S10: obtaining a target word string of a search term input by a user, and determining the semantic importance of the target word string.

[0064] It should be noted that the execution subject of the method of the present embodiment can be a computing service device with document retrieval, network communication and program running functions, such as a mobile phone, a tablet computer, a personal computer, etc., and can also be other electronic devices that realize the same or similar functions. The above document retrieval device is used to describe the present embodiment and the following embodiments.

[0065] It can be understood that the above search term can be a search term input by a user for retrieving a document. For example, if the search term input by the user is "5G technology specific use", the user expects to obtain documents related to the use of 5G technology.

[0066] It should be noted that the above target word string can be a word string composed of segmented words of the search term. For example, if the search term input by the user is "5G development strategy path", the target word string can be composed of "5G", "development", "strategy" and "development strategy".

[0067] It can be understood that the above semantic importance is the weight of each word string in the target word string corresponding to the search term. If the weight of a segmented word is higher, the semantic importance of the segmented word is higher, and vice versa. In the document retrieval process, the retrieved document should match the segmented word with high semantic importance to better meet the user's expectations.

[0068] In a specific implementation, the document retrieval device can be applied to an online ranking model, or the document retrieval device can be an entity terminal of the online ranking model. The document retrieval device can receive a search term input by a user, perform word segmentation on the search term, obtain a target word string, and analyze the weight of each segmented word in the target word string. The weight of each segmented word can be determined according to the search habits of the user, for example, a segmented word with a higher search frequency can be given a higher weight, a noun with a higher appearance frequency in an industry can be given a higher weight, or a professional noun can be given a higher weight. The embodiment is not limited in this regard. The semantic importance of the target word string can be determined after the weight of each segmented word is determined.

[0069] It should be understood that the online ranking model can be trained by a large number of sample documents and core keywords of each sample document, such as core words in title or paragraph information. The online ranking document needs to be connected to the network to be used by the user. The user inputs the search term into the retrieval device, which is considered as inputting the search term into the online ranking model. The online ranking model can finally output the ranked candidate documents to the user.

[0070] It should be noted that, in the process of performing word segmentation on the technical term, it can also be detected whether the technical term input by the user has a wrong character or a misinput, such as inputting “server” as “accessor”. The search term with the above situation is corrected, and the corrected search term is re-segmented to improve the accuracy of document retrieval and improve the user experience.

[0071] Step S20: obtaining a candidate document matched with the target word string, and determining the relevance between the target word string and the candidate document.

[0072] It should be noted that the candidate document can be a document stored in a preset database. The preset database can be a database constructed in advance and storing a large number of sample documents.

[0073] It should be understood that the relevance can be a measurement value representing the degree of correlation between the candidate document and the target word string. That is, if the relevance is higher, the degree of correlation between the target word string and the candidate document is higher, and the frequency of the target word string appearing in the candidate document is higher. On the contrary, if the relevance is lower, the degree of correlation between the target word string and the candidate document is lower, and the frequency of the target word string appearing in the candidate document is lower.

[0074] In a specific implementation, the document retrieval device can match each word in the target word string with each sample document in the preset database as a matching word, and if a document containing the matching word or related to the matching word exists in the sample document, the document can be recalled as a candidate document. All the recalled candidate documents are obtained, and the relevance between each word and the candidate document corresponding to the word is calculated.

[0075] Step S30: The relevance is optimized according to the semantic importance, and an optimized relevance is obtained.

[0076] In a specific implementation, if the candidate documents sorted by the relevance are displayed to the user, the relevance of some words with low semantic importance may be high, and the words may be ranked in the front, so that the retrieval result does not meet the user's expectation. The document retrieval device can combine the semantic importance of each word with the relevance of each candidate document, multiply the semantic importance and the relevance, map the semantic importance and the relevance to the same latitude, optimize the relevance, superimpose the optimized relevance of each word, and obtain the semantic relevance of the target word string, so that the optimized relevance can also reflect the semantic importance of the target word string, and the relevance of the words with low semantic importance is high and ranked in the front is avoided.

[0077] Step S40: The candidate documents are sorted and displayed according to the optimized relevance.

[0078] In a specific implementation, the document retrieval device can sort the candidate documents from high to low according to the optimized relevance, the candidate documents in the front have higher relevance with the target word string, and the candidate documents in the back have lower relevance with the target word string. In order to avoid excessive number of displayed documents, the number of displayed documents can be limited, and the first preset number (such as the first 100) of candidate documents are displayed. Since the relevance of the displayed candidate documents with the target word string is optimized by the semantic importance, each candidate document can also be related to the semantic importance of the target word string, the semantic importance of the candidate document with high relevance is also high, so that the retrieval result meets the user's expectation.

[0079] The embodiment obtains a target word string of a search keyword input by a user, and determines semantic importance of the target word string; then obtains a candidate document matched with the target word string, and determines relevance between the target word string and the candidate document; finally, the relevance is optimized according to the semantic importance to obtain optimized relevance; and the candidate document is displayed after being sorted according to the optimized relevance. The embodiment determines the semantic importance of the target word string, and optimizes the relevance between the target word string and the candidate document according to the semantic importance, avoids the situation that when the candidate document is a document matched with a word with low importance, the candidate document is also displayed if the relevance of the candidate document is high, and makes the search result more meet the user demand, and effectively improves the user experience.

[0080] Reference Figure 3 , Figure 3 The flowchart of the second embodiment of the document retrieval method is shown.

[0081] Based on the first embodiment, in the embodiment, the step of determining the semantic importance of the target word string comprises:

[0082] Step S101: obtaining original word importance of each word in the target word string relative to the search keyword.

[0083] It should be noted that the original word importance can be a weight calculated based on a sentence length of the search keyword in the target word string, and the sentence length of the search keyword is a sum of the number of words in the search keyword.

[0084] In a specific implementation, the document retrieval device can determine the weight of each word in the target word string relative to the search keyword in sequence, and take the determined weight as the original word importance of each word. Since the original word importance is determined based on the number of words in the search keyword, the sum of the original word importance of each word is 1.

[0085] Step S102: determining a segmented sentence length of the search keyword according to the number of words.

[0086] It should be noted that since the sum of the original word importance of each word is 1, there can be a situation that the original word importance of two words in two sentences with different lengths is the same, but the importance of the two words relative to the search keyword is different, which leads to that the original word importance of each word in search keywords with different lengths cannot be compared, and therefore, it is necessary to determine the relative importance of each word relative to the segmented sentence length of the search keyword, so that the word importance of each word in search keywords with different lengths can be compared.

[0087] In a specific implementation, the document retrieval device determines the length of the segmented sentence of the search term after segmenting the search term, and the length of the segmented sentence can be the sum of the number of segments.

[0088] Step S103: optimizing the original word importance based on the length of the segmented sentence to obtain the semantic importance of the target word string.

[0089] In a specific implementation, the document retrieval device can add the length of the segmented sentence to the original word importance in combination with the length of the segmented sentence relative to the original word importance, optimize the original word importance, and make the optimized word importance the word importance relative to the length of the segmented sentence, so that the word importance of the segments corresponding to search terms of different lengths can also be compared, and the optimization and the word importance more accurately reflect the importance of each segment in the search term.

[0090] Further, in the embodiment, the step S103 includes:

[0091] Step S1031: uniformly processing the original word importance based on the length of the segmented sentence by using a preset word importance optimization formula to obtain the semantic importance of the target word string, wherein the preset word importance optimization formula is:

[0092] W = Important (Sent) * len (Sent),

[0093] In the formula, W is the uniformized word importance, Important (C) is the original word importance, and len (Sent) is the length of the segmented sentence.

[0094] In a specific implementation, the document retrieval device can multiply the original word importance of each segment by the length of the segmented sentence by using the preset word importance optimization formula to obtain the uniformized word importance, which is the semantic importance of the target word string, i.e., the semantic importance is the word importance of each segment relative to the length of the segmented sentence.

[0095] Further, in the embodiment, the step S30 includes:

[0096] Step S301: optimizing the relevance based on the semantic importance by using a preset relevance optimization formula to obtain the optimized segment relevance of each segment, wherein the preset relevance optimization formula is:

[0097] S tw = W i *S i ,

[0098] In the formula, S twTo optimize the post-word segmentation relevance, W i S is the unified word importance of word i in the target word string i is the relevance of word i in the target word string and the candidate document.

[0099] In a specific implementation, the above document retrieval device can multiply the semantic importance by the relevance between each word segmentation and the candidate document through the above preset relevance optimization formula, update the semantic importance to the relevance, and make the optimized post-word segmentation relevance reflect the semantic importance of each word segmentation.

[0100] Step S302: superimpose the optimized post-word segmentation relevance of each word segmentation through a preset relevance superposition formula to obtain the optimized post-relevance of the target word string, wherein the preset relevance superposition formula is:

[0101]

[0102] In the formula, R doc S is the optimized post-relevance tw is the optimized post-word segmentation relevance.

[0103] In a specific implementation, the above document retrieval device can superimpose the optimized post-word segmentation relevance of each word segmentation through a preset relevance superposition formula to obtain the overall relevance of the target word string, i.e., the optimized post-relevance of the target word string. For example, the search term is "5G development strategy path", the target word string is "5G", "development strategy", and "path", and the semantic importance of each word segmentation is "0.3334", "0.4066", and "0.2600", respectively. After determining the semantic importance of each word segmentation, the semantic importance is multiplied by the relevance, and the multiplication results of each word segmentation are superimposed to obtain the relevance between the target word string and the candidate document.

[0104] The embodiment obtains the original word importance of each word segmentation in the target word string relative to the search term, determines the post-word segmentation sentence length of the search term according to the number of word segmentations, optimizes the original word importance based on the post-word segmentation sentence length to obtain the semantic importance of the target word string, makes the optimized post-relevance obtained by optimizing the relevance by the semantic importance better reflect the word importance of each word segmentation, reduces the probability that a document with a non-keyword obtaining a higher relevance is recalled, and effectively improves the precision of candidate document acquisition.

[0105] Reference Figure 4 , Figure 4 is a flowchart of the third embodiment of the document retrieval method of the present application.

[0106] Based on the above embodiments, in the present embodiment, the step of obtaining the candidate document matching the target word string comprises:

[0107] Step S201: querying a matching word string matching each word in the target word string from a preset inverted database.

[0108] It should be noted that the preset inverted database can be a database storing inverted indexes, which can be obtained by cutting the titles of sample documents.

[0109] In a specific implementation, the document retrieval device can query and match from the preset inverted database based on each word in the target word string to query a matching word string matching each word in the target word string.

[0110] Step S202: concurrently querying a matching document matching the matching word string from a preset forward database based on the matching word string.

[0111] It should be noted that the preset forward database is a database storing sample documents, and the inverted indexes in the preset inverted database can be obtained by cutting the titles of the sample documents.

[0112] In a specific implementation, the document retrieval device can perform inverted indexing according to the matching word string and concurrently query the preset forward database to query a matching document matching the matching word string from the preset forward database.

[0113] Step S203: recalling the matching document and taking the matching document as a candidate document matching the target word string.

[0114] In a specific implementation, the document retrieval device can recall the queried matching document, and the recalled matching document can be a candidate document matching the target word string. In the recall process, the preset forward database can be concurrently queried to improve the query efficiency.

[0115] The embodiment queries a matching word string matching each word in the target word string from a preset inverted database, concurrently queries a matching document matching the matching word string from a preset forward database based on the matching word string, recalls the matching document, and takes the matching document as a candidate document matching the target word string. The embodiment combines inverted indexing and forward indexing to perform document recall, and further improves the document search efficiency.

[0116] Further, before the step of obtaining a candidate document matching the target word string, the embodiment further includes:

[0117] Step S21: obtaining a sample document, selecting a corresponding analysis strategy according to the format of the sample document to analyze the sample document, and obtaining a parsed document with a unified format.

[0118] It should be noted that, due to the difference in the format of the document, the style and position of the title and sub-title, for example, in the pdf and word document, the sub-title can be located at the beginning of each chapter or paragraph, and the sub-title of the ppt can be located at the title position of the chapter starting page, therefore, for the document with the title format, the corresponding parsing strategy needs to be selected to parse the sample document into a parsed document with uniform format, so as to facilitate the construction of the preset inverted database and the preset forward database.

[0119] In a specific implementation, the document retrieval device can first convert the sample document into an html format file. For a pdf document, the pdf is cut into single pages, if the cutting is incorrect, the mis-cut paragraphs and tables need to be spliced completely, and the character style and paragraph format are recognized, and a third-party tool of the type of pdfconverter can be selected to convert into an html format; for an office document, since the underlying is similar to the markdown format, it contains multiple types of external labels, and contains accurate format information, therefore, a third-party tool of the type of liberofficelei can be selected to convert into an html format file. After converting into an html format file, the html format file is converted into a uniform format json structured data, which is the parsed document mentioned above, and the storage field of the parsed document contains title, sub-title, paragraph, table and other information.

[0120] Step S22: performing word segmentation on the title of the parsed document using a multi-granularity word segmentation granularity to obtain an inverted index.

[0121] In a specific implementation, the document retrieval recognition can perform word segmentation on the title of the parsed document using a combination of coarse-grained and fine-grained word segmentation granularity, so that the obtained inverted index has more rich inflectional pointing to the document. For example, the title of the indexed file is "5G development strategy path", the fine-grained segmentation can be "5G", "development", "strategy" and "path", and the coarse-grained segmentation can be "5G development" and "development strategy". Compared with the prior art, after word segmentation, the frequency of each word in the document, i.e. the word frequency, needs to be counted, which reflects the importance of the word in the title, and the semantic importance of the word in the title can reflect the importance of each word in the title in the subsequent optimization of the relevance, therefore, the word frequency of each word can not be counted in this embodiment, so as to reduce the performance loss.

[0122] Step S23: constructing a preset inverted database based on the inverted index, and constructing a preset forward database according to a target sample document corresponding to the inverted index.

[0123] In a specific implementation, the document retrieval device can store the obtained inverted index into the same database to construct a preset inverted database. When the preset inverted database is constructed, for the title, sub-title and content of each document, an inverted index field can be established respectively, and different weights can be given to different inverted index fields, which can be the original word importance mentioned above. When the inverted index field is used to search the title or sub-title of the semantic meaning of the statement, the matching word can obtain a higher weight, and the retrieval result can be further optimized.

[0124] It should be noted that the inverted index mentioned above can be stored in the same database for the sample document to construct a preset forward database. When each sample document is stored in the preset forward database, the document retrieval device can traverse each paragraph in the blocks field of each sample document (the blocks field stores the word segmentation information of the full text, paragraph classification and other results), obtain the paragraph information, and if the paragraph is a title or sub-title, store the word segmentation information and the word importance of each word in the word segmentation information, and if the paragraph is a text, store the word segmentation information, which can be the information formed by each word obtained in the word segmentation process. After processing all the paragraphs, the obtained storage result is converted into a binary string using an encryption algorithm to further compress the data and achieve data encryption effect, further protecting data privacy. Since the inverted index recalls the document process queries the preset forward database concurrently, the preset forward database needs to face the problem of high concurrency request to obtain a small data volume, so the sample document can be stored in the mongo database to construct the preset forward database to avoid the above problem and further improve the overall efficiency of the device.

[0125] It can be understood that the preset inverted database and the preset forward database mentioned above can be constructed when the document retrieval device is in an offline state, and when the document retrieval device is in an online state, the document retrieval function can be executed in response to the retrieval word input by the user.

[0126] It should be noted that in the recall of the preset positive rank database, first, long connection can be used instead of end connection on the underlying interactive protocol, that is, the three-way handshake and four-way handshake process of the http protocol is not required each time the access is requested, so as to reduce the time consumption in the recall stage. Secondly, in the cache mechanism, an inverted index preloading mechanism is adopted, which is mainly applied to the termvector and filter of the inverted index. The termvector stores the document index corresponding to each segmented word, and the filter is other filtering information that may be attached during query, such as whether the document creation time is in the given interval or whether the document has a subheading. After preloading the termvector and filter related information into the memory, the originally io-intensive and inefficient disk interaction is converted into a very efficient memory retrieval process in the subsequent query process, greatly improving the efficiency in the case of infrequent name queries. Therefore, by using long connection instead of short connection and the memory preloading mechanism in the underlying protocol in the conversion stage, the recall efficiency is effectively improved.

[0127] For the sake of understanding, reference Figure 5 will be made to the accompanying drawings, but the present solution is not limited thereto. Figure 5 The third embodiment of the document retrieval method of the present application is illustrated in the whole flowchart of the document retrieval, Figure 5 In the offline flow, sample documents are obtained, and the types of the sample documents are determined. The first type of sample document can be an office document, and the second type of sample document can be a pdf document. According to the type of the sample document, a corresponding analysis strategy is used to analyze the sample document, and a preset inverted database and a preset positive rank database are constructed according to the analyzed document. In the online flow, the user input retrieval term is obtained, and the retrieval term is analyzed, that is, the target word string and the semantic importance of each target word string are obtained. Then, based on the target word string, the document recall is performed. The target word string is indexed based on the preset inverted database, so as to recall the matched subsequent document from the preset positive rank database. Then, the document is sorted according to the relevance optimized according to the semantic importance, and finally, the sorted candidate document is displayed to the user.

[0128] Further, in the present embodiment, the construction of the online sorting model comprises:

[0129] Step S01: obtaining initial data composed of preset documents, and labeling the initial data according to preset industry keywords in the preset documents to obtain labeled data.

[0130] It should be noted that the above-mentioned preset document can be a high-precision scene related document, and part of the document carries a preset industry keyword. The preset industry keyword can be a keyword labeled for the purpose of improving the efficiency of the model.

[0131] In a specific implementation, the above document retrieval device can acquire initial data composed of the above preset documents, and determine whether the preset industry keyword can be extracted from the preset documents. If the preset industry keyword can be extracted, the preset industry keyword is extracted, and the document is marked as 1. On the contrary, if the preset industry keyword cannot be extracted, the document is marked as 0. When the marking is completed, all the marked documents are extracted to obtain the labeled data.

[0132] Step S02: training the transformer model by using the labeled data to obtain an offline ranking model, wherein the output result of the offline ranking model is a single-layer result obtained by combining the output results of the last two layers of the transformer model.

[0133] It should be noted that the above transformer model can be a deep learning model using self-attention and can assign different weights according to the importance of the input data.

[0134] In a specific implementation, the transformer model can be trained by dividing the above labeled data into a training set and a validation set, and the accuracy of the transformer model can be adjusted according to the validation set to obtain the above offline ranking model.

[0135] It should be understood that in the above transformer model, the transformer structures of different layers focus on different semantic contents. The transformer structure closer to the input layer tends to extract shallow semantic features such as word, syntax and dependency relationship, and the transformer structure farther from the input layer tends to express deep semantic meanings in the input data. Since the document retrieval has high demand for deep semantic understanding, the transformer model of the present embodiment focuses on deep semantic understanding.

[0136] For ease of understanding, reference Figure 6 is made to Figure 6 illustrate the present solution, but does not limit the present solution. Figure 6In the specific implementation, the transformer first layer is the first layer transformer structure, the transformer second layer is the second layer transformer structure, and so on. The transformer N-1 layer is the N-1 layer transformer structure, and the transformer N layer is the N layer transformer structure. The input data first enters the transformer first layer, is processed, and is sequentially input to the next layer until the last layer, that is, the transformer N layer. After being processed by the transformer N layer, the semantic representation can be output. In order to optimize the offline ranking model and improve the accuracy of the offline ranking model, the output results of the last two layers, that is, the transformer N-1 layer and the transformer N layer, can be combined. Specifically, the output results of the transformer N-1 layer and the transformer N layer can be spliced, pooled, and mapped to a single layer transformer output latitude for output.

[0137] Step S03: predicting the preset unlabeled document data based on the offline ranking model to obtain a sample relevance result of the preset unlabeled document data.

[0138] It should be noted that the preset unlabeled document data can be data composed of randomly selected unlabeled documents.

[0139] In the specific implementation, the preset unlabeled document data can be input to the offline ranking model, predicted by the offline ranking model, and output a sample relevance result of the preset unlabeled document data. The sample relevance result can be used as input of a subsequent model.

[0140] Step S04: training an XGBoost model according to the sample relevance result to obtain an online ranking model.

[0141] It should be noted that the XGBoost (eXtreme Gradient Boosting) model can be an integrated machine learning model based on a decision tree.

[0142] In a specific implementation, since the offline ranking model mainly optimizes the model structure for the semantic expression level, the offline ranking model has better effect in the scene of semantic deep representation and higher similarity. For the online ranking model, due to the high requirement of the online process on performance and the difficulty of meeting the condition of the GPU acceleration card, a heavy model with similar magnitude of parameters as the offline ranking model cannot be used, and therefore the XGBoost model can be used. Since the offline ranking model has better effect in the semantic deep representation, the sample relevance result output by the offline ranking model can be input into the XGBoost model for efficiency, or the sample relevance result and other sample data can be input into the XGBoost model for training to obtain the online ranking model, thereby improving the accuracy of the online ranking model.

[0143] For the sake of understanding, reference Figure 7 will be made to the accompanying drawings, but the present solution is not limited thereto. Figure 7 For the sake of understanding, reference Figure 7 will be made to the accompanying drawings, but the present solution is not limited thereto. In the training process of the offline ranking model, a small amount of labeled data is used to train the transformer model to obtain the offline ranking model, then in the prediction process of the offline ranking model, a large amount of unlabeled document data is predicted by the offline ranking model, and finally in the training process of the online ranking model, the XGBoost model is trained by the prediction result to obtain the online ranking model. In the online mode, the user can input the retrieval term to the online ranking model, the online ranking model performs the document retrieval operation, outputs the ranked candidate documents, and displays them to the user.

[0144] In addition, the embodiment of the present application also provides a storage medium, wherein the storage medium stores a document retrieval program, and the document retrieval program is executed by a processor to realize the steps of the document retrieval method as described above.

[0145] For the sake of understanding, reference Figure 8 , Figure 8 is made to the accompanying drawings, but the present solution is not limited thereto.

[0146] As Figure 8 shown, the document retrieval device provided by the embodiment of the present application comprises:

[0147] The semantic importance module 501 is configured to acquire the target word string of the retrieval term input by the user, and determine the semantic importance of the target word string.

[0148] The document relevance module 502 is configured to acquire the candidate document matched with the target word string, and determine the relevance between the target word string and the candidate document.

[0149] The relevance optimization module 503 is configured to optimize the relevance according to the semantic importance, and obtain an optimized relevance.

[0150] The candidate document display module 504 is configured to display the candidate documents after sorting according to the optimized relevance.

[0151] In the embodiment, the target word string of the search term input by the user is obtained, and the semantic importance of the target word string is determined. Then, the candidate documents matched with the target word string are obtained, and the relevance between the target word string and the candidate documents is determined. Finally, the relevance is optimized according to the semantic importance, and the optimized relevance is obtained. The candidate documents are displayed after sorting according to the optimized relevance. In the embodiment, the semantic importance of the target word string is determined, and the relevance between the target word string and the candidate documents is optimized according to the semantic importance. When the candidate documents are the documents matched with the words with low importance, if the relevance of the candidate documents is high, the candidate documents will also be displayed, which makes the search result more in line with the user demand, and effectively improves the user experience.

[0152] Based on the first embodiment of the document retrieval device, the second embodiment of the document retrieval device is provided.

[0153] In the embodiment, the semantic importance module 501 is further configured to obtain the original word importance of each word in the target word string relative to the search term, determine the length of the sentence after word segmentation of the search term according to the number of the words, and optimize the original word importance based on the length of the sentence after word segmentation, to obtain the semantic importance of the target word string.

[0154] As an implementation form, the semantic importance module 501 is further configured to perform uniformization processing on the original word importance based on the length of the sentence after word segmentation by using a preset word importance optimization formula, to obtain the semantic importance of the target word string, wherein the preset word importance optimization formula is:

[0155] W = Important (Sent) * len (Sent),

[0156] In the formula, W is the uniformized word importance, Important (C) is the original word importance, and len (Sent) is the length of the sentence after word segmentation.

[0157] As an implementation form, the relevance optimization module 503 is further configured to optimize the relevance based on the semantic importance by using a preset relevance optimization formula, to obtain the optimized relevance of each word, wherein the preset relevance optimization formula is:

[0158] S tw = Wi S i ,

[0159] In the formula, S tw is the unified word importance of word i in the target word string, S i is the relevance of word i in the target word string to the candidate document; i

[0160] The optimized word relevance of each word is superimposed by a preset relevance superposition formula to obtain the optimized relevance of the target word string, wherein the preset relevance superposition formula is:

[0161]

[0162] In the formula, R doc is the optimized relevance, and S tw is the optimized word relevance.

[0163] Based on the above-mentioned embodiments of the document updating device, a third embodiment of the document updating device is provided.

[0164] In this embodiment, the document relevance module 502 is further configured to query a matching word string matching each word in the target word string from a preset inverted database; concurrently query a matching document matching the matching word string from a preset forward database based on the matching word string; recall the matching document, and take the matching document as a candidate document matching the target word string.

[0165] As an implementation manner, the document relevance module 502 is further configured to obtain a sample document, parse the sample document according to a corresponding parsing strategy selected according to the format of the sample document, and obtain a parsed document with a unified format; perform word segmentation on the title of the parsed document by using a multi-granularity word segmentation granularity, and obtain an inverted index; construct a preset inverted database based on the inverted index, and construct a preset forward database according to a target sample document corresponding to the inverted index.

[0166] ​As an implementation form, the construction of the online ranking model comprises: obtaining initial data constituted by preset documents, and labeling the initial data according to preset industry keywords in the preset documents to obtain labeled data; training a transformer model through the labeled data to obtain an offline ranking model, wherein an output result of the offline ranking model is a single-layer result obtained by merging output results of last two layers of the transformer model; predicting preset unlabeled document data based on the offline ranking model to obtain a sample relevance result of the preset unlabeled document data; and training an XGBoost model according to the sample relevance result to obtain an online ranking model.

[0167] The specific implementation of the document retrieval device of the present application can refer to the above-mentioned method embodiments, which will not be described here.

[0168] It should be noted that in this document, the terms "comprising", "containing" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0169] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, an optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0171] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A document retrieval method characterized by, The document retrieval method is applied to an online ranking model, and the method comprises the following steps: obtaining a target word string of a search term input by a user, and determining a semantic importance of the target word string, the semantic importance being determined based on an original word importance of each word in the target word string relative to the search term and a sentence length after word segmentation; obtaining candidate documents matching the target word string, and determining a relevance between the target word string and the candidate documents; optimizing the relevance according to the semantic importance to obtain an optimized relevance; displaying the candidate documents after ranking according to the optimized relevance; wherein the step of optimizing the relevance according to the semantic importance to obtain an optimized relevance comprises: mapping the semantic importance of each word in the target word string and the relevance of each candidate document to the same latitude to obtain an optimized word relevance of each word; and superimposing the optimized relevance of each word to obtain an optimized relevance of the target word string; wherein the construction of the online ranking model comprises: obtaining initial data composed of preset documents, if a preset industry keyword is extracted from the preset documents, the corresponding preset document is marked as 1, if no preset industry keyword is extracted from the preset documents, the corresponding preset document is marked as 0, when the marking is completed, all marked documents are extracted to obtain labeled data; The transformer model is trained by using the labeled data, and an offline ranking model is obtained, wherein the transformer model comprises a first layer structure to an Nth layer structure, the Nth layer structure of the transformer model outputs an Nth layer output result, and the first layer structure to the Nth layer structure of the transformer model are sequentially connected. 1layer structure and the output result of the Nth layer structure of the transformer model are spliced and pooled, and are mapped to the latitude of the output of the single-layer transformer model for output. based on the offline ranking model, predicting the preset unlabeled document data to obtain a sample relevance result of the preset unlabeled document data; training the XGBoost model according to the sample relevance result to obtain an online ranking model.

2. The document retrieval method of claim 1, wherein, The step of determining the semantic importance of the target word string comprises: obtaining an original word importance of each word in the target word string relative to the search term; determining a sentence length after word segmentation of the search term according to the number of words; optimizing the original word importance based on the sentence length after word segmentation to obtain the semantic importance of the target word string.

3. The document retrieval method of claim 2, wherein, The step of optimizing the original word importance based on the sentence length after word segmentation to obtain the semantic importance of the target word string comprises: unifying the original word importance based on the sentence length after word segmentation by a preset word importance optimization formula to obtain the semantic importance of the target word string, wherein the preset word importance optimization formula is: ， wherein W is the unified word importance, Important(C) is the original word importance, and len(Sent) is the sentence length after word segmentation.

4. The document retrieval method of claim 1, wherein, The step of obtaining candidate documents matching the target word string comprises: querying matching word strings matching each word in the target word string from a preset inverted database; concurrently querying matching documents matching the matching word strings from a preset forward database based on the matching word strings; recalling the matching documents, and taking the matching documents as candidate documents matching the target word string.

5. The document retrieval method of claim 4, wherein, Before the step of obtaining candidate documents matching the target word string, the method further comprises: The sample document is obtained, and a corresponding parsing strategy is selected according to the format of the sample document to parse the sample document, and a parsed document with uniform format is obtained; The title of the parsed document is cut with a multi-granularity cutting granularity to obtain an inverted index; A preset inverted database is constructed based on the inverted index, and a preset forward database is constructed according to a target sample document corresponding to the inverted index.

6. A document retrieval apparatus characterized by comprising: The device comprises: The semantic importance module is configured to obtain a target word string of a search term input by a user and determine a semantic importance of the target word string, wherein the semantic importance is determined based on the importance of each word in the target word string relative to the original words of the search term and the length of the sentence after word segmentation. The document relevance module is configured to obtain candidate documents matching the target word string and determine the relevance between the target word string and the candidate documents. The relevance optimization module is configured to optimize the relevance based on the semantic importance to obtain an optimized relevance. The candidate document display module is configured to sort and display the candidate documents based on the optimized relevance. The relevance optimization module is further configured to map the semantic importance of each word in the target word string and the relevance of each candidate document to the same latitude for optimization to obtain the optimized relevance of each word, and superimpose the optimized relevance of each word to obtain the optimized relevance of the target word string. The construction of the online sorting model comprises: An initial data composed of preset documents is obtained, and if a preset industry keyword is extracted from the preset documents, the corresponding preset document is marked as 1, and if no preset industry keyword is extracted from the preset documents, the corresponding preset document is marked as 0. When the marking is completed, all marked documents are extracted to obtain labeled data. The transformer model is trained by using the labeled data, and an offline ranking model is obtained, wherein the transformer model comprises a first layer structure to an Nth layer structure, the Nth layer structure of the transformer model outputs an Nth layer output result, and the first layer structure to the Nth layer structure of the transformer model are sequentially connected. 1layer structure and the output result of the Nth layer structure of the transformer model are spliced and pooled, and are mapped to the latitude of the output of the single-layer transformer model for output. The preset unlabeled document data is predicted based on the offline sorting model to obtain a sample relevance result of the preset unlabeled document data; and the XGBoost model is trained based on the sample relevance result to obtain an online sorting model.

7. A document retrieval apparatus characterized by comprising: The device comprises a memory, a processor, and a document retrieval program stored on the memory and executable on the processor, wherein the document retrieval program is configured to implement the steps of the document retrieval method according to any one of claims 1 to 5.

8. A storage medium, characterized by The storage medium stores a document retrieval program, and the document retrieval program is executed by the processor to implement the steps of the document retrieval method according to any one of claims 1 to 5.

Images