Information retrieval method and system based on natural language

Abstract

The invention provides an information retrieval method and system based on a natural language, and the method comprises the steps: obtaining retrieval information inputted by a user, and generating standardized data of the retrieval information according to the semantics of the retrieval information; aiming at the standardized data, generating a plurality of query requests divided according to a semantic range; and obtaining retrieval results corresponding to the query requests, sorting the retrieval results according to the semantic range, and displaying the sorted retrieval results to the user. According to the technical scheme provided by the invention, the retrieval information meeting user requirements can be provided.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to an information retrieval method and system based on natural language. Background Art

[0002] With the continuous development of information technology, the amount of information in the network has also increased explosively. How to query the information needed by oneself from the vast amount of information is an urgent problem to be solved at present.

[0003] Some existing mature information retrieval frameworks usually query corresponding retrieval results based on the retrieval requests input by users. However, in actual applications, users often cannot put forward accurate retrieval requests, resulting in the information retrieval framework being unable to provide the retrieval results actually needed by users. Summary of the Invention

[0004] The present invention provides an information retrieval method and system based on natural language, which can provide retrieval information that meets the needs of users.

[0005] The present invention provides an information retrieval method based on natural language, and the method includes:

[0006] Obtain the retrieval information input by the user, and generate standardized data of the retrieval information according to the semantics of the retrieval information;

[0007] Generate a plurality of query requests divided according to the semantic range for the standardized data;

[0008] Obtain the retrieval results corresponding to each of the query requests, sort the retrieval results according to the semantic range, and display the sorted retrieval results to the user.

[0009] In one embodiment, generating the standardized data of the retrieval information according to the semantics of the retrieval information includes:

[0010] Extract the semantic features of the retrieval information, and calculate the feature similarity between the semantic features and each standard feature in a preset standard information library; A

[0011] Determine the standard feature with the highest feature similarity, and use the standardized data corresponding to the standard feature with the highest feature similarity as the standardized data of the retrieval information.

[0012] In one embodiment, generating a plurality of query requests divided according to the semantic range includes:

[0013] Determine a plurality of different semantic ranges including the semantics of the retrieval information;

[0014] For any semantic range, generate the standardized data corresponding to the semantic range, and use the generated standardized data as the query request corresponding to the semantic range;

[0015] Summarize the query requests for each semantic range to obtain multiple query requests divided according to semantic range.

[0016] In one embodiment, after presenting the sorted retrieval results to the user, the method further includes:

[0017] If the retrieval result is triggered, present the detailed page of the retrieval result, and at a specified position on the detailed page, display the query request applicable to the retrieval result;

[0018] If the query request is selected, present an extended page frame on the detailed page, and in the extended page frame, display the query requests corresponding to other semantic ranges.

[0019] In one embodiment, sorting the retrieval results includes:

[0020] Step 1: Let t be a word in one of the multiple query requests divided according to semantic range. Then the word frequency of t in document W is:

[0021] P Wt = 1 + log(F Wt )

[0022] where P Wt is the word frequency of word t in document W, and F Wt is the statistical frequency of word t in document W; <F

[0023] Step 2: Let the total number of documents in the entire corpus be N, and the number of documents containing word t be Nt. Then the weight factor Qt of word t in the entire corpus is:

[0024]

[0025] Step 3: According to the results of Step 1 and Step 2, calculate the sorting score of document W when the query request is word t as follows:

[0026]

[0027] where K Wt is the sorting score of document W when the query request is word t, k is an empirical factor, taking 0.4, and F W is the statistical frequency of all words in document W;

[0028] Step 4: Calculate the sorting scores for all words in each of the multiple query requests divided according to the semantic scope against the documents in the entire corpus according to the above steps. Finally, sum up the sorting scores of all words in each of the multiple query requests divided according to the semantic scope for each document, and arrange them in reverse order to obtain the final retrieval result sorting.

[0029] Wherein, before generating the standardized data of the retrieval information according to the semantics of the retrieval information, the following steps are further included:

[0030] Streamline the redundant information in the retrieval information, and the specific steps include:

[0031] Step A1: Use formula (1) to obtain the corresponding semantic value density scores for each word in the retrieved information according to the retrieved information input by the user.

[0032]

[0033] Where V(w i ) represents the semantic value density score of word w i ; f(w i ) represents the word frequency of word w i in the input; t i , represents the time position (word ordinal number) where word w i first appears, simulating the attenuation of attention; represents the cosine similarity of the semantic embedding of word w i with the core theme C; N(w i ) represents the set of direct related words of word w i in the pre-trained semantic network; d(w i ,w j ) represents the path distance associated with word w i and word w j ; α, β represent the adjustment coefficients of the theme relevance and the semantic network depth;

[0034] Step A2: Use formula (2) to obtain the redundancy entropy value of each word according to the semantic value density score corresponding to each word in the retrieved information.

[0035]

[0036] Where R(w i ) represents the redundancy entropy value of word w i , and the larger the value, the higher the redundancy; S represents the set of keywords that have been retained currently; sim(w i ,w k ) represents the relationship between word w i and word w kSemantic similarity (based on BERT embeddings); λ represents the sensitivity coefficient of semantic similarity; min(t i ,t k ) represents the minimum of the occurrence time positions of the word w i and the word w k ; ε represents a small constant to prevent division by zero; || represents taking the absolute value; e represents the natural constant;

[0037] Step A3: Use formula (3) to obtain the corresponding semantic value density score for each word in the retrieved information and the redundancy entropy value of each word to control the retention or deletion status of the word

[0038]

[0039] where J(w i ) represents the retention or deletion status control value of the word w i ; when its value is 1, it means to retain the word w i , and when its value is -1, it means to delete the word w i ; τ0 represents the base threshold; γ represents the gain coefficient of the total value of the retained words to the threshold; η represents the normalization factor of the value gain; represents compressing the influence range of the retained value to [-1, 1] to prevent threshold drift.

[0040] The beneficial effects of the above technical solution are as follows: Using formula (1) in step A1, according to the retrieved information input by the user, the corresponding semantic value density score is obtained for each word in the retrieved information, thereby simultaneously capturing the timeliness (position decay), topic relevance, and semantic network depth of the words, exceeding traditional TF-IDF or word frequency statistics, and realizing multi-dimensional value fusion to ensure the accuracy of the score; then using formula (2) in step A2, according to the semantic value density score corresponding to each word in the retrieved information, the redundancy entropy value of each word is obtained, and non-linear redundancy detection is realized through the formula to distinguish synonyms (high similarity) from vaguely related words; then using formula (3) in step A3, according to the semantic value density score corresponding to each word in the retrieved information and the redundancy entropy value of each word, the retention or deletion status of the word is controlled. For the first time, the retention decision is transformed into a ratio comparison of value and redundancy, replacing the traditional absolute threshold, and the threshold is adjusted reversely by the total value of the retained content, realizing the "more refined, more strict" adaptive control.

[0041] The present invention also provides an information retrieval system based on natural language, and the system includes:

[0042] A normalization processing unit, configured to obtain the retrieved information input by the user and generate normalized data of the retrieved information according to the semantics of the retrieved information;

[0043] A query request generation unit, configured to generate a plurality of query requests divided according to semantic ranges for the standardized data;

[0044] A result display unit, configured to obtain retrieval results corresponding to each of the query requests, sort the retrieval results according to the semantic ranges, and display the sorted retrieval results to the user.

[0045] In one embodiment, the standardization processing unit is specifically configured to extract semantic features of the retrieval information, and calculate feature similarity degrees between the semantic features and each standard feature in a preset standard information library; determine the standard feature with the highest feature similarity degree, and use the standardized data corresponding to the standard feature with the highest feature similarity degree as the standardized data of the retrieval information.

[0046] In one embodiment, the query request generation unit is specifically configured to determine a plurality of different semantic ranges including the semantics of the retrieval information; for any semantic range, generate the standardized data corresponding to the semantic range, and use the generated standardized data as the query request corresponding to the semantic range; summarize the query requests of each semantic range to obtain a plurality of query requests divided according to semantic ranges.

[0047] In one embodiment, the system further includes:

[0048] A detail page display unit, configured to display a detail page of the retrieval result if the retrieval result is triggered, and display the query request applicable to the retrieval result at a specified position on the detail page;

[0049] An extended page display unit, configured to display an extended page frame in the detail page if the query request is selected, and the extended page frame displays query requests corresponding to other semantic ranges.

[0050] In one embodiment, the standardization processing unit is further configured to streamline redundant information in the retrieval information before generating the standardized data of the retrieval information according to the semantics of the retrieval information. The specific steps include:

[0051] Step A1: Use formula (1) to obtain a corresponding semantic value density score for each word in the retrieval information according to the retrieved user input retrieval information

[0052]

[0053] where V(w i ) represents the semantic value density score of word w i ; f(w i ) represents word wi The word frequency of the word in the input; t i , representing the word w i The time position (word ordinal number) of the first occurrence, simulating the attenuation of attention; representing the word w i The cosine similarity of the semantic embedding with the core theme C; N(w i ) represents the word w i The set of direct related words of the word in the pre-trained semantic network; d(w i , w j ) represents the word w i The path distance associated with the word w j ; α, β represent the adjustment coefficients of the topic relevance and the depth of the semantic network;

[0054] Step A2: Use formula (2) to obtain the redundancy entropy value of each word according to the semantic value density score corresponding to each word in the retrieved information

[0055]

[0056] where R(w i ) represents the redundancy entropy value of the word w i , and the larger the value, the higher the redundancy; S represents the set of keywords that have been retained; sim(w i , w k ) represents the semantic similarity between the word w i and the word w k (based on BERT embedding); λ represents the sensitivity coefficient of the semantic similarity; min(t i , t k ) represents the minimum value of the occurrence time positions of the word w i and the word w k ; ε represents a small constant to prevent division by zero; || represents taking the absolute value; e represents the natural constant;

[0057] Step A3: Use formula (3) to obtain the corresponding semantic value density score and the redundancy entropy value of each word in the retrieved information to control the elimination and retention status of the word

[0058]

[0059] where J(w i ) represents the control value of the elimination and retention status of the word w i ; among them, when its value is 1, it means to retain the word w i , and when its value is -1, it means to eliminate the word w i ; τ0 represents the basic threshold; γ represents the gain coefficient of the total value of the retained words to the threshold; η represents the normalization factor of the value gain; It means to compress the influence range of the reserved value to [-1, 1] to prevent threshold drift.

[0060] For the technical solution provided by the present invention, the semantics of the retrieval information can be extracted according to the retrieval information input by the user. If only providing retrieval results based on this retrieval information, it may not meet the actual needs of the user. In view of this, in the present invention, based on the semantics of the retrieval information, query requests in different semantic ranges can be determined. These query requests in different semantic ranges can cover the true needs of the user with a very high probability. Subsequently, the retrieval results provided according to these query requests can also be sorted according to the semantic range, so that corresponding retrieval results can be provided to the user within different semantic ranges, enriching the quantity of the provided retrieval results, and thus being able to meet the actual needs of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. Among them:

[0062] Figure 1 It is a schematic diagram of the steps of an information retrieval method based on natural language provided by the present invention;

[0063] Figure 2 It is a schematic diagram of the functional modules of an information retrieval system based on natural language provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will make a detailed description of the specific embodiments of the present application in conjunction with the drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the sake of description, only the parts related to the present application are shown in the drawings rather than all the structures. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0065] The terms "including" and "having" and any variations thereof in the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0066] As used herein, the mention of "embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0067] Please refer to Figure 1 As shown, an embodiment of the present application provides an information retrieval method based on natural language. The method includes:

[0068] S1: Obtain the retrieval information input by the user, and generate standardized data of the retrieval information according to the semantics of the retrieval information;

[0069] [[ID=1,2]]S2: Generate a plurality of query requests divided according to the semantic range for the standardized data;

[0070] S3: Obtain the retrieval results corresponding to each of the query requests, sort the retrieval results according to the semantic range, and display the sorted retrieval results to the user.

[0071] Since the sorting of the retrieval results is directly related to the user experience of using the system, if the user information retrieval experience is not good and the user cannot easily find the retrieval results they need, it will be very difficult to retain users, and the value of the entire system will be greatly discounted. To improve the user experience, when sorting the user's retrieval results, the following algorithm is used:

[0072] Step 1: Let t be a word in a plurality of query requests divided according to the semantic range. Then the word frequency of t in the document W is:

[0073] P Wt = 1 + log(F Wt )

[0074] where P Wt is the word frequency of the word t in the document W, and F Wt is the statistical frequency of the word t in the document W.

[0075] Step 2: Let the total number of documents in the entire corpus be N, and the number of documents containing the word t be Nt. Then the weight factor Qt of the word t in the entire corpus is:

[0076]

[0077] Step 3: According to the results of Steps 1 and 2, calculate the sorting score of the document W when the query request is the word t as follows:

[0078]

[0079] where K Wt is the ranking score of document W when the query request is word t, k is an empirical factor, generally taken as 0.4, and F W is the statistical frequency of all words in document W.

[0080] Step 4: Calculate the ranking scores of all words in multiple query requests divided according to the semantic range for the documents in the entire corpus according to the above steps. Finally, sum up the ranking scores of all words in multiple query requests divided according to the semantic range for each document, and arrange them in reverse order, which is the final retrieval result ranking.

[0081] This algorithm calculates the ranking scores of all documents in the corpus for multiple query words divided according to the semantic range of the retrieval information input by the user, and performs ranking display according to the final calculation results. It fully considers the relationship between words and between words and documents, displays the results desired by the user in the front row, effectively improves the accuracy of information retrieval, and enhances the user experience.

[0082] In a specific application example, the retrieval information input by the user can be: "I want to know how to raise a cute puppy". Based on the semantics of this retrieval information, the standardized data "how to raise a dog" can be obtained. Obviously, this standardized data is relatively broad. In actual applications, multiple different semantic ranges can be determined based on the semantics of this standardized data. For example, under the semantic condition of "how to raise a dog", multiple query requests such as "how to raise a dog", "how to raise a 2-month-old dog", "how to raise a 2-month-old Husky", "how to bathe a puppy", and "how to feed a puppy" can be obtained. These query requests can correspond to different semantic ranges. For example, a query request like "how to raise a dog" has the largest corresponding semantic range, "how to bathe a puppy" and "how to feed a puppy" can have the same semantic range, and "how to raise a 2-month-old Husky" can have the smallest semantic range. In this way, multiple query requests with different semantic ranges can be generated through the retrieval information input by the user, and these query requests are very likely to cover the user's real needs. For different query requests, respective retrieval results can be obtained, and these retrieval results can be sorted in descending order according to the semantic range. In this way, the user can be gradually guided to dig deeper into their needs, so as to obtain the actual required retrieval results.

[0083] In one embodiment, generating the standardized data of the retrieval information according to the semantics of the retrieval information includes:

[0084] Extracting the semantic features of the retrieval information and calculating the feature similarity between the semantic features and each standard feature in the preset standard information library;

[0085] Determine the standard feature with the highest feature similarity, and use the standardized data corresponding to the standard feature with the highest feature similarity as the standardized data of the retrieval information.

[0086] In one embodiment, generating multiple query requests divided according to semantic ranges includes:

[0087] Determine multiple different semantic ranges including the semantics of the retrieval information;

[0088] For any semantic range, generate the standardized data corresponding to the semantic range, and use the generated standardized data as the query request corresponding to the semantic range;

[0089] Summarize the query requests of each semantic range to obtain multiple query requests divided according to semantic ranges.

[0090] In one embodiment, after presenting the sorted retrieval results to the user, the method further includes:

[0091] If the retrieval result is triggered, display the detail page of the retrieval result, and at a specified position on the detail page, display the query request applicable to the retrieval result;

[0092] If the query request is selected, display an expansion page frame on the detail page, and the query requests corresponding to other semantic ranges are displayed in the expansion page frame.

[0093] In this embodiment, on the detail page of the retrieval result, the query request applicable to the retrieval result can be displayed at a preset specified position, so that the user can know which query request the currently viewed retrieval result is retrieved for. At the same time, if the query request is selected, other query requests with different semantic ranges from this query request can be further displayed, so as to conveniently select query requests in other semantic ranges. Similarly, after the query request in the expansion page frame is selected, the detail page corresponding to the query result corresponding to the query request can be displayed.

[0094] In one embodiment, before generating the standardized data of the retrieval information according to the semantics of the retrieval information, it further includes:

[0095] Streamline the redundant information in the retrieval information, and its specific steps include,

[0096] Step A1: Use formula (1) to obtain the corresponding semantic value density score for each word in the retrieved information according to the retrieved information input by the user

[0097]

[0098] Among them, V(w i ) represents the semantic value density score of the word w i ; f(w i ) represents the word frequency of the word w i in the input; t i , represents the time position (word ordinal number) when the word w i first appears, simulating the attenuation of attention; represents the cosine similarity of the semantic embedding of the word w i with the core theme C; N(w i ) represents the set of direct related words of the word w i in the pre-trained semantic network; d(w i , w j ) represents the path distance associated with the word w i and the word w j ; α, β represent the adjustment coefficients of the theme relevance and the semantic network depth;

[0099] Step A2: Use formula (2) to obtain the redundancy entropy value of each word according to the semantic value density score corresponding to each word in the retrieved information

[0100]

[0101] Among them, R(w i ) represents the redundancy entropy value of the word w i , and the larger the value, the higher the redundancy; S represents the set of keywords that have been retained currently; sim(w i , w k ) represents the semantic similarity between the word w i and the word w k (based on BERT embedding); λ represents the sensitivity coefficient of the semantic similarity; min(t i , t k ) represents the minimum value of the appearance time positions of the word w i and the word w k ; ε represents a small constant to prevent division by zero; || represents taking the absolute value; e represents the natural constant;

[0102] Step A3: Use formula (3) to control the elimination and retention status of each word according to the semantic value density score corresponding to each word in the retrieved information and the redundancy entropy value of each word

[0103]

[0104] Among them, J(w i ) represents the word w iThe control value for deletion and retention status; where when its value is 1, it means to retain the word w i and when its value is -1, it means to delete the word w i ; τ0 represents the base threshold; γ represents the gain coefficient of the total value of the retained words with respect to the threshold; η represents the normalization factor of value gain; represents compressing the influence range of the retained value to [-1, 1] to prevent threshold drift.

[0105] The beneficial effects of the above technical solution are as follows: Using the formula (1) in step A1, according to the retrieved information input by the user, the corresponding semantic value density score is obtained for each word in the retrieved information, thereby simultaneously capturing the timeliness (position decay), topic relevance, and semantic network depth of the words, exceeding traditional TF-IDF or word frequency statistics, achieving multi-dimensional value fusion to ensure the accuracy of the score; then using the formula (2) in step A2 to obtain the redundancy entropy value of each word according to the semantic value density score corresponding to each word in the retrieved information, realizing non-linear redundancy detection through the formula to distinguish synonyms (high similarity) from vaguely related words; then using the formula (3) in step A3 to control the deletion and retention status of each word according to the semantic value density score corresponding to each word in the retrieved information and the redundancy entropy value of each word, for the first time converting the retention decision into a ratio comparison of value and redundancy, replacing the traditional absolute threshold, and reversely adjusting the threshold through the total value of the retained content to achieve "the more refined, the more strict" adaptive control.

[0106] Please refer to Figure 2 , the present invention also provides an information retrieval system based on natural language, and the system includes:

[0107] A normalization processing unit, configured to obtain the retrieved information input by the user and generate normalized data of the retrieved information according to the semantics of the retrieved information;

[0108] A query request generation unit, configured to generate a plurality of query requests divided according to the semantic range for the normalized data;

[0109] A result display unit, configured to obtain the retrieval results corresponding to each of the query requests, sort the retrieval results according to the semantic range, and display the sorted retrieval results to the user.

[0110] In one embodiment, the normalization processing unit is specifically configured to extract the semantic features of the retrieved information, calculate the feature similarity between the semantic features and each standard feature in a preset standard information library; determine the standard feature with the highest feature similarity, and use the normalized data corresponding to the standard feature with the highest feature similarity as the normalized data of the retrieved information.

[0111] In one embodiment, the query request generation unit is specifically configured to determine a plurality of different semantic ranges including the semantics of the retrieval information; for any semantic range, generate standardized data corresponding to the semantic range, and use the generated standardized data as the query request corresponding to the semantic range; summarize the query requests of each semantic range to obtain a plurality of query requests divided according to the semantic range.

[0112] In one embodiment, the system further includes:

[0113] A detail page display unit, configured to display a detail page of the retrieval result if the retrieval result is triggered, and display the query request applicable to the retrieval result at a specified position on the detail page;

[0114] An extended page display unit, configured to display an extended page frame in the detail page if the query request is selected, and display query requests corresponding to other semantic ranges in the extended page frame.

[0115] In one embodiment, the standardization processing unit is further configured to streamline redundant information in the retrieval information before generating the standardized data of the retrieval information according to the semantics of the retrieval information. The specific steps include:

[0116] Step A1: Use formula (1) to obtain the corresponding semantic value density score for each word in the retrieval information according to the retrieved user input information

[0117]

[0118] Where V(w i ) represents the semantic value density score of word w i ; f(w i ) represents the word frequency of word w i in the input; t i , represents the time position (word ordinal number) where word w i first appears, simulating attention attenuation; represents the cosine similarity of the semantic embedding of word w i with the core theme C; N(w i ) represents the set of direct associated words of word w i in the pre-trained semantic network; d(w i , w j ) represents the path distance associated with word w i and word w j ; α, β represent the adjustment coefficients of the topic relevance and the semantic network depth;

[0119] Step A2: Obtain the redundancy entropy value of each word according to the semantic value density score corresponding to each word in the retrieved information by using formula (2).

[0120]

[0121] Where R(w i ) represents the redundancy entropy value of word w i , and the larger the value, the higher the redundancy; S represents the set of keywords that have been retained; sim(w i , w k ) represents the semantic similarity between word w i and word w k (based on BERT embedding); λ represents the sensitivity coefficient of semantic similarity; min(t i , t k ) represents the minimum value of the occurrence time positions of word w i and word w k ; ε represents a small constant to prevent division by zero; || represents taking the absolute value; e represents the natural constant;

[0122] Step A3: Control the elimination and retention status of each word according to the semantic value density score and the redundancy entropy value corresponding to each word in the retrieved information by using formula (3).

[0123]

[0124] Where J(w i ) represents the control value of the elimination and retention status of word w i ; among them, when its value is 1, it means to retain word w i , and when its value is -1, it means to eliminate word w i ; τ0 represents the basic threshold; γ represents the gain coefficient of the total value of the retained words to the threshold; η represents the normalization factor of value gain; represents compressing the influence range of the retained value to [-1, 1] to prevent threshold drift.

[0125] The beneficial effects of the above technical solution are as follows: Using the formula (1) in step A1, based on the retrieved information input by the user, the corresponding semantic value density score is obtained for each word in the retrieved information, thereby simultaneously capturing the timeliness (position attenuation), topic relevance, and semantic network depth of the words, exceeding traditional TF-IDF or word frequency statistics, and achieving multi-dimensional value fusion to ensure the accuracy of the scores; then using the formula (2) in step A2, the redundancy entropy value of each word is obtained according to the semantic value density score corresponding to each word in the retrieved information, and non-linear redundancy detection is achieved through the formula to distinguish synonyms (high similarity) from vaguely related words; then using the formula (3) in step A3, the retention or elimination status of each word is controlled according to the semantic value density score corresponding to each word in the retrieved information and the redundancy entropy value of each word. For the first time, the retention decision is transformed into a ratio comparison of value and redundancy, replacing the traditional absolute threshold, and the threshold is reversely adjusted by the total value of the retained content to achieve "the more refined, the stricter" adaptive control.

[0126] The technical solution provided by the present invention can extract the semantics of the retrieved information according to the retrieved information input by the user. If only the retrieved information is used to provide retrieval results, it may not meet the actual needs of the user. In view of this, in the present invention, based on the semantics of the retrieved information, query requests with different semantic ranges can be determined. These query requests with different semantic ranges can very likely cover the true needs of the user. Subsequently, the retrieval results provided according to these query requests can also be sorted according to the semantic range, so that corresponding retrieval results are provided to the user within different semantic ranges, enriching the quantity of the provided retrieval results, and thus being able to meet the actual needs of the user.

[0127] The above are only some specific embodiments of the present invention. Any improvements made on the premise of the present invention's concept are regarded as the protection scope of the present invention.

Claims

1. An information retrieval method based on natural language, characterized in that, The method includes: Obtaining the retrieval information input by the user, and generating standardized data of the retrieval information according to the semantics of the retrieval information; Generating multiple query requests divided according to the semantic range for the standardized data; Obtaining the retrieval results corresponding to each query request, sorting the retrieval results according to the semantic range, and presenting the sorted retrieval results to the user.

2. The method according to claim 1, wherein Generating the standardized data of the retrieval information according to the semantics of the retrieval information includes: Extracting the semantic features of the retrieval information, and calculating the feature similarity between the semantic features and each standard feature in the preset standard information library; Determining the standard feature with the highest feature similarity, and using the standardized data corresponding to the standard feature with the highest feature similarity as the standardized data of the retrieval information.

3. The method according to claim 1, wherein Generating multiple query requests divided according to the semantic range includes: Determining multiple different semantic ranges including the semantics of the retrieval information; For any semantic range, generating the standardized data corresponding to the semantic range, and using the generated standardized data as the query request corresponding to the semantic range; Summarizing the query requests of each semantic range to obtain multiple query requests divided according to the semantic range.

4. The method according to claim 1, characterized in that After presenting the sorted retrieval results to the user, the method further includes: If the retrieval result is triggered, presenting the detailed page of the retrieval result, and displaying the query request applicable to the retrieval result at a specified position on the detailed page; If the query request is selected, presenting an extended page box on the detailed page, and displaying the query requests corresponding to other semantic ranges in the extended page box.

5. The method according to claim 1, wherein Sorting the retrieval results includes: Step 1: Let t be a word in multiple query requests divided according to the semantic range, then the word frequency of t in document W is: P Wt = 1 + log(F Wt ) where P Wt is the word frequency of word t in document W, and F Wt is the statistical frequency of word t in document W; Step 2: Let the total number of documents in the entire corpus be N, and the number of documents containing word t be Nt, then the weight factor Qt of word t in the entire corpus is: Step 3: According to the results of Step 1 and Step 2, calculate the sorting score of document W when the query request is word t as follows: Among which K Wt is the sorting score of document W when the query request is word t, k is an empirical factor, taking 0.4, F W is the statistical frequency of all words in document W; Step 4: Calculate the sorting scores of the documents in the entire corpus for all words in multiple query requests divided according to the semantic range according to the above steps. Finally, sum up the sorting scores of all words in multiple query requests divided according to the semantic range of each document, and arrange them in reverse order as the final retrieval result sorting.

6. The method according to claim 1, wherein Before generating the standardized data of the retrieval information according to the semantics of the retrieval information, it further includes: Streamlining the redundant information in the retrieval information, and its specific steps include, Step A1: Using formula (1), according to the retrieved information input by the user obtained, obtaining the corresponding semantic value density score for each word in the retrieved information where V(w i ) represents the semantic value density score of the word w i ; f(w i ) represents the word frequency of the word w i in the input; t i , represents the time position (word ordinal number) where the word w i first appears, simulating the attention attenuation; represents the cosine similarity of the semantic embedding of the word w i with the core theme C; N(w i ) represents the set of direct related words of the word w i in the pre-trained semantic network; d(w i , w j ) represents the path distance associated with the word w i and the word w j ; α, β represent the adjustment coefficients of the theme relevance and the semantic network depth; Step A2: Using formula (2), obtaining the redundancy entropy value of each word according to the semantic value density score corresponding to each word in the retrieved information where \(R(w\) i ) represents the redundancy entropy value of the word \(w\) i , the larger the value, the higher the redundancy; \(S\) represents the set of keywords that have been retained; \(sim(w\) i , w\) k ) represents the semantic similarity between the word \(w\) i and the word \(w\) k (based on BERT embedding); \(\lambda\) represents the sensitivity coefficient of semantic similarity; \(min(t\) i , t\) k ) represents the minimum value among the occurrence time positions of the word \(w\) i and the word \(w\) k ; \(\varepsilon\) represents a small constant to prevent division by zero; \(||\) represents taking the absolute value; \(e\) represents the natural constant; Step A3: Use formula (3) to obtain the corresponding semantic value density score and the redundancy entropy value of each word in the retrieved information, and control the deletion and retention status of the word based on these values Where J(w i ) represents the deletion and retention status control value of the word w i ; among them, when its value is 1, it means retaining the word w i , and when its value is -1, it means deleting the word w i ; τ0 represents the base threshold; γ represents the gain coefficient of the total value of the retained words with respect to the threshold; η represents the normalization factor of the value gain; means compressing the influence range of the retained value to [-1, 1].

7. An information retrieval system based on natural language, characterized in that, The system includes: A normalization processing unit, configured to obtain the retrieved information input by the user, and generate normalized data of the retrieved information according to the semantics of the retrieved information; A query request generation unit, configured to generate a plurality of query requests divided according to the semantic range for the normalized data; A result display unit, configured to obtain the retrieval results corresponding to each of the query requests, sort the retrieval results according to the semantic range, and display the sorted retrieval results to the user.

8. The system according to claim 7, wherein Specifically, the normalization processing unit is configured to extract the semantic features of the retrieved information, and calculate the feature similarity between the semantic features and each standard feature in a preset standard information library; Determine the standard feature with the highest feature similarity, and use the normalized data corresponding to the standard feature with the highest feature similarity as the normalized data of the retrieved information; and / or Specifically, the query request generation unit is configured to determine a plurality of different semantic ranges including the semantics of the retrieved information; for any semantic range, generate the normalized data corresponding to the semantic range, and use the generated normalized data as the query request corresponding to the semantic range; summarize the query requests of each semantic range to obtain a plurality of query requests divided according to the semantic range.

9. The system according to claim 7, wherein The system further includes: A detail page display unit, configured to display the detail page of the retrieval result if the retrieval result is triggered, and display the query request applicable to the retrieval result at a specified position on the detail page; An extended page display unit, configured to display an extended page box in the detail page if the query request is selected, and the extended page box displays query requests corresponding to other semantic ranges.

10. The system according to claim 7, wherein The normalization processing unit is further configured to streamline the redundant information in the retrieved information before generating the normalized data of the retrieved information according to the semantics of the retrieved information. The specific steps include Step A1: Use formula (1) to obtain the corresponding semantic value density score of each word in the retrieved information according to the retrieved information input by the user Among them, V(w i ) represents the semantic value density score of the word w i ; f(w i ) represents the word frequency of the word w i in the input; t i , represents the time position (word ordinal number) when the word w i first appears, simulating the attenuation of attention; represents the cosine similarity of the semantic embedding between the word w i and the core theme C; N(w i ) represents the set of direct associated words of the word w i in the pre-trained semantic network; d(w i , w j ) represents the path distance associated with the word w i and the word w j ; α, β represent the adjustment coefficients of the theme relevance and the semantic network depth; Step A2: Use formula (2) to obtain the redundancy entropy value of each word according to the semantic value density score corresponding to each word in the retrieved information where \(R(w\) i ) represents the redundancy entropy value of the word \(w\) i , the larger the value, the higher the redundancy; \(S\) represents the set of keywords that have been retained currently; \(sim(w\) i , \(w\) k ) represents the semantic similarity between the word \(w\) i and the word \(w\) k (based on BERT embedding); \(\lambda\) represents the sensitivity coefficient of semantic similarity; \(min(t\) i , \(t\) k ) represents the minimum value among the occurrence time positions of the word \(w\) i and the word \(w\) k ; \(\varepsilon\) represents a small constant to prevent division by zero; \(||\) represents taking the absolute value; \(e\) represents the natural constant; Step A3: Use formula (3) to obtain the corresponding semantic value density score and the redundancy entropy value of each word in the retrieved information, and control the deletion and retention status of the word based on these values Among them, J(w i ) represents the control value of the retained or removed status of the word w i ; among them, when its value is 1, it means retaining the word w i , and when its value is -1, it means removing the word w i ; τ0 represents the basic threshold; γ represents the gain coefficient of the total value of the retained words with respect to the threshold; η represents the normalization factor of the value gain; means compressing the influence range of the retained value to [-1, 1] to prevent threshold drift.

Images