A semantic search method and device
By extracting and filtering the similarity between key words in the statement and the clustering center and documents, the comprehensive score is calculated, and the complexity and resource occupation of traditional semantic search is solved, efficient, fast and accurate semantic search results are achieved, and users are improved Experience.
Patent Information
- Application Number
- CN202410551204.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-07
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-05-07
Smart Images
Figure CN118484509B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data recognition, and in particular, relates to a method and device for semantic retrieval. Background Art
[0002] Semantic search refers to a search technique that aims to consider the semantic information of the user's query intent and the content of the query, rather than just the literal matching of keywords. This approach usually uses natural language processing (NLP) technology to understand and interpret the true meaning of the user's query. Compared with traditional keyword-based search, semantic search can more accurately understand the user's needs because it considers synonyms, context, intent, and multiple possible meanings of the query. Therefore, it can provide more relevant and precise search results. When implementing semantic search, various technologies are used, including but not limited to, Natural Language Processing (NLP): Technology for analyzing and understanding human language, including grammatical analysis, entity recognition, sentiment analysis, etc. Ontology and Knowledge Graph: Using a structured way to define the relationship between different concepts. Machine Learning and Artificial Intelligence: Learning and predicting user intent and the most relevant search results through models. Vector Space Models: Such as Word Embeddings, they can capture the semantic relationship between words and convert words into points in a continuous vector space so that their similarity can be calculated. Semantic search is very important in many applications, such as improving the effectiveness of search engines, improving recommendation systems, and enhancing automatic question-answering systems in customer service. With the continuous advancement of deep learning and NLP technology, the accuracy and efficiency of semantic retrieval are constantly improving.
[0003] From this we can see that traditional semantic retrieval has relatively complex processing logic and requires more computing resources. This is a technical problem that needs to be solved urgently. Summary of the invention
[0004] In view of this, an embodiment of the present invention provides a method and apparatus for semantic retrieval to solve the technical problem that traditional semantic retrieval has relatively complex processing logic and requires more computing resources.
[0005] A first aspect of an embodiment of the present invention provides a semantic retrieval method, wherein the load regulation is applied to a microgrid, and the semantic retrieval method includes:
[0006] Extract key words in the search statement;
[0007] Calculating the first Jaccard similarity coefficient between the key words and multiple target cluster centers, and obtaining multiple document data corresponding to the target cluster center with the smallest first Jaccard similarity coefficient in the database;
[0008] Calculating the similarity between the key words and the document names of the plurality of document data;
[0009] Extracting a plurality of first document data whose similarity is greater than a first threshold;
[0010] Acquire multiple content parts in the first document data, and traverse a first number of target key words in each of the content parts; the multiple content parts include an abstract, a claim, and a description;
[0011] Calculating a comprehensive score according to first preset weights corresponding to the plurality of content parts and a first number of the target key words;
[0012] According to the comprehensive score, the contents of the plurality of documents are sorted to obtain a semantic retrieval result.
[0013] Furthermore, the step of extracting key words in the search sentence includes:
[0014] Performing word segmentation processing on the search sentence to obtain multiple word units;
[0015] Matching the parts of speech of a plurality of the word units;
[0016] Obtaining the frequency of occurrence of the word unit in the search sentence;
[0017] Multiplying the occurrence frequency of the word unit by the second preset weight corresponding to the part of speech to obtain an importance score;
[0018] extracting target word units whose importance scores are greater than a second threshold;
[0019] The target word unit and the synonyms corresponding to the target word unit are used as the key words.
[0020] Furthermore, the step of calculating the similarity between the key words and the document names of the plurality of document data includes:
[0021] Counting key words in the document names;
[0022] Obtaining the importance score of the key words;
[0023] The similarity is obtained by adding up the multiple importance scores.
[0024] Furthermore, the step of calculating the comprehensive score according to the first preset weights corresponding to the plurality of content parts and the first number of the target key words includes:
[0025] Obtain the number of occurrences of the target key word in the search sentence, the number of occurrences of the target key word in the current content part, the total number of words in the search sentence and the total number of words in the current content part;
[0026] Calculating a similarity score based on the first number of target key words, the number of occurrences of the target key words in the search sentence, the number of occurrences of the target key words in the current content portion, the total number of words in the search sentence, and the total number of words in the current content portion;
[0027] Multiplying the similarity score corresponding to each of the content parts by the first preset weight corresponding to the content part to obtain a current similarity;
[0028] The maximum current similarity is used as the comprehensive score.
[0029] Further, the step of calculating the similarity score according to the first number of target key words, the number of occurrences of the target key words in the search sentence, the number of occurrences of the target key words in the current content part, the total number of words in the search sentence and the total number of words in the current content part includes:
[0030] Substituting the first number of target key words, the number of occurrences of the target key words in the search sentence, the number of occurrences of the target key words in the current content part, the total number of words in the search sentence and the total number of words in the current content part into a preset formula to obtain the similarity score;
[0031] The preset formula is:
[0032]
[0033] Among them, Score(T1,T2) represents the similarity score, N represents the first number of target key words, C 1,k represents the number of occurrences of the kth target key word in the search sentence, C 2,k represents the number of occurrences of the kth target key word in the current content part, W k represents the second preset weight of the part of speech corresponding to the k-th key word, f(N) represents the normalization function, γ represents the hyperparameter, S1 represents the total number of words in the search sentence, and S2 represents the total number of words in the current content part.
[0034] Furthermore, before the step of calculating the first Jaccard similarity coefficient between the key words and the plurality of target cluster centers and obtaining the plurality of document data corresponding to the target cluster center with the minimum first Jaccard similarity coefficient in the database, the method further includes:
[0035] Obtain multiple training document data corresponding to different classification numbers;
[0036] Extract summary content from the training document data corresponding to each classification number;
[0037] Extracting key training words from the summary content, encoding the key training words to obtain word codes;
[0038] The target cluster center is calculated according to the word codes corresponding to each abstract content.
[0039] Furthermore, the step of calculating the target cluster center according to the word code corresponding to each abstract content includes:
[0040] Calculating a window size according to the number of words in the summary content; the window size is used to accommodate a fixed number of summary contents;
[0041] According to the window size, a corresponding number of abstract contents are obtained from all abstract contents under each classification number;
[0042] Calculate the second Jaccard similarity coefficient between the word codes corresponding to the multiple summary contents in each window;
[0043] Acquire multiple identical word codes in the target summary content whose second Jaccard similarity coefficient exceeds the third threshold; the identical word codes refer to identical codes existing in the word codes corresponding to the two summary contents;
[0044] Encode multiple identical words as initial cluster centers;
[0045] Calculating the distances between multiple initial cluster centers, and merging the initial cluster centers whose distances are less than a fourth threshold to obtain a merged cluster center;
[0046] The unmerged cluster centers and the merged cluster centers are used as target cluster centers; the unmerged cluster centers refer to cluster centers that have not been merged.
[0047] A second aspect of an embodiment of the present invention provides a semantic search device, including:
[0048] A first extraction unit is used to extract key words in the search sentence;
[0049] A first calculation unit is used to calculate the first Jaccard similarity coefficient between the key word and multiple target cluster centers, and obtain multiple document data corresponding to the target cluster center with the minimum first Jaccard similarity coefficient in the database;
[0050] A second calculation unit, configured to calculate the similarity between the key words and the document names of the plurality of document data;
[0051] A second extraction unit, configured to extract a plurality of first document data having a similarity greater than a first threshold;
[0052] A traversal unit, configured to obtain a plurality of content parts in the first document data, and traverse a first number of target key words in each of the content parts; the plurality of content parts include an abstract, a claim, and a description;
[0053] A third calculation unit, configured to calculate a comprehensive score according to the first preset weights corresponding to the plurality of content parts and the first number of the target key words;
[0054] The sorting unit is used to sort the contents of the plurality of documents according to the comprehensive scores to obtain a semantic retrieval result.
[0055] A third aspect of an embodiment of the present invention provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.
[0056] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0057] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by extracting the key words in the search statement, the demand for processing the entire search statement is reduced, thereby reducing the consumption of computing resources. There is no need to perform complex natural language processing on the statement, and the Jaccard similarity coefficient between the keywords and the target cluster center is directly used for rapid screening, which greatly improves the efficiency of resource utilization. By calculating the similarity between the key words in the search statement and the target cluster center in the database, the search scope can be quickly narrowed, and relevant document data can be quickly located therefrom. This is faster than the traditional method of traversing the entire database, and the search speed is accelerated. By calculating the similarity between the document name and the key words, and setting a threshold to screen out documents with high relevance, the accuracy of the search is improved. Only those documents that are highly relevant to the search statement will be extracted, reducing the interference of irrelevant information. By obtaining the content parts such as the abstract, claims and instructions in the document data, and calculating the comprehensive score in combination with the preset weight and the number of target key words, the present invention can more comprehensively evaluate the relevance of the document and the search statement, and further strengthen the relevance of the results. Finally, the present invention provides a document content sorting method based on comprehensive scores, so that users can quickly obtain the most relevant search results based on the scores, significantly improving the user experience and reducing the time and effort of users to find the required content in massive information. In summary, the semantic retrieval method of the present invention has shown significant technical effects in improving efficiency, speeding up, increasing accuracy, strengthening relevance and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0059] Figure 1 A schematic flow chart of a semantic retrieval method provided by the present invention is shown;
[0060] Figure 2 A schematic diagram of a semantic search device provided by an embodiment of the present invention is shown;
[0061] Figure 3 A schematic diagram of a terminal device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0062] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.
[0063] In view of this, an embodiment of the present invention provides a method and apparatus for semantic retrieval to solve the technical problem that traditional semantic retrieval has relatively complex processing logic and requires more computing resources.
[0064] First, the present invention provides a semantic retrieval method. Figure 1 , Figure 1 FIG. 4 is a schematic flow chart of a semantic search method provided by the present invention. Figure 1 As shown, the semantic retrieval method may include the following steps:
[0065] Step 101: extract key words in the search sentence;
[0066] In order to improve the search efficiency, this embodiment extracts key words from the search sentence and then searches based on the key words to avoid unnecessary calculations caused by full-text search. The specific logic of extracting key words is as follows:
[0067] Specifically, step 101 specifically includes steps 1011 to 1016:
[0068] Step 1011: performing word segmentation processing on the search sentence to obtain a plurality of word units;
[0069] The search sentence is segmented and stop words are removed to obtain multiple word units. The segmentation process is a traditional technology and will not be described in detail here.
[0070] Step 1012: Matching the parts of speech of the plurality of word units;
[0071] Parts of speech include but are not limited to nouns, verbs, adjectives, and quantifiers, etc.
[0072] Step 1013: Obtain the occurrence frequency of the word unit in the search sentence;
[0073] Step 1014: multiplying the occurrence frequency of the word unit by the second preset weight corresponding to the part of speech to obtain an importance score;
[0074] Since different parts of speech have different importance in the search content, this application presets different weights for each part of speech to balance the impact of word units of different parts of speech on the search process, thereby improving the search accuracy.
[0075] The importance score is used to characterize the importance of the word unit in the retrieval process.
[0076] Step 1015: extracting target word units whose importance scores are greater than a second threshold;
[0077] Step 1016: The target word unit and the synonyms corresponding to the target word unit are used as the key words.
[0078] Since there may be multiple descriptions of the same meaning in the search content, this application uses the synonyms corresponding to the target word unit and the target word unit as key words.
[0079] In this embodiment, through efficient word segmentation processing and part-of-speech matching algorithm, the word units in the search sentence can be accurately extracted and analyzed. By accurately identifying the part of speech of the word unit, combined with the first preset weight set for each part of speech, the present solution can calculate the importance score of each word unit, thereby effectively measuring its relative importance in the search sentence. The frequency of occurrence is used as the key parameter for measuring the importance of words, ensuring that the scoring system can reflect the actual use of the word unit in the search sentence. By combining the frequency of occurrence of the word unit with the preset weight, the present solution can objectively identify the key words that have a decisive influence on the search intention. By setting a reasonable second threshold, the present technical solution can effectively filter out non-target word units with low scores, and only extract those target word units with importance scores higher than the threshold, which helps to improve the relevance and accuracy of the search results. This process not only optimizes the search efficiency, but also improves the quality of the search results. Finally, the present technical solution also takes into account the relevance of synonyms, and takes the target word unit and its corresponding synonyms as key words. This step greatly expands the semantic coverage of the search sentence and improves the comprehensiveness and flexibility of the search. This means that even if the user does not use the exact target vocabulary when expressing, this technical solution can ensure that the user's intention is accurately understood and effectively captured through the relevance of synonyms. It can significantly improve the performance of the information retrieval system and present more relevant and accurate search results to users.
[0080] Step 102: Calculate the first Jaccard similarity coefficient between the key word and multiple target cluster centers, and obtain multiple document data corresponding to the target cluster center with the minimum first Jaccard similarity coefficient in the database;
[0081] The Jaccard similarity coefficient, also known as the Jaccard index, is a statistic used to compare the similarity and diversity between finite sample sets. The coefficient measures the ratio between the size of the intersection and the size of the union of the members of two sets. When the coefficient is 1, it means that the two sets are exactly the same. When the coefficient is 0, it means that the two sets have no common elements. The larger the coefficient, the higher the similarity; the smaller the coefficient, the lower the similarity.
[0082] Among them, the target cluster center is the pre-trained cluster center, and the training logic is as follows:
[0083] Optionally, before step 102, steps A1 to A4 are also included:
[0084] Step A1: Obtain multiple training document data corresponding to different classification numbers;
[0085] In the patent database, the classification number is used to classify and store patent documents in different technical fields, and the classification number is a rough classification mode. There are still large differences in the document data under the same classification number, so it is necessary to perform further clustering processing based on the document data under the classification number.
[0086] Step A2: extracting summary content from the training document data corresponding to each classification number;
[0087] The summary content can reflect the main content of the entire document file, so the summary content in the training document data is extracted and clustered.
[0088] Step A3: extracting key training words from the summary content, encoding the key training words, and obtaining word codes;
[0089] The extraction logic of the key training words is consistent with step 1011 to step 1016, and will not be repeated here.
[0090] Step A4: Calculate the target cluster center according to the word code corresponding to each abstract content.
[0091] Specifically, step A4 includes steps A41 to A47:
[0092] Step A41: Calculate the window size according to the number of words in the summary content; the window size is used to accommodate a fixed number of summary contents;
[0093] The number interval of the number of matching words is used, and the window size corresponding to the number interval is used as the window size corresponding to the number of the words. Different number intervals are mapped to different window sizes.
[0094] Step A42: acquiring a corresponding number of summary contents from all summary contents under each classification number according to the window size;
[0095] All abstract contents under each classification number are divided by window size to obtain multiple data groups.
[0096] Step A43: calculating the second Jaccard similarity coefficient between the word codes corresponding to the multiple summary contents in each window;
[0097] Step A44: obtaining a plurality of identical word codes in the target summary content whose second Jaccard similarity coefficient exceeds a third threshold; the identical word codes refer to identical codes existing in the word codes corresponding to the two summary contents;
[0098] Step A45: Encoding multiple identical words as initial cluster centers;
[0099] The same words are encoded as the common points between different abstract contents, so they can be used as the initial clustering centers for clustering processing.
[0100] Step A46: calculating the distances between multiple initial cluster centers, and merging the initial cluster centers whose distances are less than a fourth threshold to obtain a merged cluster center;
[0101] Since the number of initial cluster centers is large and the distances between different initial cluster centers may be small, it is necessary to merge the initial cluster centers with smaller distances.
[0102] Step A47: using the unmerged cluster centers and the merged cluster centers as target cluster centers; the unmerged cluster centers refer to the cluster centers that have not been merged.
[0103] In this embodiment, the window size is calculated according to the number of words in the summary content to ensure that each window can accommodate a fixed number of summary contents, so as to effectively control the granularity and scope of cluster analysis, thereby improving the operability and flexibility of clustering. According to the calculated window size, a corresponding number of summary contents are selected under each classification number, which ensures the calculation efficiency of the summary data and provides a balanced and critical data basis for cluster analysis. The second Jaccard similarity coefficient between the word codes of multiple summary contents in each window is calculated, so as to identify highly similar word codes between the summary contents, which is conducive to revealing and capturing the core concepts and themes in the text data. By obtaining the same word codes in the target summary content whose second Jaccard similarity coefficient exceeds a specific threshold, this scheme can accurately identify the initial cluster center, providing an accurate starting point for subsequent cluster analysis. The distance between multiple initial cluster centers is calculated, and the initial cluster centers whose distance is less than a preset threshold are merged. This merging mechanism can effectively reduce overlapping or redundant cluster centers and improve the clarity and efficiency of clustering results. By using the unmerged cluster centers and the merged cluster centers as the final target cluster centers, this technical solution can ensure the diversity and accuracy of the cluster centers, facilitating more sophisticated clustering operations in the future. In general, this technical solution improves the accuracy and efficiency of text cluster analysis through a series of calculation and processing steps.
[0104] Step 103: Calculate the similarity between the key words and the document names of the plurality of document data;
[0105] In order to reduce the amount of calculation, the search and matching is first performed based on the document name of the document data, and the similarity between the key words and the document name is calculated. The multiple first document data with similarity greater than the first threshold are used as the search data basis to narrow the search scope and improve the search efficiency.
[0106] Specifically, step 103 specifically includes steps 1031 to 1033:
[0107] Step 1031: Count the key words in the document name;
[0108] Step 1032: Obtain the importance score of the key words;
[0109] Step 1033: Add multiple importance scores to obtain the similarity.
[0110] In this embodiment, by counting the key words in the document name, the theme and content focus of the document name can be grasped more accurately, so that the relevance between documents can be more accurately reflected when calculating the similarity. Obtaining the importance score of the key words enables the technical solution to distinguish the contribution of different words to the document similarity, thereby improving the accuracy and reliability of the similarity calculation. By adding multiple importance scores to obtain the similarity, the influence of multiple factors on the similarity can be comprehensively considered, thereby obtaining a more comprehensive similarity evaluation result. This technical solution can effectively filter out the noise caused by irrelevant words or low-importance words in the document name by paying attention to the key words and their importance scores, thereby improving the accuracy of the similarity calculation. Since the number of key words is often much less than the total number of words in the document, calculating the similarity of the key words can greatly reduce the amount of calculation compared to calculating the similarity of all words, thereby improving the processing speed.
[0111] Step 104: extracting a plurality of first document data whose similarity is greater than a first threshold;
[0112] Step 105: Acquire multiple content parts in the first document data, and traverse a first number of target key words in each of the content parts; the multiple content parts include an abstract, a claim, and a description;
[0113] The multiple content parts include but are not limited to one or more content parts such as abstract, claims and description.
[0114] Step 106: Calculate a comprehensive score according to the first preset weights corresponding to the plurality of content parts and the first number of the target key words;
[0115] Specifically, step 106 specifically includes steps 1061 to 1064:
[0116] Step 1061: Obtain the number of occurrences of the target key word in the search sentence, the number of occurrences of the target key word in the current content portion, the total number of words in the search sentence, and the total number of words in the current content portion;
[0117] Step 1062: Calculate a similarity score based on the first number of target key words, the number of occurrences of the target key words in the search sentence, the number of occurrences of the target key words in the current content portion, the total number of words in the search sentence, and the total number of words in the current content portion;
[0118] Specifically, step 1062 specifically includes: substituting the first number of target key words, the number of occurrences of the target key words in the search sentence, the number of occurrences of the target key words in the current content part, the total number of words in the search sentence and the total number of words in the current content part into a preset formula to obtain the similarity score;
[0119] The preset formula is:
[0120]
[0121] Among them, Score(T1,T2) represents the similarity score, N represents the first number of target key words, C 1,k represents the number of occurrences of the kth target key word in the search sentence, C 2,k represents the number of occurrences of the kth target key word in the current content part, W k represents the second preset weight of the part of speech corresponding to the k-th key word, f(N) represents the normalization function, γ represents the hyperparameter, S1 represents the total number of words in the search sentence, and S2 represents the total number of words in the current content part.
[0122] It is worth noting that is a length difference penalty term that reduces the score change caused by text length differences.
[0123] The normalization function f(N) adjusts the effect of the number of target key words on the total score. A logarithmic function is usually used to achieve this effect, ensuring that the total score does not grow excessively as the number of target key words increases.
[0124] The role of the normalization function f(N) in this formula is to balance the score, especially when the number of keywords is increasing. This is to prevent the similarity score from increasing indefinitely due to the increase in the number of target key words. Such a score increase does not necessarily reflect the actual similarity improvement between the texts. The normalization function usually chooses a function that grows as the input grows, but the growth rate slows down as the input increases. In this case, a common choice is a logarithmic function, for example:
[0125] f(N)=log(1+N)
[0126] log stands for natural logarithm. This function satisfies that when N grows, f(N) also grows, but its growth rate will gradually slow down. This ensures that when the number of target key words increases, the impact on the overall score is limited, avoiding excessive expansion of the score.
[0127] Step 1063: multiplying the similarity score corresponding to each content part by the first preset weight corresponding to the content part to obtain the current similarity;
[0128] Step 1064: taking the maximum current similarity as the comprehensive score.
[0129] In this embodiment, by obtaining the number of occurrences of the target key words in the search statement and the content part, combined with the total number of words in the search statement and the content part, the technical solution can accurately calculate the similarity score. Then, the technical solution multiplies the similarity score of each content part with the first preset weight corresponding to the content part to obtain the current similarity score. The parameter of the first preset weight allows differentiated importance assessments of different content parts, so that the score is more in line with the degree of importance of specific content parts in practical applications. Finally, the technical solution realizes a comprehensive evaluation of the relevance of the content part by selecting the maximum current similarity score as the comprehensive score. Such an evaluation mechanism ensures the relevance and accuracy of the search results, so that the final comprehensive score more truly reflects the distribution and importance of the target key words in the entire document or data set. Improve the relevance and accuracy of the search results, and ensure that users can quickly obtain content that is highly relevant to the search statement.
[0130] Step 107: Sort the contents of the plurality of documents according to the comprehensive scores to obtain semantic retrieval results.
[0131] In this embodiment, by extracting the key words in the search statement, the demand for processing the entire search statement is reduced, thereby reducing the consumption of computing resources. There is no need to perform complex natural language processing on the statement, and the Jaccard similarity coefficient between the keywords and the target cluster center is directly used for rapid screening, which greatly improves the efficiency of resource utilization. By calculating the similarity between the key words in the search statement and the target cluster center in the database, the search scope can be quickly narrowed, and relevant document data can be quickly located therefrom. This is faster than the traditional method of traversing the entire database, and the retrieval speed is accelerated. By calculating the similarity between the document name and the key words, and setting a threshold to screen out documents with high relevance, the accuracy of the retrieval is improved. Only those documents that are highly relevant to the search statement will be extracted, reducing the interference of irrelevant information. By obtaining the content parts such as the abstract, claims and instructions in the document data, and combining the preset weights and the number of target key words to calculate the comprehensive score, the present invention can more comprehensively evaluate the relevance of the document and the search statement, and further strengthen the relevance of the results. Finally, the present invention provides a document content sorting method based on comprehensive scores, so that users can quickly obtain the most relevant search results based on the scores, significantly improving the user experience and reducing the time and effort of users to find the required content in massive information. In summary, the semantic retrieval method of the present invention has shown significant technical effects in improving efficiency, speeding up, increasing accuracy, strengthening relevance and improving user experience.
[0132] like Figure 2 The present invention provides a semantic search device 2, see Figure 2 , Figure 2 A schematic diagram of a semantic search device provided by the present invention is shown, Figure 2 The apparatus for semantic retrieval includes:
[0133] A first extraction unit 21, used to extract key words in the search sentence;
[0134] A first calculation unit 22 is used to calculate the first Jaccard similarity coefficient between the key word and multiple target cluster centers, and obtain multiple document data corresponding to the target cluster center with the minimum first Jaccard similarity coefficient in the database;
[0135] A second calculation unit 23, used to calculate the similarity between the key word and the document names of the plurality of document data;
[0136] A second extraction unit 24 is used to extract a plurality of first document data whose similarity is greater than a first threshold;
[0137] A traversal unit 25 is used to obtain multiple content parts in the first document data, and traverse a first number of target key words in each of the content parts; the multiple content parts include an abstract, a claim, and a description;
[0138] A third calculation unit 26, configured to calculate a comprehensive score according to the first preset weights corresponding to the plurality of content parts and the first number of the target key words;
[0139] The sorting unit 27 is used to sort the contents of the plurality of documents according to the comprehensive scores to obtain a semantic retrieval result.
[0140] The present invention provides a semantic search device, which reduces the need for processing the entire search statement by extracting key words in the search statement, thereby reducing the consumption of computing resources. There is no need to perform complex natural language processing on the statement, and the Jaccard similarity coefficient between the keyword and the target cluster center is directly used for rapid screening, which greatly improves the efficiency of resource use. By calculating the similarity between the key words in the search statement and the target cluster center in the database, the search scope can be quickly narrowed, and relevant document data can be quickly located therefrom. This is faster than the traditional method of traversing the entire database, and the search speed is accelerated. By calculating the similarity between the document name and the key words, and setting a threshold to screen out documents with high relevance, the accuracy of the search is improved. Only those documents that are highly relevant to the search statement will be extracted, reducing the interference of irrelevant information. By obtaining the content parts such as the abstract, claims and instructions in the document data, and calculating the comprehensive score in combination with the preset weight and the number of target key words, the present invention can more comprehensively evaluate the relevance of the document and the search statement, and further strengthen the relevance of the result. Finally, the present invention provides a document content sorting method based on comprehensive scores, so that users can quickly obtain the most relevant search results based on the scores, significantly improving the user experience and reducing the time and effort of users to find the required content in massive information. In summary, the semantic retrieval method of the present invention has shown significant technical effects in improving efficiency, speeding up, increasing accuracy, strengthening relevance and improving user experience.
[0141] Figure 3 Schematic diagram of a terminal device provided by an embodiment of the present invention. Figure 3 As shown, a terminal device 3 of this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a semantic search program. When the processor 30 executes the computer program 32, the steps in the above-mentioned method embodiments of semantic search are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of each unit in the above-mentioned device embodiments are realized, for example, Figure 2 Function of the unit shown.
[0142] Exemplarily, the computer program 32 may be divided into one or more units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 32 in the terminal device 3. For example, the computer program 32 may be divided into the following specific functions of each unit:
[0143] A first extraction unit is used to extract key words in the search sentence;
[0144] A first calculation unit is used to calculate the first Jaccard similarity coefficient between the key word and multiple target cluster centers, and obtain multiple document data corresponding to the target cluster center with the minimum first Jaccard similarity coefficient in the database;
[0145] A second calculation unit, configured to calculate the similarity between the key words and the document names of the plurality of document data;
[0146] A second extraction unit, configured to extract a plurality of first document data having a similarity greater than a first threshold;
[0147] A traversal unit, configured to obtain a plurality of content parts in the first document data, and traverse a first number of target key words in each of the content parts; the plurality of content parts include an abstract, a claim, and a description;
[0148] A third calculation unit, configured to calculate a comprehensive score according to the first preset weights corresponding to the plurality of content parts and the first number of the target key words;
[0149] The sorting unit is used to sort the contents of the plurality of documents according to the comprehensive scores to obtain a semantic retrieval result.
[0150] The terminal device includes but is not limited to a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of a terminal device 3 and does not constitute a limitation on the terminal device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.
[0151] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0152] The memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. The memory 31 may also be an external storage device of the terminal device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the terminal device 3. The memory 31 is used to store the computer program and other programs and data required by the roaming control device. The memory 31 may also be used to temporarily store data that has been output or is to be output.
[0153] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0154] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0155] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0156] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0157] An embodiment of the present invention provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.
[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0160] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0161] In the embodiments provided by the present invention, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed over multiple network units.
[0163] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0164] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0165] As used in the present specification and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to monitoring, depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is monitored" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is monitored" or "in response to monitoring [described condition or event]", depending on the context.
[0166] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0167] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0168] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A semantic retrieval method, characterized in that: The semantic retrieval method comprises: Extract key words in the search statement; Calculating the first Jaccard similarity coefficient between the key words and multiple target cluster centers, and obtaining multiple document data corresponding to the target cluster center with the smallest first Jaccard similarity coefficient in the database; Calculating the similarity between the key words and the document names of the plurality of document data; Extracting a plurality of first document data whose similarity is greater than a first threshold; Acquire multiple content parts in the first document data, and traverse a first number of target key words in each of the content parts; the multiple content parts include an abstract, a claim, and a description; Obtain the number of occurrences of the target key word in the search sentence, the number of occurrences of the target key word in the current content part, the total number of words in the search sentence and the total number of words in the current content part; Substituting the first number of target key words, the number of occurrences of the target key words in the search sentence, the number of occurrences of the target key words in the current content part, the total number of words in the search sentence and the total number of words in the current content part into a preset formula to obtain a similarity score; The preset formula is: Among them, Score(T1,T2) represents the similarity score, N represents the first number of target key words, C 1,k represents the number of occurrences of the kth target key word in the search sentence, C 2,k represents the number of occurrences of the kth target key word in the current content part, W k represents the second preset weight of the part of speech corresponding to the k-th key word, f(N) represents the normalization function, γ represents the hyperparameter, S1 represents the total number of words in the search sentence, and S2 represents the total number of words in the current content part; Multiplying the similarity score corresponding to each of the content parts by the first preset weight corresponding to the content part to obtain a current similarity; The maximum current similarity is used as the comprehensive score; According to the comprehensive score, the contents of the plurality of documents are sorted to obtain a semantic retrieval result.
2. The semantic search method according to claim 1, characterized in that: The step of extracting key words in the search sentence comprises: Performing word segmentation processing on the search sentence to obtain multiple word units; Matching the parts of speech of a plurality of the word units; Obtaining the frequency of occurrence of the word unit in the search sentence; Multiplying the occurrence frequency of the word unit by the second preset weight corresponding to the part of speech to obtain an importance score; extracting target word units whose importance scores are greater than a second threshold; The target word unit and the synonyms corresponding to the target word unit are used as the key words.
3. The semantic search method according to claim 2, characterized in that: The step of calculating the similarity between the key words and the document names of the plurality of document data comprises: Counting key words in the document names; Obtaining the importance score of the key words; The similarity is obtained by adding up the multiple importance scores.
4. The semantic search method according to claim 1, characterized in that: Before the step of calculating the first Jaccard similarity coefficient between the key words and the multiple target cluster centers and obtaining multiple document data corresponding to the target cluster center with the minimum first Jaccard similarity coefficient in the database, the method further includes: Obtain multiple training document data corresponding to different classification numbers; Extract summary content from the training document data corresponding to each classification number; Extracting key training words from the summary content, encoding the key training words to obtain word codes; The target cluster center is calculated according to the word codes corresponding to each abstract content.
5. The semantic search method according to claim 4, characterized in that: The step of calculating the target cluster center according to the word code corresponding to each abstract content includes: Calculating a window size according to the number of words in the summary content; the window size is used to accommodate a fixed number of summary contents; According to the window size, a corresponding number of abstract contents are obtained from all abstract contents under each classification number; Calculate the second Jaccard similarity coefficient between the word codes corresponding to the multiple summary contents in each window; Acquire multiple identical word codes in the target summary content whose second Jaccard similarity coefficient exceeds the third threshold; the identical word codes refer to identical codes existing in the word codes corresponding to the two summary contents; Encode multiple identical words as initial cluster centers; Calculating the distances between multiple initial cluster centers, and merging the initial cluster centers whose distances are less than a fourth threshold to obtain a merged cluster center; The unmerged cluster centers and the merged cluster centers are used as target cluster centers; the unmerged cluster centers refer to cluster centers that have not been merged.
6. A semantic search device, characterized in that: The semantic retrieval device comprises: A first extraction unit is used to extract key words in the search sentence; A first calculation unit is used to calculate the first Jaccard similarity coefficient between the key word and multiple target cluster centers, and obtain multiple document data corresponding to the target cluster center with the minimum first Jaccard similarity coefficient in the database; A second calculation unit, configured to calculate the similarity between the key words and the document names of the plurality of document data; A second extraction unit, configured to extract a plurality of first document data having a similarity greater than a first threshold; A traversal unit, configured to obtain a plurality of content parts in the first document data, and traverse a first number of target key words in each of the content parts; the plurality of content parts include an abstract, a claim, and a description; A third calculation unit is used to obtain the number of occurrences of the target key word in the search sentence, the number of occurrences of the target key word in the current content part, the total number of words in the search sentence and the total number of words in the current content part; Substituting the first number of target key words, the number of occurrences of the target key words in the search sentence, the number of occurrences of the target key words in the current content part, the total number of words in the search sentence and the total number of words in the current content part into a preset formula to obtain a similarity score; The preset formula is: Among them, Score(T1,T2) represents the similarity score, N represents the first number of target key words, C 1,k represents the number of occurrences of the kth target key word in the search sentence, C 2,k represents the number of occurrences of the kth target key word in the current content part, W k represents the second preset weight of the part of speech corresponding to the k-th key word, f(N) represents the normalization function, γ represents the hyperparameter, S1 represents the total number of words in the search sentence, and S2 represents the total number of words in the current content part; Multiplying the similarity score corresponding to each of the content parts by the first preset weight corresponding to the content part to obtain a current similarity; The maximum current similarity is used as the comprehensive score; The sorting unit is used to sort the contents of the plurality of documents according to the comprehensive scores to obtain a semantic retrieval result.
7. A terminal device, characterized in that: The device comprises: a memory, a processor, and a load regulation program stored in the memory and executable on the processor, wherein the load regulation program is configured to implement the steps of semantic retrieval as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.