Text matching method and device

Abstract

The invention discloses a text matching method and device, and belongs to the technical field of text retrieval. The text matching method comprises the following steps: processing an obtained initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library; processing the obtained problem text to obtain a third vector corresponding to the problem text; obtaining a first similarity between the question text and the paragraph text based on the first vector and the third vector; based on the second vector and the third vector, obtaining a second similarity between the question text and the paragraph text; processing at least one keyword included in the question text to obtain the importance degree of each keyword to the paragraph text; and determining a target paragraph text corresponding to the question text from the plurality of paragraph texts based on the first similarity, the second similarity and the importance degree. According to the text matching method, the precision and the accuracy of a text matching result are improved.

Description

Technical Field

[0001] The present application belongs to the field of text retrieval technology, and in particular, relates to a text matching method and device. Background Art

[0002] With the rise of large language model technology, the retrieval of private domain knowledge has attracted widespread attention from users, and there is an urgent need to solve how to match the knowledge fragments that can solve the problem from the massive local private domain knowledge base according to the user's question. There are methods of text matching through deep learning in related technologies, but this method is highly dependent on training data and is not suitable for retrieval from knowledge bases that have never participated in training. The applicable scenarios are limited, and the accuracy and precision of text matching are low. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a text matching method and device, which improves the precision and accuracy of the text matching results; and does not need to rely on training data sets for text matching, and can also search for knowledge bases that have never participated in training, thus broadening the applicable scenarios.

[0004] In a first aspect, the present application provides a text matching method, the method comprising:

[0005] Processing the acquired initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library; processing the acquired question text to obtain a third vector corresponding to the question text;

[0006] Based on the first vector and the third vector, a first similarity between the question text and the paragraph text is obtained; based on the second vector and the third vector, a second similarity between the question text and the paragraph text is obtained;

[0007] Processing at least one keyword included in the question text to obtain the importance of each keyword to the paragraph text;

[0008] Based on the first similarity, the second similarity and the importance, a target paragraph text corresponding to the question text is determined from the plurality of paragraph texts.

[0009] According to the text matching method provided in the embodiment of the present application, semantic embedding is performed at the paragraph granularity and the sentence granularity respectively to obtain the similarity between the question text and the paragraph text from different angles, and the target paragraph text corresponding to the question text is matched from multiple paragraph texts in combination with the importance of the keywords in the question text. This can more comprehensively obtain all the information in the local knowledge text and improve the precision and accuracy of the text matching results. Moreover, there is no need to rely on training data sets for text matching, and knowledge bases that have never participated in training can also be retrieved, which broadens the applicable scenarios.

[0010] In a text matching method according to an embodiment of the present application, the processing of at least one keyword included in the question text to obtain the importance of each keyword to the paragraph text includes:

[0011] Processing the question text to obtain the keyword quantity of the at least one keyword included in the question text and all the keywords included in the question text;

[0012] Based on the question text and the initial text library, obtaining the inverse document frequency corresponding to each of the keywords and the occurrence frequency of each of the keywords in the initial text library;

[0013] The importance of the keyword to the paragraph text is calculated based on the number of keywords, the inverse document frequency and the occurrence frequency.

[0014] In a text matching method according to an embodiment of the present application, determining a target paragraph text corresponding to the question text from a plurality of paragraph texts based on the first similarity, the second similarity and the importance, comprises:

[0015] The first similarity, the second similarity and the importance are fused to obtain a third similarity between the paragraph text and the question text;

[0016] The paragraph text corresponding to the target third similarity greater than the target similarity threshold among the plurality of the third similarities is determined as the target paragraph text.

[0017] In a text matching method according to an embodiment of the present application, the first similarity, the second similarity and the importance are fused to obtain a third similarity between the paragraph text and the question text, including:

[0018] Obtaining a first weight of the first similarity relative to the third similarity, a second weight of the second similarity relative to the third similarity, and a third weight of the importance relative to the third similarity;

[0019] Based on the first weight, the second weight and the third weight, the first similarity, the second similarity and the importance are fused to obtain the third similarity.

[0020] In a text matching method according to an embodiment of the present application, obtaining a second similarity between the question text and the paragraph text based on the second vector and the third vector includes:

[0021] Based on the second vector and the third vector, a fourth similarity between the question text and the sentence text is calculated;

[0022] Based on the mapping relationship between the sentence text and the paragraph text, a maximum value among fourth similarities corresponding to a plurality of sentence texts included in the paragraph text is determined as a second similarity corresponding to the paragraph text.

[0023] In a text matching method according to an embodiment of the present application, the acquired initial text library is processed to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library, including:

[0024] Segmenting the initial text library into a plurality of paragraph texts;

[0025] Performing vectorization processing on each of the paragraph texts to obtain a first vector corresponding to each of the paragraph texts;

[0026] Segmenting the initial text library into a plurality of sentence texts;

[0027] Vectorization is performed on each of the sentence texts to obtain a second vector corresponding to each of the sentence texts.

[0028] In a second aspect, the present application provides a text matching device, comprising:

[0029] A first processing module is used to process the acquired initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library; and to process the acquired question text to obtain a third vector corresponding to the question text;

[0030] A second processing module, configured to obtain a first similarity between the question text and the paragraph text based on the first vector and the third vector; and obtain a second similarity between the question text and the paragraph text based on the second vector and the third vector;

[0031] A third processing module is used to process at least one keyword included in the question text to obtain the importance of each keyword relative to the question text and the initial text library;

[0032] The fourth processing module is used to determine a target paragraph text corresponding to the question text from the plurality of paragraph texts based on the first similarity, the second similarity and the importance.

[0033] According to the text matching device provided in the embodiment of the present application, semantic embedding is performed at the paragraph granularity and the sentence granularity respectively to obtain the similarity between the question text and the paragraph text at different angles, and the target paragraph text corresponding to the question text is matched from multiple paragraph texts in combination with the importance of the keywords in the question text. This can more comprehensively obtain all the information in the local knowledge text and improve the precision and accuracy of the text matching results. Moreover, there is no need to rely on training data sets for text matching, and knowledge bases that have never participated in training can also be retrieved, which broadens the applicable scenarios.

[0034] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text matching method as described in the first aspect above when executing the computer program.

[0035] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the text matching method as described in the first aspect above.

[0036] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the text matching method as described in the first aspect above.

[0037] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0038] By performing semantic embedding at the paragraph granularity and sentence granularity respectively, the similarity between the question text and the paragraph text from different angles is obtained, and the target paragraph text corresponding to the question text is matched from multiple paragraph texts based on the importance of the keywords in the question text. This can more comprehensively obtain all the information in the local knowledge text and improve the precision and accuracy of the text matching results. Moreover, there is no need to rely on training data sets for text matching, and knowledge bases that have never participated in training can also be retrieved, which broadens the applicable scenarios.

[0039] Furthermore, by performing semantic embedding at the paragraph granularity and sentence granularity respectively, and obtaining the similarity between the paragraph text and the question text calculated at different granularities through mapping processing, all the information included in the local knowledge segment can be obtained more comprehensively, thereby improving the precision and accuracy of subsequent text matching.

[0040] Furthermore, by extracting keywords from the question text and calculating the corresponding importance of the keywords based on the inverse document frequency of the keywords and the frequency of occurrence in local documents, the non-semantic information in the local knowledge is obtained based on the importance. This can take into account the importance of keywords in both the general database and the local knowledge base, thereby improving the precision and accuracy of text matching.

[0041] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0043] Figure 1 This is one of the flowcharts of the text matching method provided in the embodiment of the present application;

[0044] Figure 2 This is the second flowchart of the text matching method provided in the embodiment of the present application;

[0045] Figure 3 is a structural diagram of a text matching device provided in an embodiment of the present application;

[0046] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0048] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0049] The text matching method, text matching device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0050] The text matching method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0051] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0052] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0053] The text matching method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the text matching method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The text matching method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.

[0054] like Figure 1 As shown, the text matching method includes: step 110, step 120, step 130 and step 140.

[0055] Step 110: Process the acquired initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library; process the acquired question text to obtain a third vector corresponding to the question text;

[0056] In this step, the initial text library may include a local text library.

[0057] The initial text base can be extracted from locally stored knowledge files.

[0058] The locally stored knowledge files may include files in the form of PDF, TXT, DOC, Excel, images or voice.

[0059] For example, for files containing text content such as PDF, TXT, DOC, and Excel, you can use the corresponding library to read and parse the file content.

[0060] For images in formats such as PNG and JPG, the text in the image can be converted into editable text through OCR (optical character recognition) technology.

[0061] For voice files, speech can be converted into text through speech recognition technology.

[0062] The initial text library may include text data such as common words, phrases, terms, knowledge items, and articles.

[0063] The initial text library may include a plurality of paragraph texts, wherein the paragraph text may include a plurality of sentence texts.

[0064] Paragraph text can include paragraphs, chapters, or topic blocks, etc.

[0065] Processing the acquired initial text library may include preprocessing and feature extraction of the initial text library.

[0066] The question text may be the question to be searched entered by the user.

[0067] The first vector is obtained by vectorizing the paragraph text. For example, the paragraph text may be converted into a numerical form to obtain the first vector.

[0068] The second vector is obtained by vectorizing the sentence text. For example, the sentence text may be converted into a numerical form to obtain the second vector.

[0069] The third vector is obtained by vectorizing the question text. For example, the question text may be converted into a numerical form to obtain the third vector.

[0070] Among them, the text can be vectorized based on the bag-of-words model, TF-IDF (Term Frequency-Inverse Document Frequency), or word embedding, which can be selected based on user needs and is not limited in this application.

[0071] In some embodiments, processing the acquired initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library may include:

[0072] Segment the initial text base into multiple paragraphs of text;

[0073] Perform vectorization processing on each paragraph text to obtain a first vector corresponding to each paragraph text;

[0074] Segment the initial text base into multiple sentence texts;

[0075] Each sentence text is vectorized to obtain a second vector corresponding to each sentence text.

[0076] In this embodiment, the initial text library may be segmented into a plurality of paragraph texts according to a preset rule or logic.

[0077] For example, a text processing library may be used to identify paragraph separators (eg, line breaks or specific HTML tags, etc.) in a text, and then the initial text library may be segmented into a plurality of paragraph texts according to the paragraph separators.

[0078] By converting the paragraph text into a numerical vector form that can be understood and processed by a computer, the first vector corresponding to each paragraph text can be obtained.

[0079] For example, each word in a paragraph can be converted into a vector using word embedding techniques in natural language processing (such as Word2Vec, GloVe, or BERT).

[0080] For the entire paragraph, the vector representation of the paragraph (ie, the first vector) can be obtained by aggregating the vectors of all words in the paragraph (such as summing, averaging, or TF-IDF weighted summing, etc.).

[0081] You can also directly use models or libraries (such as Doc2Vec, etc.) to obtain vector representations of paragraphs.

[0082] Each paragraph text may be segmented into a plurality of natural sentences, for example, segmentation may be performed by identifying sentence terminators (eg, a period, a question mark, or an exclamation mark, etc.).

[0083] For example, the vector representation of the sentence (i.e., the second vector) can be obtained by aggregating the vectors of all words in the sentence through word embedding technology.

[0084] Alternatively, you can use a sentence embedding model (such as Sentence-BERT or SBERT, etc.) to obtain a vector representation of the sentence.

[0085] According to the text matching method provided in the embodiment of the present application, by converting the original knowledge file into paragraph text and sentence text that are easy to manage and use, and converting the paragraph text and sentence text into a vector form that is easy to process, a basis is provided for subsequent knowledge management, retrieval, question-answering and other applications, the processing volume of text matching is reduced, and processing efficiency is improved.

[0086] Step 120: Based on the first vector and the third vector, obtain a first similarity between the question text and the paragraph text; based on the second vector and the third vector, obtain a second similarity between the question text and the paragraph text;

[0087] In this step, the similarity between the first vector and the third vector may be calculated based on a similarity measurement method to obtain a first similarity.

[0088] For example, the first similarity may be calculated by cosine similarity or the inverse of the Euclidean distance.

[0089] The similarity may be a score, or may be in other forms, and may be customized based on user needs, and this application does not limit this.

[0090] The similarity between each sentence text and the question text may be calculated based on the second vector and the third vector, and then the similarities corresponding to the sentence texts may be mapped to obtain the second similarity between the paragraph text and the question text.

[0091] In some embodiments, after obtaining the similarity between each sentence text included in the paragraph text and the question text, a second similarity between the paragraph text and the question text may be determined by an aggregation method.

[0092] For example, the average value or weighted average value between the similarities corresponding to multiple sentence texts can be determined as the second similarity, or the maximum value among the similarities corresponding to multiple sentence texts can be determined as the second similarity. The selection can be based on user needs and is not limited in this application.

[0093] The first similarity and the second similarity are similarities between the paragraph text and the question text calculated based on different granularities.

[0094] In some embodiments, obtaining the second similarity between the question text and the paragraph text based on the second vector and the third vector may include:

[0095] Based on the second vector and the third vector, a fourth similarity between the question text and the sentence text is calculated;

[0096] Based on the mapping relationship between the sentence text and the paragraph text, the maximum value among the fourth similarities corresponding to the plurality of sentence texts included in the paragraph text is determined as the second similarity corresponding to the paragraph text.

[0097] In this embodiment, the fourth similarity can be obtained by calculating the similarity between the third vector corresponding to the question text and the second vector corresponding to each sentence text.

[0098] Among them, the fourth similarity can be calculated using a text embedding model, etc.

[0099] The fourth similarity may be obtained based on cosine similarity, Euclidean distance or other similarity measurement methods.

[0100] Each paragraph text includes a plurality of sentence texts. Based on the mapping relationship between the sentence text and the paragraph text, the fourth similarities corresponding to the plurality of sentence texts included in the paragraph text can be obtained from the plurality of fourth similarities.

[0101] The maximum value among the fourth similarities corresponding to the plurality of sentence texts included in the paragraph text may be determined as the second similarity corresponding to the paragraph text.

[0102] According to the text matching method provided in the embodiment of the present application, by performing semantic embedding at the paragraph granularity and the sentence granularity respectively, and obtaining the similarity between the paragraph text and the question text calculated at different granularities through mapping processing, all the information included in the local knowledge segment can be obtained more comprehensively, thereby improving the precision and accuracy of subsequent text matching.

[0103] In the actual implementation process, Figure 2 As shown, based on the local knowledge fragmentation module 1, the knowledge in the local knowledge base can be cut into paragraph text Seg_p and sentence text Seg_s at the paragraph granularity and sentence granularity respectively, and Seg_p and Seg_s are input into the text vectorization module 2.

[0104] Seg_p and Seg_s can be vectorized respectively based on the text embedding model in the text vectorization module 2 to obtain a first vector Vec_p and a second vector Vec_s, and save them locally.

[0105] The question text Q is input into the text vectorization module 2, and the third vector Vec_q output by the text vectorization module 2 can be obtained.

[0106] By performing cosine operations on the third vector Vec_q with the first vector Vec_p and the second vector Vec_s respectively, we can obtain the score value S_p (i.e., the first similarity) and the score value S_s (i.e., the fourth similarity) of the question text Q with respect to Seg_p and Seg_s, wherein S_p and S_s can be normalized, and then S_s can be remapped from the sentence granularity score back to the paragraph granularity score. For example, the value with the largest sentence score in the paragraph can be taken as the paragraph score to obtain the second similarity of the question text with respect to the paragraph text.

[0107] Among them, the text embedding model can be an open source model, or it can be a trained model, which can be selected based on user needs and is not limited in this application.

[0108] The first vector and the second vector output by the text embedding model satisfy: Vec_p∈R n×d , Vec_s∈R m×d , where n and m are the number of segmented paragraph texts and sentence texts respectively, and d is the dimension of the text embedding model.

[0109] The first similarity S_p and the fourth similarity S_s are vectors with dimensions n and m respectively.

[0110] S_p and S_s may be normalized based on Min-Max Normalization, Z-score normalization (standard deviation normalization), or linear scaling, etc., and may be selected based on user needs, and this application does not limit this.

[0111] Step 130: Process at least one keyword included in the question text to obtain the importance of each keyword to the paragraph text;

[0112] In this step, keyword extraction may be performed on the question text to obtain at least one keyword included in the question text.

[0113] For example, keywords can be extracted from the question text based on the jieba package, pkuseg package, TF-IDF (term frequency-inverse document frequency), part of speech or machine learning methods in Python. The selection can be based on user needs and is not limited in this application.

[0114] For each keyword, the weight of the keyword in the question text and the weight of the keyword in the initial text library can be calculated. Then, the keyword score can be obtained based on the weight of the keyword in the question text and in the initial text library, and the importance of the keyword to the paragraph text can be calculated based on the keyword score.

[0115] In some embodiments, the semantic similarity between the keyword and the question text or other content in the initial text library may also be calculated, and then the keyword score may be obtained based on the semantic similarity.

[0116] For example, word embeddings (such as Word2Vec, GloVe, BERT embeddings, etc.) can be used to calculate the semantic similarity corresponding to keywords.

[0117] In some embodiments, step 130 may include:

[0118] Processing the question text to obtain the keyword quantity of at least one keyword included in the question text and all keywords included in the question text;

[0119] Based on the question text and the initial text library, obtain the inverse document frequency corresponding to each keyword and the frequency of occurrence of each keyword in the initial text library;

[0120] Based on the number of keywords, inverse document frequency and occurrence frequency, the importance of keywords to the paragraph text is calculated.

[0121] In this embodiment, text analysis may be performed on the question text. For example, part-of-speech tagging, named entity recognition, or a keyword extraction algorithm may be used to extract at least one keyword included in the question text.

[0122] The number of keywords is the total number of all keywords included in the question text.

[0123] Inverse document frequency is used to measure the general importance of keywords.

[0124] When a keyword appears more frequently in a certain text and less frequently in other texts, the inverse document frequency of the keyword is higher.

[0125] The frequency of occurrence is the total number of times or the average number of times a keyword appears in the initial text library. The frequency of occurrence can be used to characterize the prevalence and importance of a keyword in the overall text data.

[0126] For example, when the frequency of occurrence of a keyword in the initial text library is high, it can be considered that the prevalence and importance of the keyword in the initial text library are both high.

[0127] In the actual implementation process, Figure 2 As shown, the question text Q can be input into the keyword extraction module 3 to obtain the IDF value (inverse document frequency) corresponding to each keyword output by the keyword extraction module 3, the number of keywords Key_num and the frequency KeyFre_i of each keyword appearing in the local knowledge Seg_p.

[0128] Among them, the inverse document frequency corresponding to the keyword can be calculated based on the following formula:

[0129]

[0130] Among them, IDF i For keyword q i The corresponding inverse document frequency, n(q i ) contains the keyword q i N is the total number of documents in the general knowledge base.

[0131] Based on the number of keywords, inverse document frequency and occurrence frequency, the keyword score can be calculated, where the keyword score is the relative importance of the keyword, which can be used to characterize the importance of the keyword in the general knowledge base and the local proprietary knowledge base.

[0132] In the actual implementation process, Figure 2 As shown, the keyword score can be calculated based on the formula:

[0133]

[0134] Among them, KeyScore_i is the keyword score, IDF_i is the keyword q iThe corresponding inverse document frequency, Key_num is the number of keywords, KeyFre_i is the frequency of occurrence, k0 and k1 are hyperparameters, which can be searched and obtained through grid search method or swarm intelligence optimization algorithm.

[0135] Based on the above formula, it can be seen that the keyword score is proportional to the inverse document frequency of the keyword in the general knowledge base, and inversely proportional to the frequency of occurrence of the keyword in the local knowledge base and the number of all keywords included in the question text.

[0136] The importance of keywords to paragraph text can be determined based on the formula:

[0137]

[0138] Among them, Seg_pkj is the importance of the i-th keyword to the j-th paragraph text, and KeyScore_i is the keyword score.

[0139] According to the text matching method provided in the embodiment of the present application, keywords are extracted from the question text, and the importance of the keywords is calculated based on the inverse document frequency of the keywords and the frequency of occurrence in local documents, so as to obtain non-semantic information in local knowledge based on the importance. The importance of keywords in the general database and the local knowledge base can be taken into account at the same time, thereby improving the precision and accuracy of text matching.

[0140] Step 140: Determine a target paragraph text corresponding to the question text from the plurality of paragraph texts based on the first similarity, the second similarity and the importance.

[0141] In this step, the multiple paragraph texts are a text set of candidate answers corresponding to the question text.

[0142] The first similarity and the second similarity may be weighted and summed to obtain a comprehensive similarity score.

[0143] The importance of the keywords is then incorporated into the comprehensive similarity score to obtain the final similarity score, and the paragraph text with the highest similarity score is used as the target paragraph text.

[0144] The target paragraph text is the paragraph text that can answer the question text.

[0145] According to the text matching method provided in the embodiment of the present application, semantic embedding is performed at the paragraph granularity and the sentence granularity respectively to obtain the similarity between the question text and the paragraph text from different angles, and the target paragraph text corresponding to the question text is matched from multiple paragraph texts in combination with the importance of the keywords in the question text. This can more comprehensively obtain all the information in the local knowledge text and improve the precision and accuracy of the text matching results. Moreover, there is no need to rely on training data sets for text matching, and knowledge bases that have never participated in training can also be retrieved, which broadens the applicable scenarios.

[0146] In some embodiments, step 140 may include:

[0147] The first similarity, the second similarity and the importance are fused to obtain the third similarity between the paragraph text and the question text;

[0148] The paragraph text corresponding to the target third similarity greater than the target similarity threshold among the multiple third similarities is determined as the target paragraph text.

[0149] In this embodiment, the first similarity, the second similarity and the importance may be fused based on a fusion strategy. For example, the fusion may be performed by average fusion, product fusion or weighted fusion.

[0150] The third similarity is used to characterize the overall matching degree between the paragraph text and the question text.

[0151] The target similarity threshold is a preset threshold used to determine whether the similarity between the paragraph text and the question text is high enough.

[0152] The target similarity threshold can be determined based on the specific application scenario and data set, and is not limited in this application.

[0153] By traversing the third similarities corresponding to all paragraph texts, a target third similarity greater than a target similarity threshold can be obtained, and the paragraph text corresponding to the target third similarity is determined as the target paragraph text.

[0154] When there are multiple paragraph texts that meet the conditions, the target paragraph text can be determined from the multiple paragraph texts according to other standards (such as text length, readability, information content, etc.).

[0155] In the actual implementation process, Figure 2 As shown, each third similarity can be input into the receiving-rejecting module 5 to obtain the target paragraph text output by the receiving-rejecting module 5.

[0156] When the third similarity is greater than the target similarity threshold, it means that the paragraph text corresponding to the third similarity has sufficient confidence to become the object of question text retrieval.

[0157] The receiving-rejecting module 5 may be obtained by training an MLP model, or may be other types of models, which is not limited in this application.

[0158] In some embodiments, the first similarity, the second similarity and the importance are fused to obtain a third similarity between the paragraph text and the question text, which may include:

[0159] Obtaining a first weight of the first similarity relative to the third similarity, a second weight of the second similarity relative to the third similarity, and a third weight of the importance relative to the third similarity;

[0160] Based on the first weight, the second weight and the third weight, the first similarity, the second similarity and the importance are fused to obtain the third similarity.

[0161] In this embodiment, the initial weights corresponding to the similarities can be determined according to user preferences or data characteristics, and then different weight combinations can be used to conduct experiments on the test set to evaluate the performance of the system under different weights, and the initial weights can be adjusted according to the experimental results to obtain the first weight, the second weight, and the third weight.

[0162] The first weight is used to represent the importance of the first similarity in evaluating the overall similarity.

[0163] The second weight is used to represent the importance of the second similarity in evaluating the overall similarity.

[0164] The third weight is used to characterize the importance of the importance in evaluating the overall similarity.

[0165] The first similarity, the second similarity and the importance are weighted and summed based on the first weight, the second weight and the third weight to obtain the third similarity.

[0166] In the actual implementation process, the first similarity, the second similarity and the importance of the keyword to the paragraph text can be integrated based on the following formula to obtain the third similarity of the paragraph text to the question text:

[0167] Seg_sj=S_pj+S_sj+Seg_pk

[0168] Among them, Seg_sj is the third similarity, S_pj is the first similarity, S_sj is the second similarity, and Seg_pk is the importance of the keyword to the paragraph text.

[0169] The text matching device provided in the present application is described below. The text matching device described below and the text matching method described above can be referenced to each other.

[0170] The text matching method provided in the embodiment of the present application can be executed by a text matching device. In the embodiment of the present application, the text matching device provided in the embodiment of the present application is described by taking the text matching method executed by the text matching device as an example.

[0171] The embodiment of the present application also provides a text matching device.

[0172] like Figure 3 As shown, the text matching device includes: a first processing module 310 , a second processing module 320 , a third processing module 330 and a fourth processing module 340 .

[0173] The first processing module 310 is used to process the acquired initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library; and to process the acquired question text to obtain a third vector corresponding to the question text;

[0174] The second processing module 320 is used to obtain a first similarity between the question text and the paragraph text based on the first vector and the third vector; and obtain a second similarity between the question text and the paragraph text based on the second vector and the third vector;

[0175] The third processing module 330 is used to process at least one keyword included in the question text to obtain the importance of each keyword relative to the question text and the initial text library;

[0176] The fourth processing module 340 is used to determine a target paragraph text corresponding to the question text from the plurality of paragraph texts based on the first similarity, the second similarity and the importance.

[0177] According to the text matching device provided in the embodiment of the present application, semantic embedding is performed at the paragraph granularity and the sentence granularity respectively to obtain the similarity between the question text and the paragraph text at different angles, and the target paragraph text corresponding to the question text is matched from multiple paragraph texts in combination with the importance of the keywords in the question text. This can more comprehensively obtain all the information in the local knowledge text and improve the precision and accuracy of the text matching results. Moreover, there is no need to rely on training data sets for text matching, and knowledge bases that have never participated in training can also be retrieved, which broadens the applicable scenarios.

[0178] In some embodiments, the third processing module 330 may also be used to:

[0179] Processing the question text to obtain the keyword quantity of at least one keyword included in the question text and all keywords included in the question text;

[0180] Based on the question text and the initial text library, obtain the inverse document frequency corresponding to each keyword and the frequency of occurrence of each keyword in the initial text library;

[0181] Based on the number of keywords, inverse document frequency and occurrence frequency, the importance of keywords to the paragraph text is calculated.

[0182] In some embodiments, the fourth processing module 340 may also be used to:

[0183] The first similarity, the second similarity and the importance are fused to obtain the third similarity between the paragraph text and the question text;

[0184] The paragraph text corresponding to the target third similarity greater than the target similarity threshold among the multiple third similarities is determined as the target paragraph text.

[0185] In some embodiments, the fourth processing module 340 may also be used to:

[0186] Obtaining a first weight of the first similarity relative to the third similarity, a second weight of the second similarity relative to the third similarity, and a third weight of the importance relative to the third similarity;

[0187] Based on the first weight, the second weight and the third weight, the first similarity, the second similarity and the importance are fused to obtain the third similarity.

[0188] In some embodiments, the second processing module 320 may also be used to:

[0189] Based on the second vector and the third vector, a fourth similarity between the question text and the sentence text is calculated;

[0190] Based on the mapping relationship between the sentence text and the paragraph text, the maximum value among the fourth similarities corresponding to the plurality of sentence texts included in the paragraph text is determined as the second similarity corresponding to the paragraph text.

[0191] In some embodiments, the first processing module 310 may also be used to:

[0192] Segment the initial text base into multiple paragraphs of text;

[0193] Perform vectorization processing on each paragraph text to obtain a first vector corresponding to each paragraph text;

[0194] Segment the initial text base into multiple sentence texts;

[0195] Each sentence text is vectorized to obtain a second vector corresponding to each sentence text.

[0196] The text matching device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0197] The text matching device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0198] The text matching device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0199] In some embodiments, Figure 4 As shown, an embodiment of the present application also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the above-mentioned text matching method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0200] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0201] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the various processes of the above-mentioned text matching method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0202] On the other hand, the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is implemented to perform the various processes of the above-mentioned text matching method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0203] On the other hand, an embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned text matching method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0204] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0205] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0206] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A text matching method, characterized in that: include: Processing the acquired initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library; processing the acquired question text to obtain a third vector corresponding to the question text; Based on the first vector and the third vector, obtaining a first similarity between the question text and the paragraph text; based on the second vector and the third vector, obtaining a second similarity between the question text and the paragraph text; Processing at least one keyword included in the question text to obtain the importance of each keyword to the paragraph text; Based on the first similarity, the second similarity and the importance, a target paragraph text corresponding to the question text is determined from the plurality of paragraph texts.

2. The text matching method according to claim 1, characterized in that: The processing of at least one keyword included in the question text to obtain the importance of each keyword to the paragraph text includes: Processing the question text to obtain the keyword quantity of the at least one keyword included in the question text and all the keywords included in the question text; Based on the question text and the initial text library, obtaining the inverse document frequency corresponding to each of the keywords and the occurrence frequency of each of the keywords in the initial text library; The importance of the keyword to the paragraph text is calculated based on the number of keywords, the inverse document frequency and the occurrence frequency.

3. The text matching method according to claim 1, characterized in that: The step of determining a target paragraph text corresponding to the question text from the plurality of paragraph texts based on the first similarity, the second similarity and the importance includes: The first similarity, the second similarity and the importance are fused to obtain a third similarity between the paragraph text and the question text; The paragraph text corresponding to the target third similarity greater than the target similarity threshold among the plurality of the third similarities is determined as the target paragraph text.

4. The text matching method according to claim 3, characterized in that: The fusing the first similarity, the second similarity and the importance to obtain a third similarity between the paragraph text and the question text includes: Obtaining a first weight of the first similarity relative to the third similarity, a second weight of the second similarity relative to the third similarity, and a third weight of the importance relative to the third similarity; Based on the first weight, the second weight and the third weight, the first similarity, the second similarity and the importance are fused to obtain the third similarity.

5. The text matching method according to any one of claims 1 to 4, characterized in that: The obtaining, based on the second vector and the third vector, a second similarity between the question text and the paragraph text includes: Based on the second vector and the third vector, a fourth similarity between the question text and the sentence text is calculated; Based on the mapping relationship between the sentence text and the paragraph text, a maximum value among fourth similarities corresponding to a plurality of sentence texts included in the paragraph text is determined as a second similarity corresponding to the paragraph text.

6. The text matching method according to any one of claims 1 to 4, characterized in that: The processing of the acquired initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library includes: Segmenting the initial text library into a plurality of paragraph texts; Performing vectorization processing on each of the paragraph texts to obtain a first vector corresponding to each of the paragraph texts; Segmenting the initial text library into a plurality of sentence texts; Vectorization is performed on each of the sentence texts to obtain a second vector corresponding to each of the sentence texts.

7. A text matching device, characterized in that: include: A first processing module is used to process the acquired initial text library to obtain a first vector corresponding to each paragraph text and a second vector corresponding to each sentence text included in the initial text library; and to process the acquired question text to obtain a third vector corresponding to the question text; A second processing module is used to obtain a first similarity between the question text and the paragraph text based on the first vector and the third vector; and obtain a second similarity between the question text and the paragraph text based on the second vector and the third vector; A third processing module is used to process at least one keyword included in the question text to obtain the importance of each keyword relative to the question text and the initial text library; The fourth processing module is used to determine a target paragraph text corresponding to the question text from the plurality of paragraph texts based on the first similarity, the second similarity and the importance.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the text matching method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text matching method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the text matching method according to any one of claims 1 to 6 is implemented.

Images